A training method for long-tail image recognition models

Abstract

In an embodiment of the present invention, a training method for a long-tail image recognition model is provided, which belongs to the field of data processing technology, and specifically includes: sampling pictures using a balanced sampling strategy and a random sampling strategy respectively, and performing data enhancement on the randomly sampled pictures to obtain a first sample; performing a CamMix operation to obtain a second sample; inputting the first sample and the second sample into a backbone network to obtain sample features; obtaining a cross entropy loss, and using the parameters in the linear classifier as prototype features that have not been normalized by L2; obtaining L2 normalized features from the prototype features and sample features through a multi-layer perceptron network, and inputting them into a prototype decoupled contrast learning loss function to obtain contrast loss; combining the cross entropy loss and the contrast loss to obtain a total loss function and guiding the backbone network training accordingly to obtain a long-tail image recognition model. Through the scheme of the present invention, the recognition accuracy and robustness of the model are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a training method for a long-tail image recognition model. Background Art

[0002] At present, with the development of deep learning technology, image recognition, especially in the application of long-tail distribution data with limited resources, has become increasingly important. However, existing image recognition methods often have performance degradation when processing long-tail distribution data sets, especially in the recognition accuracy of tail categories. This is mainly due to the scarcity of tail category samples, which makes it difficult for the model to learn effective feature expressions from them. In addition, existing methods also have shortcomings in dealing with feature differences between different categories and feature consistency within a class.

[0003] It can be seen that there is an urgent need for a training method for long-tail image recognition models that can improve the recognition accuracy and robustness of the model. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a training method for a long-tail image recognition model, which at least partially solves the problem of poor model recognition accuracy and robustness in the prior art.

[0005] The embodiment of the present invention provides a training method for a long-tail image recognition model, comprising:

[0006] Step 1: Sample pictures using a balanced sampling strategy and a random sampling strategy respectively to obtain a balanced sampling picture and a random sampling picture, and perform data enhancement on the random sampling picture to obtain a first sample, wherein the first sample includes an anchor sample , anchor samples are generated by image enhancement and other samples in a batch that have the same class label as the anchor sample ;

[0007] Step 2: Perform CamMix operation on the balanced sampling image and the random sampling image to obtain the second sample ;

[0008] Step 3: Input the first sample and the second sample into the backbone network to obtain the sample features without L2 normalization. , , , ;

[0009] Step 4: Input the linear classifier to get the cross entropy loss, and use the parameters in the linear classifier as the prototype features without L2 normalization ;

[0010] Step 5: Prototype features and sample characteristics , , The L2 normalized features are obtained through the multi-layer perceptron network , , , , and input it into the prototype decoupled contrastive learning loss function to obtain the contrastive loss;

[0011] Step 6: Combine the cross entropy loss and contrast loss to obtain the total loss function and use it to guide the training of the backbone network to obtain a long-tail image recognition model.

[0012] According to a specific implementation of the embodiment of the present invention, step 1 specifically includes:

[0013] Step 1.1: Use the balanced sampling strategy to obtain a balanced sampling image , choose with equal probability The category label of

[0014] Step 1.2: Use a random sampler to randomly sample from the distribution of the original data set to obtain a random sample image. and And use the data enhancement function Perform data enhancement to obtain the first sample , and , wherein the data enhancement function includes grayscale transformation, random color jitter and horizontal flipping.

[0015] According to a specific implementation of the embodiment of the present invention, step 2 specifically includes:

[0016] Input the balanced sampling images into the backbone network to obtain the class activation heat map, and use the threshold The class activation heat map is processed, and the part greater than or equal to the threshold is set to 1, and the part less than or equal to the threshold is set to 0, to obtain the region boundary for performing the CutMix operation. According to the region boundary, the balanced sampling image is Mix to randomly sampled images The second sample is obtained .

[0017] According to a specific implementation of an embodiment of the present invention, the expression of the cross entropy loss is:

[0018]

[0019] in, For a control category The parameters of importance, Represents the logit value of the model output, To indicate input The value of logit for each category, is the category to which the loss sample currently calculated belongs, Indicates other categories, represents the number of categories, Yes Category The compensation value is defined as and ,in Yes Category The prior probability information of the category Frequency of occurrence.

[0020] According to a specific implementation of the embodiment of the present invention, the expression of the contrast loss is:

[0021]

[0022]

[0023] in, represents the set of all samples in a batch, represents all the features in a batch, express A positive sample set, wherein the positive sample set includes features of the sample features corresponding to the first sample after L2 normalization, express The negative sample set, represents the set of all prototype features after L2 normalization, express The negative prototype feature set after L2 normalization, Indicates that different weights are assigned to different types of positive samples. represents the hyperparameter, represents the temperature parameter in contrastive learning, Representing a collection The elements in Representing a collection The elements in .

[0024] According to a specific implementation of the embodiment of the present invention, the total loss function is:

[0025]

[0026] in, is a hyperparameter.

[0027] The training scheme for the long-tail image recognition model in the embodiment of the present invention includes: step 1, sampling pictures using a balanced sampling strategy and a random sampling strategy respectively, obtaining a balanced sampling picture and a random sampling picture, and performing data enhancement on the random sampling picture to obtain a first sample, wherein the first sample includes an anchor sample , anchor samples are generated by image enhancement and other samples in a batch that have the same class label as the anchor sample ; Step 2, perform CamMix operation on the balanced sampling picture and the random sampling picture to obtain the second sample ; Step 3, input the first sample and the second sample into the backbone network to obtain the sample features without L2 normalization , , , ; Step 4, the sample features Input the linear classifier to get the cross entropy loss, and use the parameters in the linear classifier as the prototype features without L2 normalization ; Step 5, prototype features and sample characteristics , , The L2 normalized features are obtained through the multi-layer perceptron network , , , , and input it into the prototype decoupled contrastive learning loss function to obtain the contrastive loss; Step 6, combine the cross entropy loss and the contrastive loss to obtain the total loss function and use it to guide the backbone network training to obtain the long-tail image recognition model.

[0028] The beneficial effects of the embodiments of the present invention are as follows: through the scheme of the present invention, in the contrastive learning branch, the present invention uses the proposed prototype decoupled contrastive learning as the contrastive learning loss function, and the introduction of the prototype can help the model better learn the features of different categories. The decoupling strategy mainly solves the problem that contrastive learning under long-tail distribution is prone to biased optimization of intra-class features by assigning different weights to different types of positive samples, helping the model to learn more uniform and robust features; in the classification branch, the present invention uses the cross entropy loss with logical compensation as the classification loss function, and the logical compensation corrects the classification boundary by fine-tuning the logical value of the model output. At the same time, we apply CamMix to this branch. CamMix combines the class activation heat map with the traditional CutMix, and uses the good positioning ability of the class activation heat map to mix samples. CamMix mixes the valuable areas in the tail class samples with other samples to enrich the tail class information, thereby improving the accuracy of long-tail image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of a training method for a long-tail image recognition model provided by an embodiment of the present invention;

[0031] Figure 2 A system framework diagram of a training method for a long-tail image recognition model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0034] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0035] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0037] Although deep learning models have achieved remarkable success in computer vision tasks, such as image recognition, ResNet, DenseNet, object detection, and semantic segmentation, they often face the problem of long-tail distribution of training samples in real-world applications, that is, the head category has a large number of samples while the tail category has only a few samples. This distribution leads to significant performance differences between the head and tail categories when training deep models.

[0038] In order to solve the data imbalance problem in long-tail image recognition, researchers have proposed a variety of methods, including resampling training data, reweighting loss functions, logistic compensation, data augmentation and other strategies. In particular, supervised contrastive learning (SCL) has attracted increasing attention due to its excellent representation learning ability. Its core idea is to minimize the distance between samples of the same type while maximizing the distance between samples of different types. Recently, applying SCL to long-tail image recognition has shown good performance.

[0039] Contrastive learning has also received significant attention due to its effective feature learning capabilities and has inspired various methods to address the challenges of long-tail image recognition. For example, KCL enhances SCL by introducing K positive samples for each category to ensure a balanced feature space between categories. TSC uniformly defines positions for each category in the feature space and uses contrastive learning to guide different category features to move closer to these predefined positions, thereby promoting uniform distribution of features. Hybrid-SC adopts a dual-branch network, combines contrastive learning to improve feature expression, and combines it with classifier training to improve recognition performance. PaCo and GPaCo optimize long-tail recognition by introducing learnable class center parameters within the SCL framework. BCL points out that SCL cannot learn uniform feature distribution in long-tail data distribution, and therefore introduces class supplementation and class averaging strategies to guide the model to learn uniform feature distribution. SBCL decomposes the head category into multiple semantically consistent subcategories through adaptive clustering, and uses subcategory labels and instance labels to calculate the dual-granularity contrastive loss. ProCo uses von Mises-Fisher (vMF) distribution to model feature distribution, allowing the estimation of cross-category distribution from small batch features and addressing the limitations of SCL on large batch sizes. DSCL proposes a patch-based self-distillation strategy to enhance the representation learning of tail categories by optimizing inter-class distances through self-distillation and transferring knowledge from head categories to tail categories based on shared visual patterns.

[0040] Although recent research has achieved certain results, there are still few studies on intra-class feature consistency under long-tail distribution. How to further improve the performance of long-tail image recognition tasks remains a major challenge in the research community.

[0041] The present invention aims to prevent biased optimization between samples within a class through prototype decoupled contrastive learning, thereby improving the overall performance of image recognition and the recognition accuracy of tail categories. The proposed dual-branch framework trains the feature extractor and classifier simultaneously in an end-to-end manner, which provides a more effective and robust solution for long-tail image recognition.

[0042] An embodiment of the present invention provides a training method for a long-tail image recognition model, which can be applied to the long-tail image recognition process in a deep learning scenario.

[0043] See also Figure 1 , is a flow chart of a training method for a long-tail image recognition model provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method mainly comprises the following steps:

[0044] Step 1: Sample pictures using a balanced sampling strategy and a random sampling strategy respectively to obtain a balanced sampling picture and a random sampling picture, and perform data enhancement on the random sampling picture to obtain a first sample, wherein the first sample includes an anchor sample , anchor samples are generated by image enhancement and other samples in a batch that have the same class label as the anchor sample ;

[0045] Furthermore, the step 1 specifically includes:

[0046] Step 1.1: Use the balanced sampling strategy to obtain a balanced sampling image , with equal probability The category label of

[0047] Step 1.2: Use a random sampler to randomly sample from the distribution of the original data set to obtain a random sample image. and And use the data enhancement function Perform data enhancement to obtain the first sample , and , wherein the data enhancement function includes grayscale transformation, random color jitter and horizontal flipping.

[0048] In specific implementation, a balanced sampler can be used to sample images with equal probability. , the random sampler randomly samples images from the original distribution and The balanced sampler aims to provide equal representation for all categories. The random sampler selects samples according to the distribution of the original dataset without any category balancing. The sampled images are augmented using the data augmentation function Enhance the sample , and , It includes common operations such as grayscale transformation, random color jitter, horizontal flip, etc. This function is used to provide rich feature representation for the contrastive learning branch.

[0049] Step 2: Perform CamMix operation on the balanced sampling image and the random sampling image to obtain the second sample ;

[0050] Based on the above embodiment, step 2 specifically includes:

[0051] Input the balanced sampling images into the backbone network to obtain the class activation heat map, and use the threshold The class activation heat map is processed, and the part greater than or equal to the threshold is set to 1, and the part less than or equal to the threshold is set to 0, to obtain the region boundary for performing the CutMix operation. According to the region boundary, the balanced sampling image is Mix to randomly sampled images The second sample is obtained .

[0052] In specific implementation, CamMix combines the class activation heatmap with CutMix, and uses the good positioning ability of the class activation heatmap to obtain the area that the model pays most attention to. Mixing the area of ​​the tail sample into other images can effectively enrich the information of the tail class samples, and input the image obtained by the balanced sampler into the backbone network to obtain its class activation heatmap. Use threshold The heat map is processed, and the part greater than or equal to the threshold is set to 1, and the part less than or equal to the threshold is set to 0, to obtain the region boundary for performing the CutMix operation. Mix to randomly sampled images The second sample is obtained , thereby enriching the tail class sample information.

[0053] Specifically, and are the foreground image obtained by the balanced sampler and the background image obtained by the random sampler. We hope to mix them by the following equation and Generate new images :

[0054]

[0055] in are the parameters in the linear classifier, is the feature map obtained from the last layer. Represents the normalization operation. From the feature map Export a mask if the element is in The value in is greater than or equal to a predefined threshold , then the value of the element is set to 1, otherwise it is 0. is the maximum connected area in the mask, the mixed image Defined as:

[0056]

[0057] in Represents a binary mask used to select CamMix bounding boxes. is a binary mask filled with 1s, Represents element-wise multiplication. Labels for mixed images Defined as:

[0058]

[0059] in and are the labels of foreground and background images respectively. Corresponding to the ratio of foreground area mixing. In this way, tail samples are mixed into other samples, thus enriching the tail class sample information.

[0060] Step 3: Input the first sample and the second sample into the backbone network to obtain the sample features without L2 normalization. , , , ;

[0061] In the specific implementation, considering that the two branches in the framework share the same backbone network, all images are input into the network backbone to obtain sample features that have not been L2 normalized. , , , .

[0062] Step 4: Input the linear classifier to get the cross entropy loss, and use the parameters in the linear classifier as the prototype features without L2 normalization ;

[0063] Furthermore, the expression of the cross entropy loss is:

[0064]

[0065] in, For a control category The parameters of importance, Represents the logit value of the model output, To indicate input The value of logit for each category, is the category to which the loss sample currently calculated belongs, Indicates other categories, Yes Category The compensation value is defined as and ,in Yes Category The prior probability information of the category Frequency of occurrence.

[0066] In the specific implementation, the features are processed by a multi-layer perceptron (MLP) network to obtain L2-normalized features. In the classification branch, we use a cross-entropy loss function with a logistic compensation strategy. Logistic compensation fine-tunes the logistic value output by the linear classifier to alleviate data imbalance. It is defined as follows:

[0067]

[0068] in, For a control class The parameters of importance, Represents the logit value of the model output, To indicate input The value of the logit for each category. is the category to which the loss sample currently calculated belongs, Indicates other categories, represents the number of categories, Yes Category The compensation value is defined as and ,in Yes Category The prior probability information of the category Frequency of occurrence.

[0069] Step 5: Prototype features and sample characteristics , , The L2 normalized features are obtained through the multi-layer perceptron network , , , , and input it into the prototype decoupled contrastive learning loss function to obtain the contrastive loss;

[0070] Furthermore, the expression of the contrast loss is:

[0071]

[0072]

[0073] in, represents the set of all samples in a batch, represents all the features in a batch, express A positive sample set, wherein the positive sample set includes features of the sample features corresponding to the first sample after L2 normalization, express The negative sample set, represents the set of all prototype features after L2 normalization, express The negative prototype feature set after L2 normalization, Indicates that different weights are assigned to different types of positive samples. represents the hyperparameter, represents the temperature parameter in the contrastive learning loss function, Representing a collection The elements in Representing a collection In the specific implementation, we use the proposed prototype disentangled contrastive learning (PDCL) loss function in the contrastive learning branch, and the parameter definitions are shown in Table 1.

[0074] Table 1

[0075]

[0076] In the contrastive learning branch, we use the proposed prototype decoupled contrastive learning loss function. Given an anchor sample in a batch, there are three types of positive samples in the loss function, including samples generated by the anchor sample through image enhancement (a), the corresponding prototype samples of the anchor sample (b), and other samples in a batch with the same class label as the anchor sample (c). The prototype decoupled contrastive learning loss function assigns different weights to the three different types of positive samples to prevent biased optimization of intra-class features, where the weight assigned to class (a) is , the weight given to class (b) , the weight given to class (c) Given a training data set ,in It is samples, It is The goal of PDCL is to train an excellent Parameterized Feature Extractor , and train a linear classifier , so that This means that the model should be able to accurately predict the label of each example through the feature extractor and classifier.

[0077] According to the definition of symbols in the table, we can express PDCL as follows:

[0078]

[0079] in Different weights are assigned to different types of positive samples, which are defined as follows:

[0080]

[0081] here is a hyperparameter.

[0082] Next, we calculate the contrast loss function PDCL for the anchor sample The gradient is:

[0083] ;

[0084]

[0085]

[0086]

[0087]

[0088] By introducing , the gradients for three types of positive samples in PDCL can be calculated as:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] in , , so we think ,

[0096] and then and as well as and The gradient L2 norm ratio can be calculated as:

[0097]

[0098]

[0099] When PSCL converges, and The optimal conditional probability is:

[0100] ,

[0101] Through the above analysis, the gradient contribution ratio of the three types of positive samples in PDCL is kept at a constant value, so that the optimization of the intra-class sample distance remains stable and will not be affected by the long-tail distribution and thus will not cause biased optimization.

[0102] Step 6: Combine the cross entropy loss and contrast loss to obtain the total loss function and use it to guide the training of the backbone network to obtain a long-tail image recognition model.

[0103] Furthermore, the total loss function is

[0104]

[0105] in, is a hyperparameter.

[0106] In specific implementation, after calculating the contrastive learning loss and cross entropy loss, we get the total loss function of the proposed framework:

[0107]

[0108] in is a hyperparameter that controls Impact on model training.

[0109] After obtaining the total loss function, it can be used to guide the training of the backbone network to obtain a long-tail image recognition model.

[0110] The training method for the long-tail image recognition model provided in this embodiment uses the proposed prototype decoupled contrastive learning as the contrastive learning loss function in the contrastive learning branch. The introduction of the prototype can help the model better learn the features of different categories. The decoupling strategy mainly solves the problem that contrastive learning under long-tail distribution is prone to biased optimization of intra-class features by assigning different weights to different types of positive samples, helping the model to learn more uniform and robust features; in the classification branch, the present invention uses the cross entropy loss with logical compensation as the classification loss function. The logical compensation corrects the classification boundary by fine-tuning the logical value of the model output. At the same time, we apply CamMix to this branch. CamMix combines the class activation heat map with the traditional CutMix, and uses the good positioning ability of the class activation heat map to mix samples. CamMix mixes the valuable areas in the tail class samples with other samples to enrich the tail class information, thereby improving the accuracy of long-tail image recognition.

[0111] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A training method for a long-tail image recognition model, characterized in that: include: Step 1: Sample pictures using a balanced sampling strategy and a random sampling strategy respectively to obtain a balanced sampling picture and a random sampling picture, and perform data enhancement on the random sampling picture to obtain a first sample, wherein the first sample includes an anchor sample , anchor samples are generated by image enhancement and other samples in a batch that have the same class label as the anchor sample ; Step 2: Perform CamMix operation on the balanced sampling image and the random sampling image to obtain the second sample ; Step 3: Input the first sample and the second sample into the backbone network to obtain the sample features without L2 normalization. , , , ; Step 4: Input the linear classifier to get the cross entropy loss, and use the parameters in the linear classifier as the prototype features without L2 normalization ; Step 5: Prototype features and sample characteristics , , The L2 normalized features are obtained through the multi-layer perceptron network , , , , and input it into the prototype decoupled contrast learning loss function to obtain the contrast loss, where the expression of the contrast loss is in, Represents the set of all samples in a batch represents all the features in a batch, express A positive sample set, wherein the positive sample set includes features of the sample features corresponding to the first sample after L2 normalization, express The negative sample set, represents the set of all prototype features after L2 normalization, express The negative prototype feature set after L2 normalization, Indicates that different weights are assigned to different types of positive samples. represents the hyperparameter, represents the temperature parameter in contrastive learning, Representing a collection The elements in Representing a collection Elements in Step 6: Combine the cross entropy loss and contrast loss to obtain the total loss function and use it to guide the training of the backbone network to obtain a long-tail image recognition model.

2. The method according to claim 1, characterized in that The step 1 specifically includes: Step 1.1: Use the balanced sampling strategy to obtain a balanced sampling image , choose with equal probability The category label of Step 1.2: Use a random sampler to randomly sample from the distribution of the original data set to obtain a random sample image. and And use the data enhancement function Perform data enhancement to obtain the first sample , and , wherein the data enhancement function includes grayscale transformation, random color jitter and horizontal flipping.

3. The method according to claim 2, characterized in that The step 2 specifically includes: Input the balanced sampling images into the backbone network to obtain the class activation heat map, and use the threshold The class activation heat map is processed, and the part greater than or equal to the threshold is set to 1, and the part less than or equal to the threshold is set to 0, to obtain the region boundary for performing the CutMix operation. According to the region boundary, the balanced sampling image is Mix to randomly sampled images The second sample is obtained .

4. The method according to claim 3, characterized in that The expression of the cross entropy loss is in, For a control category The parameters of importance, Represents the logit value of the model output, To indicate input The value of logit for each category, is the category to which the loss sample currently calculated belongs, Indicates other categories, represents the number of categories, Yes Category The compensation value is defined as and ,in Yes Category The prior probability information of the category Frequency of occurrence.

5. The method according to claim 4, characterized in that The total loss function is in, is a hyperparameter.

Images