A long-tail distribution image recognition method based on grouping balance and diversity enhancement

Abstract

The present invention provides a long-tail distribution image recognition method based on grouping balance and diversity enhancement. The method includes grouping by utilizing the frequency of each category in an image recognition data set, and uniformly sampling to obtain an input image; effectively utilizing the information of the head category through a diversity enhancement module to enhance the feature diversity of the tail category, so as to alleviate the phenomenon of overfitting the tail category; inputting the classification confidence and the true annotation into a grouping balance loss module to balance the gradient and update the multi-label image recognition model; and the grouping balance loss can ignore all the positive category gradients of other groups, and only update the category of the current sampled group, so as to balance the gradients between the categories, thereby improving the performance of multi-label image recognition. The present invention can effectively balance the gradients of the categories during the network training process, and enhance the feature diversity of the tail category, thereby improving the performance of the multi-label image recognition task based on the long-tail distribution.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a long-tail distribution image recognition method based on grouping balance and diversity enhancement. Background Art

[0002] With the rapid development of computer vision and neural networks, the image single-label recognition task has achieved unprecedented breakthroughs and produced very good results in various public data sets. However, in real-world scenarios, if we do not perform deliberate screening, the images we obtain often contain multiple labels, so more and more researchers have begun to pay attention to multi-label image recognition tasks. The goal of multi-label image recognition is to identify all labels contained in the image. Compared with single-label image recognition tasks, since multi-label images contain multiple labels, the number of output label combinations increases geometrically with the increase in the number of labels, which brings huge challenges to multi-label image recognition tasks.

[0003] At present, many researchers have solved the problem of multi-label image recognition by modeling the correlation between labels. Although these methods have achieved good performance, some researchers have found that multi-label image recognition tasks are also affected by long-tail distribution like single-label image recognition. Long-tail distribution means that in the entire data set, some head classes have a large number of samples, while most tail classes contain only a small number of samples. Assuming that the data set has a long-tail distribution, if the data is trained directly, the network will tend to learn the head class more, which will have a great impact on the recognition performance.

[0004] At present, the methods for dealing with long-tail distribution can be generally summarized into two strategies: resampling and reweighting. The resampling strategy is to balance the sampling of each category by increasing the number of times the tail category is sampled, or reducing the number of times the head category is sampled. However, because each multi-label image contains multiple labels, the resampling strategy cannot balance the sampling of categories for multi-label images. The reweighting strategy achieves the purpose of balanced learning of each category by increasing the weight of the loss of the tail category and reducing the weight of the loss of the head category. However, the reweighting strategy is often very sensitive to parameters, and for multi-label images it will also destroy the co-occurrence relationship of labels to a certain extent. In addition, neither of these two strategies can alleviate the overfitting problem caused by too few samples of the tail category.

[0005] In summary, providing a long-tail distribution image recognition method that effectively balances the gradients of each category and enhances the feature diversity of the tail category, thereby improving the performance of multi-label image recognition tasks based on long-tail distribution and reducing the overfitting of the tail category is an issue that technicians in this field urgently need to solve. Summary of the invention

[0006] In response to the above-mentioned problems and needs, this solution proposes a long-tail distribution image recognition method based on grouping balance and diversity enhancement, which can solve the above-mentioned technical problems by adopting the following technical solutions.

[0007] To achieve the above object, the present invention provides the following technical solution: a long-tail distribution image recognition method based on grouping balance and diversity enhancement, comprising: Step 1: obtaining an image recognition data set, the image recognition data set obeys a long-tail distribution, and counting and grouping image categories in the image recognition data set;

[0008] Step 2: uniformly sample the images to be identified in each group based on the grouping result to obtain an input image, and input the input image into a convolutional neural network for feature extraction to obtain global feature information;

[0009] Step 3: construct a multi-label image recognition model based on long-tail distribution, input the extracted features into the multi-label image recognition model based on long-tail distribution for information processing, and obtain classification confidence;

[0010] Step 4: input the classification confidence and the true annotation into the multi-label image recognition model based on long-tail distribution, balance the gradient using the grouping balance loss function, update the multi-label image recognition model based on long-tail distribution, and obtain a trained multi-label image recognition model based on long-tail distribution;

[0011] Step 5: Use the trained multi-label image recognition model based on long-tail distribution to recognize the long-tail distribution image data and output the recognition result.

[0012] Furthermore, counting and grouping the image categories in the image recognition data set includes: counting the frequency of occurrence of each category in the image recognition data set, and then sorting the counted frequencies of each category in descending order, and dividing the categories into three groups according to the sorting information, the three categories being "head", "middle", and "tail".

[0013] Furthermore, the images in the image recognition data set are all multi-label images, each multi-label image contains multiple objects and labels, and the global features contain information of all labels.

[0014] Furthermore, the multi-label image recognition model based on long-tail distribution includes a diversity enhancement module and a group balance loss module. The global feature information extracted by the convolutional neural network is processed and input into the diversity enhancement module to obtain the classification confidence, and the information of the head category is used to enhance the feature diversity of the tail category to reduce the overfitting of the tail category.

[0015] Furthermore, the diversity enhancement module is composed of a fusion matrix and several fully connected layers. After decoupling the global feature information X extracted by the convolutional neural network through the feature decoupling convolution layer, the feature Xcaf corresponding to each category is obtained. ,in, Denotes the feature decoupling convolution layer. Since the feature decoupling layer can effectively decouple the features of each category, we can also regard this decoupling convolution layer as a classifier for each category, which can effectively classify the features of each category. Then, the features of each category are fused through the fusion matrix M, so that the tail category features can absorb the information of the head category features. Finally, the fully connected layer is used to map the fused features into classification confidence. , ,in, represents the fully connected layer, It means flattening the feature matrix into feature vectors.

[0016] Furthermore, the group balancing loss function adopted by the group balancing loss module is as follows: , where C represents the number of categories, represents the predicted classification confidence, Indicates the category label, represents the sigmoid function, is the grouping weight.

[0017] Furthermore, the grouping weight The value of follows the following rules:

[0018] , where G i Indicates the set of categories contained in the group to which the input sample belongs.

[0019] It can be seen from the above technical scheme that the beneficial effects of the present invention are: the present invention can effectively perform uniform sampling on each group through the group sampling method to achieve balanced sampling of each category, and utilize the information of the head category to enhance the feature diversity of the tail category, thereby improving the performance of multi-label image recognition tasks based on long-tail distribution, reducing the overfitting of the tail category, and through the group balancing module, the gradients of each category can be effectively balanced, thereby dealing with the long-tail distribution problem.

[0020] In addition to the objects, features and advantages described above, the best embodiments for implementing the present invention will be described in more detail below with reference to the accompanying drawings so that the features and advantages of the present invention can be easily understood. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present invention or the description of the prior art, wherein the drawings are only used to show some embodiments of the present invention, rather than limiting all embodiments of the present invention thereto.

[0022] Figure 1 The figure is a schematic diagram of a specific process of a long-tail distribution image recognition method based on grouping balance and diversity enhancement of the present invention.

[0023] Figure 2 Schematic diagram of the model structure of the multi-label image recognition model based on long-tail distribution in this embodiment.

[0024] Figure 3 Schematic diagram of the process of fusing features of various categories in this embodiment.

[0025] Figure 4 Schematic diagram of the gradient accumulation of each category in this embodiment.

[0026] Figure 5 Schematic diagram of the classifier visualization diagram in this embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the technical solution of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described in conjunction with the drawings of specific embodiments of the present invention. The same figure marks in the drawings represent the same parts. It should be noted that the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] This application uses the frequency of each category in the data set to group, and then uniformly samples each group to obtain the input image. The diversity enhancement module effectively uses the information of the head category to enhance the feature diversity of the tail category to reduce the phenomenon of overfitting the tail category, and sends the classification confidence and the true annotation to the group balance loss module to balance the gradient, update the multi-label image recognition model, and finally obtain a trained multi-label image recognition model, wherein the group balance loss can ignore all positive category gradients of other groups and only update the category of the currently sampled group, so as to balance the gradients between the categories, thereby ultimately improving the performance of multi-label image recognition.

[0029] like Figures 1 to 5 As shown, the method specifically includes the following steps: Step 1: Obtain an image recognition data set, the image recognition data set obeys a long-tail distribution, and count and group the image categories in the image recognition data set.

[0030] Specifically, counting and grouping the image categories in the image recognition data set includes: counting the frequency of occurrence of each category in the image recognition data set, and then sorting the counted frequency of each category in descending order, and dividing the categories into three groups according to the sorting information, and the three categories are "head", "middle" and "tail". During the training process, one group is uniformly sampled from the three groups each time, and then a sample is uniformly sampled from the group as an input image, wherein each category refers to all image categories in the data set, such as images of cats and dogs. The frequency of occurrence of all image categories in the image recognition data set is counted, and the counted frequency can form an array. For example, if there are 20 categories in the image recognition data set, an array with a length of 20 can be constructed. If category 2 appears during the statistics, the frequency at the corresponding position of array 2 is increased by 1. After the statistics are completed, the array represents the frequency of occurrence of all categories in the data set. Then, according to this frequency, it is sorted in descending order, so that the category with high frequency will be placed in front, and the category with low frequency will be placed in the back. Then, according to the sorting results, the categories are divided into 3 groups, so the high frequency ones are divided into the same group (head), the medium frequency ones (middle) are divided into one group, and the low frequency ones (tail) are divided into one group.

[0031] Step 2: Based on the grouping result, the images to be identified in each group are uniformly sampled to obtain an input image, and the input image is input into a convolutional neural network for feature extraction to obtain global feature information.

[0032] In this embodiment, the images in the image recognition data set are all multi-label images, each multi-label image contains multiple objects and labels, and the global features contain information of all labels.

[0033] Step 3: construct a multi-label image recognition model based on long-tail distribution, input the extracted features into the multi-label image recognition model based on long-tail distribution for information processing, and obtain classification confidence;

[0034] Specifically, the multi-label image recognition model based on long-tail distribution includes a diversity enhancement module and a group balance loss module. The global feature information extracted by the convolutional neural network is processed and input into the diversity enhancement module to obtain the classification confidence. In this way, the information of the head category can be effectively utilized to enhance the feature diversity of the tail category to reduce the phenomenon of overfitting the tail category.

[0035] In this method, the diversity enhancement module is composed of a fusion matrix and several fully connected layers, and the specific process of obtaining the confidence of classification is as follows: Since the multi-label image contains multiple objects and labels, the global feature contains the information of all labels, and feature decoupling is required. The global feature information X extracted by the convolutional neural network is decoupled through the feature decoupling convolution layer to obtain the feature Xcaf corresponding to each category. ,in, Denotes the feature decoupling convolution layer. Since the feature decoupling layer can effectively decouple the features of each category, we can also regard this decoupling convolution layer as a classifier for each category, which can effectively divide the features of each category. The diversity enhancement module consists of a fusion matrix and several fully connected layers. After decoupling the global feature information X extracted by the convolutional neural network through the feature decoupling convolution layer, the feature Xcaf corresponding to each category is obtained. ,in, Representation feature decoupling convolution layer

[0036] Then, the features of each category are fused through the fusion matrix M, so that the tail category features can absorb the information of the head category features, and finally the fully connected layer is used to map the fused features into classification confidence , ,in, represents the fully connected layer, It means flattening the feature matrix into feature vectors.

[0037] like Figure 3 As shown, the feature diversity of the tail categories is enhanced, thus alleviating the problem of network overfitting to the tail categories, thereby improving the performance of the classifier.

[0038] Step 4: Input the classification confidence and true annotation into the multi-label image recognition model based on long-tail distribution, balance the gradient using the group balance loss function, update the multi-label image recognition model based on long-tail distribution, and obtain a trained multi-label image recognition model based on long-tail distribution. The group balance module can ignore all positive category gradients of other groups and only update the category of the current sampled group, so as to balance the gradients between categories, thereby ultimately improving the performance of multi-label image recognition. Figure 4 As shown, it can be seen that the baseline method has a higher cumulative gradient value for the head category, while a lower cumulative gradient value for the tail category, indicating that directly training the long-tail distribution data without taking any measures will cause the network to tend to learn the head category, while the method of the present application can make the gradients of each category tend to be balanced, where a is the cumulative gradient of the baseline method and b is the cumulative gradient of this method.

[0039] In this application, the group balancing loss function used by the group balancing loss module is as follows: , where C represents the number of categories, represents the predicted classification confidence, Indicates the category label, represents the sigmoid function, is the grouping weight. The value of follows the following rules: , where G i Indicates the set of categories contained in the group to which the input sample belongs.

[0040] Step 5: Use the trained multi-label image recognition model based on long-tail distribution to recognize the long-tail distribution image data and output the recognition result.

[0041] This application can effectively use the information of the head category to enhance the feature diversity of the tail category to reduce the phenomenon of overfitting the tail category, and can effectively balance the gradients of each category to deal with the long-tail distribution problem. Figure 5 As shown, it can be seen that the classifiers of the tail category of the baseline method are clustered together, which indicates that due to too few samples of the tail category, the network overfits the tail category phenomenon. The method of the present application makes the classifiers of the tail category separated.

[0042] It should be noted that the embodiments described in the present invention are only preferred ways to implement the present invention, and any obvious modifications that belong to the overall concept of the present invention should fall within the protection scope of the present invention.

Claims

1. A long-tail distribution image recognition method based on grouping balance and diversity enhancement, characterized in that: The following steps are involved: Step 1: Obtain an image recognition data set, the image recognition data set follows a long-tail distribution, and count and group the image categories in the image recognition data set; Step 2: uniformly sample the images to be identified in each group based on the grouping result to obtain an input image, and input the input image into a convolutional neural network for feature extraction to obtain global feature information; Step 3: construct a multi-label image recognition model based on long-tail distribution, input the extracted features into the multi-label image recognition model based on long-tail distribution for information processing, and obtain classification confidence; Step 4: input the classification confidence and the true annotation into the multi-label image recognition model based on long-tail distribution, balance the gradient using the grouping balance loss function, update the multi-label image recognition model based on long-tail distribution, and obtain a trained multi-label image recognition model based on long-tail distribution; Step 5: Use the trained multi-label image recognition model based on long-tail distribution to recognize the long-tail distribution image data and output the recognition result; The multi-label image recognition model based on long-tail distribution includes a diversity enhancement module and a grouping balance loss module. The global feature information extracted by the convolutional neural network is processed and input into the diversity enhancement module to obtain the classification confidence. The information of the head category is used to enhance the feature diversity of the tail category and reduce the overfitting of the tail category. The diversity enhancement module consists of a fusion matrix and several fully connected layers. After decoupling the global feature information X extracted by the convolutional neural network through the feature decoupling convolution layer, the feature Xcaf corresponding to each category is obtained. ,in, It represents the feature decoupling convolution layer, and then the features of each category are fused through the fusion matrix M, so that the tail category features can absorb the information of the head category features, and finally the fully connected layer is used to map the fused features into classification confidence , ,in, represents the fully connected layer, Indicates flattening the feature matrix into feature vectors; The group balance loss function used by the group balance loss module is as follows: , where C represents the number of categories, represents the predicted classification confidence, Indicates the category label, represents the sigmoid function, is the grouping weight.

2. The long-tail distribution image recognition method based on grouping balance and diversity enhancement as claimed in claim 1, characterized in that: Counting and grouping the image categories in the image recognition data set includes: counting the frequency of occurrence of each category of images in the image recognition data set, and then sorting the counted frequencies of each category in descending order, and dividing the categories into three groups according to the sorting information, the three categories being "head", "middle", and "tail".

3. The long-tail distribution image recognition method based on grouping balance and diversity enhancement as claimed in claim 2, characterized in that: The images in the image recognition dataset are all multi-label images, each of which contains multiple objects and labels, and the global features contain information of all labels.

4. The long-tail distribution image recognition method based on grouping balance and diversity enhancement as claimed in claim 1, characterized in that: The group weight The value of follows the following rules: , where G i Indicates the set of categories contained in the group to which the input sample belongs.

Images