A multi-label image classification method for optimizing category distribution imbalance

Abstract

This invention provides an optimized multi-label image classification method for imbalanced class distribution. On the one hand, when the data class labels are imbalanced, the scarce supervisory information provided by the imbalanced labels is combined with a semi-supervised learning model to mine the value of the labels, significantly improving the accuracy of the classification results. On the other hand, by discarding this imbalanced label information and adopting a self-supervised pre-training learning method, a better initial weight expression form is learned, which also has a significant effect on improving the accuracy of the classification results of multi-label images with imbalanced class distribution.

Description

Technical Field

[0001] This invention relates to the field of multi-label image classification, and more specifically, to a method for optimizing multi-label image classification with imbalanced category distribution. Background Technology

[0002] Multi-label image classification involves assigning multiple labels to an image. These labels can be user-defined or extracted from existing labels. Because image data typically has multiple labels, this classification method usually improves the accuracy and robustness of classification.

[0003] However, imbalanced data class distribution is a common problem in practical applications. Due to this imbalance, a single feature extraction method often cannot fully represent the multiple label information in an image. Furthermore, if imbalanced sample data is directly fed into a machine learning model, the model typically performs better in generalizations of classes with larger sample sizes. This is because the model learns a much larger number of samples in the imbalanced class category, resulting in biased generalization. Therefore, researching multi-label image classification methods for imbalanced class distribution is of great significance for improving the effectiveness of image classification in practical applications.

[0004] Existing technologies typically address the imbalance problem from two aspects: the data level and the model level. However, each of these approaches has its own shortcomings and limitations:

[0005] 1. Insufficient data utilization at the data level, resulting in overfitting or underfitting of the model: Existing sample resampling techniques, whether oversampling of a small number of samples or undersampling of a large number of samples, may lead to insufficient feature extraction of some samples and loss of some important feature information, which may limit the generalization ability of the model and thus fail to obtain robust prediction results.

[0006] 2. High dependence on the dataset at the model level, prone to overfitting: The model layer primarily employs reweighting techniques to modify the model's loss function, increasing the reward for fewer samples in the loss calculation. Focal Loss, as commonly used, relies on the loss function and distribution in the training dataset. If the distribution in the training dataset does not match the task requirements, it may lead to a decrease in model prediction performance. When the model uses too many negative samples, Focal Loss may cause the model to overfit to negative samples, thus resulting in a decline in model prediction performance. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-label image classification method for optimizing class imbalance, which is based on semi-supervised learning to mine the value of unlabeled data with imbalance, and significantly improves the final classification accuracy.

[0008] Another objective of this invention is to provide a multi-label image classification method for optimizing imbalanced category distribution. This method can learn a better initial weight expression by discarding imbalanced label information and adopting a self-supervised pre-training learning approach, thereby significantly improving the accuracy of classification results.

[0009] The embodiments of the present invention are implemented as follows:

[0010] An optimization method for multi-label image classification with imbalanced class distribution includes a semi-supervised learning-based method for multi-label image classification, which includes generating a new semi-supervised learning model. The steps for generating the new semi-supervised learning model include:

[0011] Obtain the original dataset with imbalanced classes and represent the class / class combination represented by each data point in the original dataset with numbers;

[0012] The original dataset was divided into labeled and unlabeled datasets in a 1:3 ratio using a random algorithm, and pseudo-labeled data was generated from the data in the unlabeled dataset.

[0013] The classifier used in the semi-supervised learning method is trained using a labeled dataset to obtain a semi-supervised classifier.

[0014] A semi-supervised classifier is used to predict the pseudo-labels of the pseudo-labeled data, resulting in a pseudo-labeled dataset.

[0015] Select labeled and pseudo-labeled datasets as inputs to minimize the loss function λ(D). L ,θ)+ωλ(D U We perform calculations on θ to obtain a new semi-supervised classifier and a new semi-supervised learning model.

[0016] In a preferred embodiment of the present invention, the above-described semi-supervised learning method is used to enhance the generalization performance and confidence of unlabeled data.

[0017] In a preferred embodiment of the present invention, the novel semi-supervised classifier is applied to a multi-label image classification task with an imbalanced class distribution. This is achieved by adding a multi-output sigmoid layer after the novel semi-supervised classifier, or by using a threshold function to convert the output of the novel semi-supervised classifier into multiple binary labels.

[0018] In a preferred embodiment of the present invention, the above-described semi-supervised learning method employs a Guassian mixture model with different means μ1 and μ2 but the same variance; the classifier is a Bayesian optimal classifier. The average mean is

[0019] A multi-label image classification method for optimizing imbalanced class distribution, based on self-supervised learning, includes the following steps:

[0020] The pretext task, which uses self-supervised learning, is used to learn feature representations of unlabeled images in the samples.

[0021] The samples are trained using a self-supervised algorithm and a contrastive learning method to obtain a new self-supervised learning model and the initial feature information parameters of the samples. The self-supervised algorithm uses a linear classifier f(x) = sign(<θ,feature>+b), b>0, where b is the bias vector value and feature represents the sample data without labels.

[0022] By combining the new self-supervised learning model with the loss function network model, a multi-objective self-supervised classifier and a multi-objective self-supervised learning model are generated using the initialized feature information parameters.

[0023] In a preferred embodiment of the present invention, the metric for the linear classifier described above is: err f =P(X,Y):P XY (f(X)≠Y), where X is a d-dimensional Guassian mixture model; the positive label Y = +1 has a probability p. + The label negative class Y = -1 has a probability p - =1-p + Assuming the negative class is the primary class, when p - When ≥0.5, it satisfies and Among them, random values ​​where β>3.

[0024] In a preferred embodiment of the present invention, the self-supervised algorithm is Rotation prediction, and the contrastive learning method is MoCo.

[0025] In a preferred embodiment of the present invention, the loss function is the LDAM-DRW loss function.

[0026] In a preferred embodiment of the present invention, the multi-object classifier generated above is used as a pre-trained parameter model for multi-label image classification.

[0027] The beneficial effects of this invention are as follows: Starting from the inherent value of imbalanced data labels, and based on the principle that supervised learning generally achieves higher accuracy than unsupervised learning on a given task, this invention addresses the issue of imbalanced data labels. Firstly, by leveraging the scarce supervisory information provided by the imbalanced labels in conjunction with a semi-supervised learning model to extract the value of the labels, it significantly improves the accuracy of the classification results. Secondly, by discarding this imbalanced label information and employing a self-supervised pre-training approach, it learns a more effective initial weight representation, which also significantly contributes to improving the accuracy of the classification results. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a summary flow of semi-supervised learning in the embodiments of the present invention;

[0030] Figure 2 This is a flowchart illustrating the learning process for initialization parameters in image classification within a semi-supervised learning framework according to an embodiment of the present invention.

[0031] Figure 3 This is a flowchart illustrating the self-supervised learning process according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0034] Existing label classification methods for imbalanced sample distribution often suffer from problems such as insufficient feature extraction or excessive dependence on the dataset, for example:

[0035] The publication number is CN111382800A, which describes a multi-label multi-classification method suitable for imbalanced sample distribution. It mainly constructs a neural network-based multi-label multi-classification model, sets a comparison object and a comparison average, and finally selects the top N labels whose comparison values ​​are closest to the comparison average as the image classification result. Problems include: 1. The selection of the comparison object is highly subjective, significantly affecting the final classification effect; 2. The label data for imbalanced samples is not fully utilized, and there is still potential for further mining of label information data.

[0036] For example, publication number CN115984607A describes a multi-label image classification method and system based on imbalanced class representativeness. This method primarily optimizes the loss function. The dynamic loss for sample representativeness is composed of classification weights and dynamic focal loss. The classification weights are calculated by inputting the co-occurrence rate of the current class with other classes and the number of classes into a representativeness coordination function. The dynamic focal loss is obtained by combining the logits output by the classifier and the classification weights calculated for each class for each sample. The problems with this method are: 1. The representativeness of the samples is highly dependent on the dataset; 2. The selection of the representativeness coordination function is highly subjective and has a strong correlation with the classification results.

[0037] To address this, this embodiment provides semi-supervised learning and self-supervised learning methods for imbalanced samples, both of which can improve the accuracy of label classification for imbalanced samples.

[0038] Semi-supervised learning steps for imbalanced samples: including generating a new semi-supervised learning model. Import the image into the new semi-supervised learning model. Perform multi-label classification.

[0039] For details, please see Figure 1 New semi-supervised learning model The generation steps include:

[0040] S1: Obtain the original dataset with imbalanced classes and represent the class / class combination represented by each data point in the original dataset with numbers;

[0041] Specifically, 20,000 natural scene images were selected as the dataset, which includes five categories of labels: desert, mountains, ocean, sunset, and trees. Different category labels form a label group. Since the images in the dataset may belong to multiple categories, each image is assigned a label group to represent its category, as detailed in Appendix Table 1.

[0042]

[0043]

[0044] Table 1

[0045] As can be seen from Table 1, the number of images belonging to more than one category accounts for more than 22% of the dataset, while some combined categories are very few. On average, each image is associated with approximately 1.24 category labels.

[0046] We represent the category of each image by calculating the power label of each label group. For example, an image in the ocean + sunset category would have a label group of (0,0,1,1,0). Converting this from right to left in binary to decimal gives us 2. 1 +2 2 =6, so this 6 can represent ocean + sunset. We will use the number 6 as the power label of the image to facilitate subsequent data processing.

[0047] S2: Use a random algorithm to divide the original dataset into labeled and unlabeled datasets in a 1:3 ratio, and generate pseudo-labeled data from the data in the unlabeled dataset;

[0048] Specifically, the dataset in S1 is randomly divided into two parts at a ratio of 1:3, and the original labeled dataset is set to D. L The unlabeled dataset is D U , ensure in D L and D U The distributions of the sample data on D are similar. U Generate pseudo-label data This embodiment uses unlabeled data, which will not cause loss to existing sample data, nor will it impose a human labeling burden on the generated sample label data.

[0049] S3: Train the classifier used in the semi-supervised learning method using a labeled dataset to obtain the semi-supervised classifier.

[0050] Specifically, in the labeled dataset D L The above uses the simplest semi-supervised learning method to train and obtain a semi-supervised classifier for intermediate steps. Semi-supervised learning methods, such as Pseudo-Label, Virtual Adversarial Training, and Temporal Ensembling, train a classifier using a small number of labeled images and a large number of unlabeled images. These algorithms typically employ an encoder-classifier network structure, where the encoder extracts image features and the classifier predicts image labels. The core idea behind these algorithms is to leverage techniques such as entropy minimization, consistency regularization, and regularization to enhance the model's generalization ability and confidence on unlabeled data.

[0051] S4: Apply a semi-supervised classifier to predict the image pseudo-labels of the pseudo-label data to obtain the pseudo-label dataset;

[0052] semi-supervised classifier Applied to unlabeled data D U pseudo-tags

[0053] Specifically, a Bayesian optimal classifier is obtained by selecting a Guassian mixture model with different means μ1 and μ2 but the same variance: The average mean is

[0054] S5: Input labeled and pseudo-labeled datasets into the minimum loss function λ(D) L ,θ)+ωλ(D U ,θ) are calculated to obtain a new semi-supervised classifier and a new semi-supervised learning model;

[0055] Specifically, in conjunction with D L and D U Data to minimize the loss function λ(D) L ,θ)+ωλ(D U The final network model is obtained by (θ). Where ω is the unlabeled weight, this process aims to reshape D U The class distribution is analyzed to obtain better class boundaries, especially for a small number of sample classes. This is specifically for... and The learning strategy can select a suitable semi-supervised learning method based on existing imbalanced algorithms. In this embodiment, the loss function to be minimized is the cross-entropy loss function CE and the LDAW-DRW loss function.

[0056] S6: Apply the trained classifier to a multi-label image classification task. This can be achieved by adding a multi-output sigmoid layer after the classifier, or by using a threshold function to convert the classifier's output into multiple binary labels. The classification results show that unlabeled data helps model clearer class boundaries, resulting in better separation between classes. Unlabeled data effectively increases the sample size in low-density regions, and combined with more robust regularization, ensures better boundary modeling by the model.

[0057] To mitigate the impact of erroneous pseudo-labels on model learning and improve the accuracy of pseudo-label data, this embodiment also incorporates a multi-level evaluation mechanism to overcome this problem. Please refer to [link / reference]. Figure 2 The specific processing procedure is as follows:

[0058] S1: Initialize the classifier using labeled image samples. First, evaluate and record the initial classifier. The effect;

[0059] S2: Initialize the classifier using real data matching. Obtain pseudo-label samples, and then store the pseudo-label samples into the candidate dataset;

[0060] S3: Select a batch of data from the candidate dataset, train a new classifier using both labeled samples and samples from the effective pseudo-label dataset. Re-evaluate the new classifier Does the performance improve? If it does, store this batch of valid pseudo-label samples in the valid pseudo-label dataset; if it doesn't improve, check if there is still data in the candidate dataset. If there is data, input it into the new classifier. Repeat the training; if there is no improvement, the training ends.

[0061] Table 2 below shows the sample classification error rates under different experimental strategies.

[0062]

[0063] Table 2

[0064] Among them, D L In this context, ρ represents the imbalance rate of the labeled dataset, and D... U ρ in U CE represents the imbalance rate in the unlabeled dataset, CE represents the cross-entropy loss function used in the network model, and CE + 3xD ULDAM-DRW represents the initialization parameters generated by using the cross-entropy loss function in the network model and combining it with the semi-supervised learning model using 3 times the amount of unlabeled dataset in this invention. LDAM-DRW represents using the state-of-the-art LDAM-DRW loss function in the network model. LDAM-DRW+3xD U This represents the application of the semi-supervised learning framework of the present invention, which uses 3 times the amount of unlabeled dataset, to generate initialization parameters.

[0065] The numbers in the table represent the sample classification error rate under each classification strategy, as can be seen in LDAM-DRW+3xD. U When ρ is 5, the optimal classification performance achieves the lowest error rate of 7.73%, reducing the error rate from 11.64 to 7.73, a decrease of nearly 4 percentage points, which translates to an improvement in classification accuracy of nearly 4 percentage points. Furthermore, using the semi-supervised learning method of this invention to obtain initialization parameters has a significant effect on reducing the classification error rate under different loss function conditions.

[0066] Imbalanced sample semi-supervised learning has limitations, especially when the correlation between unlabeled data and the original data is significantly different.

[0067] For example, in the S2 natural scene image dataset mentioned earlier, the unlabeled data obtained may not belong to any of the original 5 classes (such as an image of a car). In this case, the extra information may have a great impact on training and results, which will reduce the relevance to the natural scenery images being studied.

[0068] Therefore, in situations where it is difficult to improve classification accuracy using semi-supervised learning models, we consider using the following classification method for imbalanced classes and samples with weak correlations between classes: a self-supervised learning framework. (See [link to relevant documentation]). Figure 3 .

[0069] S1: Use a self-supervised learning pretext task to learn feature representations of unlabeled images in the samples;

[0070] Specifically, this embodiment uses a self-supervised pretext task, such as image rotation, image mosaicking, or image coloring, to learn feature representations from a large number of unlabeled images. These tasks typically use an encoder-classifier network structure, where the encoder is used to extract image features and the classifier is used to predict the target of the pretext task.

[0071] S2: Use a self-supervised algorithm and contrastive learning method to train the samples and obtain a new self-supervised learning model and the initial feature information parameters of the samples. The self-supervised algorithm uses a linear classifier f(x) = sign(<θ,feature>+b), b>0, where b is the bias vector value and feature represents the sample data without labels.

[0072] In this embodiment, the samples are: 2000 images with low relevance to natural scenes obtained from the real large imbalanced dataset iNaturalist are randomly added to the sample library of the semi-supervised learning above.

[0073] The self-supervised algorithm and contrastive learning method chosen are the classic Rotation prediction and the contrastive learning method MoCo. In this process, in order to learn more label-independent initial feature information from the imbalanced dataset, as long as the initial weights are well generated in the self-supervised learning, the network model can benefit from the pre-training task and ultimately obtain a more general representation.

[0074] Let the linear classifier be f(x) = sign(<θ, feature> + b), b > 0, where b is the bias vector value and feature represents the unlabeled sample data. X is a d-dimensional Guassian mixture model. The label Y = +1 is positive and has a probability p. + The label negative class Y = -1 has a probability p - =1-p + Here we assume the negative class is the primary class, when p - When ≥0.5, and Random values ​​where β > 3. Set err. f =P(X,Y):P XY (f(X)≠Y) is used as the metric for classifier performance. In training, the "feature" in the formula represents the unlabeled raw data. During self-supervised learning, we consider using... k1, k2 > 0 are used as inputs to the linear classifier, where k1 is the weight penalty control factor and k2 is the initial weight bias value. Both are initially random numbers greater than 0 and are adjusted during training based on the training effect. This results in a new classifier f(X) = sign(-Z + b), where b is the bias vector value.

[0075] S3: Combine the new self-supervised learning model with the loss function network model, and use the initialized feature information parameters to generate a multi-objective self-supervised classifier and a multi-objective self-supervised learning model.

[0076] The learned label-independent initial feature representations are used for multi-label image classification tasks. This can be achieved by adding a multi-output classifier after the encoder, where the encoder can be either a ResNet-50 model using the LDAM-DRW loss function or by using a contrastive learning method such as MICLe.

[0077] Contrastive learning leverages the idea of ​​multi-instance learning, treating each image as a bag containing multiple instances (e.g., image patches of different scales or regions), and considering the label of each bag as the maximum value among the instances. A contrastive loss function is then used to train the model, minimizing the distance between bags with the same label and maximizing the distance between bags with different labels.

[0078] Table 3 below shows the classification error values ​​under different imbalance rates of the dataset and different basic training algorithms, with or without the use of the self-supervised learning (SSP) mode of this invention.

[0079]

[0080] Table 3

[0081] CE represents the initialization parameters without self-supervised learning, CE+SSP represents the initialization parameters with self-supervised learning, LDAM-DRW represents the initialization parameters using the LDAM-DRW model without self-supervised learning, and LDAM-DRW+SSP represents the initialization parameters combining self-supervised learning and LDAM-DRW+SSP. As shown in Table 3, when the imbalance ratio ρ is 5, the self-supervised learning SSP can significantly reduce the classification error rate (percentage) of the samples, especially when the loss function is LDAM-DRW. The experimental data clearly demonstrates that the self-supervised learning method of this invention can significantly improve the classification accuracy of imbalanced samples.

[0082] In summary, using the semi-supervised learning method and self-supervised learning method of this invention for multi-label image classification with imbalanced class distribution has the following advantages:

[0083] 1. It uses unlabeled data without relying on additional human-labeled data. Compared to existing multi-label image classification schemes, it makes fuller use of sample data and does not add an extra burden to data labeling.

[0084] 2. By employing self-supervised learning to explore the inherent properties of the data itself, the dependence of classification performance on the dataset is reduced. Unlike existing methods such as modifying the loss function, which are highly dependent on the dataset and have strong model correlation, this approach utilizes self-supervised learning to explore the inherent properties of the data, reducing the dependence of classification performance on the dataset.

[0085] 3. The solution is plug-and-play and will not structurally alter existing optimal multi-label image classification schemes. It first fully explores the inherent attributes of existing data to obtain a pre-trained parameter model, providing optimized initialization parameters for existing conventional advanced schemes. This invention is implemented before existing schemes, using the generated optimal initialization parameters as a pre-trained model to participate in subsequent existing schemes. It can be understood as first using this solution to generate pre-trained model parameters, and then freely choosing existing technical solutions to generate the final image classification network model.

[0086] 4. A simpler classifier metric was adopted: During the training process of semi-supervised and self-supervised learning, a simpler classifier was constructed based on different combinations of the mean and variance of the Guassian mixture model, which reduced the difficulty of constructing semi-supervised and self-supervised learning models.

[0087] This specification describes examples of embodiments of the invention, but does not imply that these embodiments illustrate or describe all possible forms of the invention. It should be understood that the embodiments in the specification can be implemented in various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or reduced to show details of specific components. The specific structural and functional details disclosed should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to implement the invention in various forms. Those skilled in the art will understand that multiple features illustrated and described with reference to any of the drawings can be combined with features illustrated in one or more other drawings to form embodiments not explicitly illustrated or described. The illustrated combinations of features provide representative embodiments for typical applications. However, various combinations and variations of features consistent with the teachings of the invention may be used as needed for specific applications or implementations.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-label image classification method for optimizing imbalanced category distribution, characterized in that, This includes a multi-label image classification method based on semi-supervised learning, which includes generating a new semi-supervised learning model, the steps of which include: Obtain the class-imbalanced original dataset and represent the class / class combination represented by each data point in the original dataset with a power label, wherein the power label is obtained by converting the label group from right to left from binary to decimal; The original dataset was divided into labeled and unlabeled datasets in a 1:3 ratio using a random algorithm, and pseudo-labeled data was generated from the data in the unlabeled dataset. The classifier used in the semi-supervised learning method is trained using the labeled dataset to obtain a semi-supervised classifier. The semi-supervised classifier is used to predict the image pseudo-labels of the pseudo-label data to obtain the pseudo-label dataset; the semi-supervised learning method uses different means and However, there are Gaussian mixture models with the same variance; the classifier is a Bayesian optimal classifier. The average of them is ; The labeled dataset and the pseudo-labeled dataset are used to minimize the loss function. θ is the average mean of the semi-supervised classifier, and a new semi-supervised classifier and a new semi-supervised learning model are obtained. The novel semi-supervised classifier is applied to multi-label image classification tasks with imbalanced class distributions. This is achieved by adding a multi-output sigmoid layer after the novel semi-supervised classifier, or by using a threshold function to convert the output of the novel semi-supervised classifier into multiple binary labels.

2. The multi-label image classification method for optimizing imbalanced category distribution according to claim 1, characterized in that, The semi-supervised learning method is used to enhance the generalization performance and confidence of the unlabeled data.

Images