An unbalanced learning method for image classification model

Abstract

The application provides an unbalanced learning method for an image classification model, comprising: obtaining a training set of image classification with unbalanced sample quantity of each category; obtaining an image classification model constructed based on a neural network; repeatedly performing multiple iteration training on the model according to the following manner based on the initialized model to obtain a trained model: sampling a batch of sample images from the training set and inputting the current model; determining the classification result of the input sample image by using the current model, wherein the classification result comprises scores of each category corresponding to the image; determining whether the current training stage is an early stage or a late stage according to the current sampling number, and different configurations are performed on a score adjustment term and a reweighting term in a preset loss function in the early stage and the late stage respectively to obtain a loss function configured according to the training stage; and calculating the loss by using the loss function configured according to the training stage, according to the classification result of the current batch of sample images and the corresponding label, and updating the parameters of the current model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to the field of imbalanced learning in machine learning, and more specifically, to an imbalanced learning method for image classification models. Background Technology

[0002] In recent years, machine learning has achieved tremendous success thanks to carefully collected datasets, where the number of samples is balanced across classes. [1] However, real-world datasets are often imbalanced, meaning that only a few classes have a large number of samples (i.e., the majority class), while other classes have only a small number of samples (i.e., the minority class). [2] Because of this problem, naive empirical risk minimization (ERM) learning becomes biased towards the majority class, making generalization to the minority class more difficult. Therefore, the imbalanced learning problem has received increasing attention in recent years. [3] .

[0003] A simple yet effective approach to imbalanced learning is to modify the existing cross-entropy loss function so that the learning process focuses more on the minority classes. In this direction, existing methods typically fall into two categories: re-weighting. [4] Logits Adjustment (LA) [5][6][7] Reweighting methods allocate greater weights to the loss of a minority of classes. While this approach is intuitive, it can lead to optimization difficulties and instability. [4][5] To address this issue, reference [5] proposed a scheme called Deferred Re-Weighting (DRW), which applies the reweighting method only in the final stage of training. Score adjustment methods adjust the model score based on class sensitivity terms. For example, LDAM loss requires a large margin for the minority class to achieve strong regularization. [5] Losses in LA [6] The model score is adjusted using only additive terms. CDT loss. 0 The model score is adjusted using only the multiplication term. Recently, the above two terms were combined in the literature [7] to propose a unified vector scaling loss (VS) for imbalanced learning.

[0004] Although existing loss modification methods have achieved satisfactory performance, theoretical insights still need improvement. Specifically, reference [5] uses classical interval theory to explain the necessity of additive terms in LDAM loss, but this theory fails to explain the significant performance gain brought by DRW. Reference [6] analyzes the Fisher consistency property of additive terms in LA loss, but does not provide further generalization analysis. Reference [7] provides a generalization analysis of VS loss, but the results can only explain the role of multiplicative terms under the assumptions of linear models and linearly separable data. In addition, the inventors of this application have found that VS loss is quite incompatible with DRW, which is beyond the scope of existing theory. Therefore, there is still a gap between the theory and practice of loss modification methods, which leads to the need to improve the generalization of model classification.

[0005] The information from the above-mentioned documents is as follows:

[0006] [1]Russakovsky O,Deng J,Su H,et al.Imagenet large scale visualrecognition challenge[J].International journal of computer vision,2015,115:211-252.

[0007] [2]Van Horn G,Mac Aodha O,Song Y,et al.The inaturalist species classification and detection dataset[C] / / Proceedings of the IEEE conferenceon computer vision and pattern recognition.2018:8769-8778.

[0008] [3]Zhang Y, Kang B, Hooi B, et al. Deep long-tailed learning: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[0009] [4] Cui Y, Jia M, Lin TY, et al. Class-balanced loss based on effective number of samples[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:9268-9277.

[0010] [5] Cao K, Wei C, Gaidon A, et al. Learning imbalanced datasets with label-distribution-aware margin loss[J]. Advances in neural information processing systems, 2019, 32.

[0011] [6] Menon AK, Jayasumana S, Rawat AS, et al. Long-tail learning via logistic adjustment[C] / / International Conference on Learning Representations.

[0012] [7] Kini GR, Paraskevas O, Oymak S, et al. Label-imbalanced and group-sensitive classification under overparameterization[J]. Advances in Neural Information Processing Systems, 2021, 34:18970-18983.

[0013] [8] Ye HJ, Chen HY, Zhan DC, et al. Identifying and compensating for feature deviation in imbalanced deep learning[J]. arXiv preprint arXiv:2001.01385, 2020. Summary of the Invention

[0014] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide an imbalanced learning method for image classification models.

[0015] The objective of this invention is achieved through the following technical solution:

[0016] According to a first aspect of the present invention, an imbalanced learning method for an image classification model is provided, comprising: acquiring a training set for image classification with imbalanced sample sizes for each category, wherein the training set includes multiple sample images and labels, the labels indicating the ground truth values ​​of the categories corresponding to the sample images; acquiring an image classification model constructed based on a neural network, and initializing the parameters of the feature extractor and classifier in the model; based on the initialized model, repeatedly iterating the model in the following manner to obtain a trained model: sampling a batch of sample images from the training set and inputting them into the current model; extracting image features from the input sample images using the feature extractor in the current model, and determining the classification result of the input sample images based on the image features using the classifier in the current model, the classification result including the scores of the corresponding images in each category; determining whether the current training stage is an early stage or a late stage based on the current number of samplings, and configuring the score adjustment term and reweighting term in a preset loss function differently in the early stage and late stage respectively, to obtain a loss function configured according to the training stage; and using the loss function configured according to the training stage, calculating the loss based on the classification results and corresponding labels of the current batch of sample images, and updating the parameters of the current model.

[0017] Optionally, the preset loss function is expressed as:

[0018]

[0019] Where f(x) represents the classification result output by the image classification model f(·) for the input sample image x, y represents the label of the sample image x, and α y This represents the reweighted term, f(x). y Let f(·) represent the score output by f(·) for the sample image x under its class truth value, β y The multiplication score adjustment term, Δ, represents the true value of the category. y The additive score adjustment term represents the true value of the category; y′ represents a category, which is any one of the C categories that the image classification model wants to classify; β y′ Let f(x) represent the adjustment term for the multiplication score of category y′. y′ Δ represents the score output by f(·) for the sample image x in category y′. y′ This represents the addition score adjustment item for category y′.

[0020] Optionally, when the current number of samples is less than a first preset threshold, the current training phase is determined to be the early stage, and the score adjustment term and reweighting term are configured as follows:

[0021] The multiplication score adjustment item in the score adjustment items is configured as follows:

[0022]

[0023]

[0024] The addition score adjustment item in the score adjustment items is configured as follows:

[0025] Δ y =τlogπ y

[0026] Δ y′ =τlogπ y′

[0027] The reweighting term is configured as follows:

[0028] α y =1

[0029] Where, N y N represents the number of sample images corresponding to the ground truth value of the current sample image in the training set, N1 represents the number of sample images of the class with the largest number of sample images in the training set, and r represents r to the power of 0, where r>0. y′ τ represents the number of sample images under category y′ in the training set, τ>0, π y π represents the proportion of sample images in the training set corresponding to the ground truth value of the current sample image category. y′ This represents the proportion of sample images in category y′ in the training set.

[0030] Optionally, when the current number of samples is greater than or equal to a first preset threshold, the current training stage is determined to be the later stage, and the score adjustment term and reweighting term are configured as follows:

[0031] The multiplication score adjustment item in the score adjustment items is configured as follows:

[0032] β y =β y′ =1

[0033] The addition score adjustment item in the score adjustment items is configured as follows:

[0034] Δ y =τlogπ y

[0035] Δ y′ =τlogπ y′

[0036] The reweighting term is configured as follows:

[0037]

[0038]

[0039] Where ∝ represents proportional to, -v represents negative v to the power of 0, and v>0.

[0040] Optionally, when initializing the parameters of the feature extractor and classifier in the model, a random parameter initialization method is adopted. The sample images of the first batch are input into the initialized model, and the sample images of subsequent batches are input into the model after training and updating the parameters based on the sample images of the previous batch.

[0041] Optionally, after the sample images of the current batch are input into the model, the sub-loss corresponding to each sample image is calculated according to the loss function configured according to the training stage, and then the average is calculated to obtain the loss. The gradient is calculated based on the loss and backpropagation is used to update the parameters of the current model.

[0042] Optionally, the learning rate when updating the parameters of the current model is dynamically changed, set to 0.1 for training in batch [1, A], 0.01 for training in batch (A, B], and 0.001 for training in batch (B, C], where A < B < C.

[0043] According to a second aspect of the present invention, an image classification method is provided, the method comprising: acquiring an image to be classified; and inputting the image into a trained model obtained according to the method of the first aspect for image classification.

[0044] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the first or second aspect by executing the executable instructions. Attached Figure Description

[0045] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0046] Figure 1 This is a flowchart illustrating an imbalanced learning method for an image classification model according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0048] As mentioned in the background section, the inventors of this application discovered that VS loss is quite incompatible with DRW, which is beyond the scope of existing theory. Therefore, a gap remains between the theory and practice of loss modification methods, resulting in room for improvement in the generalization ability of model classification. To address this, this invention provides a systematic and detailed analysis of loss modification methods. After reviewing existing methods, the inventors found that in existing proofs, the classical Lipschitz continuity is the only property involved in the loss function. However, this property is global, and therefore the entire analysis cannot reveal how the loss function handles information from different categories. Inspired by this, the inventors employed a localization technique to extend the classical Lipschitz continuity. Through this method, the local Lipschitz constants on different categories precisely correspond to the category-related terms of the modified loss function. Based on this local property, the inventors proposed a new data correlation compression technique, thereby obtaining a fine-grained upper bound on the generalization error. Applying this upper bound on the generalization error to VS loss ultimately unlocked the theoretical properties of reweighting and score adjustment. Finally, based on the inventor's theoretical analysis, a theoretically sound imbalanced learning method (i.e., an imbalanced learning method for image classification models) is proposed. This method determines whether the current training stage is early or late based on the current number of samples. In the early and late stages, the score adjustment term and reweighting term in the preset loss function are configured differently to obtain a loss function configured according to the training stage. Then, using the loss function configured according to the training stage, the loss is calculated based on the classification results and corresponding labels of the sample images in the current batch, and the parameters of the current model are updated. As a result, the generalization performance of the image classification model can be improved.

[0049] According to an embodiment of the present invention, an imbalanced learning method for an image classification model is provided, comprising steps S1, S2, and S3. To better understand the present invention, each step is described in detail below with reference to specific embodiments.

[0050] Step S1: Obtain the training set for image classification with imbalanced sample sizes for each category, which includes multiple sample images and labels, where the labels indicate the ground truth values ​​of the corresponding categories for the sample images.

[0051] According to one embodiment of the present invention, the training set for image classification with imbalanced sample sizes (or following a long-tailed distribution) can be randomly sampled from image classification datasets that follow a long-tailed distribution. For example, the CIFAR10-LT dataset and the CIFAR100-LT dataset.

[0052] In addition, imbalanced data can be generated based on class-balanced datasets, i.e., training sets for image classification that follow a long-tailed distribution. To generate imbalanced data, consider resampling the original training set using two imbalanced distributions.

[0053] According to one embodiment of the present invention, long-tailed imbalance (LT) sampling is performed from an existing dataset (e.g., CIFAR10 or CIFAR100), where all sample images of one class are retained, and the number of sample images of each subsequent class decreases exponentially, thereby obtaining the training set. Let ρ = N1 / N C N represents the imbalance rate of the training set after resampling, and N1 represents the number of sample images in the category with the most sample images. C This represents the number of sample images in the category with the fewest sample images. For example, assuming ρ = 100, it means the number of sample images in the category with the most sample images is 100 times the number of sample images in the category with the fewest sample images. Taking the CIFAR10 dataset as an example, it includes sample images for ten categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Each category contains 5000 sample images for training. Assuming ρ = 100, during sampling, the 5000 sample images for airplane can be retained, and the number of sample images for each subsequent category decreases exponentially, eventually resulting in a ratio of 100 between airplane and truck sample images.

[0054] According to one embodiment of the present invention, step-imbalanced sampling is performed from an existing dataset (e.g., CIFAR10 or CIFAR100), wherein half of the sample images of each class are retained, and the other half of the sample images of each class are resampled to a certain proportion.

[0055] It should be understood that, in addition to the existing datasets mentioned above, implementers can also collect sample images and label them themselves to obtain datasets with other label spaces (the set of all label categories).

[0056] Step S2: Obtain an image classification model based on a neural network, and initialize the parameters of the feature extractor and classifier in the model.

[0057] According to one embodiment of the present invention, the image classification model can employ an existing neural network model, such as the ResNet model or the VGG model; or it can employ a user-defined neural network model. The image classification model includes a feature extractor and a classifier. The parameters of the feature extractor and classifier in the model are initialized using a random parameter initialization method.

[0058] Step S3: Based on the initialized model, repeatedly train the model iteratively multiple times in the following manner to obtain the trained model. Optionally, step S3 includes: S31, S32, S33, S34, and S35.

[0059] Step S31: Sample a batch of sample images from the training set and input them into the current model.

[0060] According to one embodiment of the present invention, the sample images of the first batch are input into the initialized model, and the sample images of subsequent batches are input into the model after the parameters are updated based on the sample images of the previous batch.

[0061] Step S32: Extract image features from the input sample image using the feature extractor in the current model, and determine the classification result of the input sample image based on the image features using the classifier in the current model. The classification result includes the score of the corresponding image in each category.

[0062] According to one embodiment of the present invention, taking a softmax classifier as the classifier and a subset extracted from the CIFAR10 dataset as the training set, the classifier will output the score (some literature also refers to confidence or probability) of the image in each of the ten categories: airplane, car, bird, cat, deer, dog, frog, horse, boat, and truck. The sum of the scores of the ten categories is 1.

[0063] Step S33: Determine whether the current training stage is the early stage or the late stage based on the current number of samples. In the early stage and the late stage, configure the score adjustment term and reweighting term in the preset loss function differently to obtain the loss function configured according to the training stage.

[0064] According to one embodiment of the present invention, the preset loss function is expressed as:

[0065]

[0066] Where f(x) represents the classification result output by the image classification model f(·) for the input sample image x, y represents the label of the sample image x, and α y This represents the reweighted term, f(x). y Let f(·) represent the score output by f(·) for the sample image x under its class truth value, βy The multiplication score adjustment term, Δ, represents the true value of the category. y The additive score adjustment term represents the true value of the category; y′ represents a category, which is any one of the C categories that the image classification model wants to classify; β y′ Let f(x) represent the adjustment term for the multiplication score of category y′. y′ Δ represents the score output by f(·) for the sample image x in category y′. y′ This represents the addition score adjustment item for category y′.

[0067] According to one embodiment of the present invention, when the current number of samples is less than a first preset threshold number, the current training phase is determined to be the early stage, and the score adjustment term and reweighting term are configured as follows:

[0068] The multiplication score adjustment item in the score adjustment items is configured as follows:

[0069]

[0070]

[0071] The addition score adjustment item in the score adjustment items is configured as follows:

[0072] Δ y =τlogπ y

[0073] Δ y′ =τlogπ y′

[0074] The reweighting term is configured as follows:

[0075] α y =1

[0076] Where, N y N represents the number of sample images corresponding to the ground truth value of the current sample image in the training set, N1 represents the number of sample images of the class with the largest number of sample images in the training set, and r represents r to the power of 0, where r>0. y′ τ represents the number of sample images under category y′ in the training set, τ>0, π y π represents the proportion of sample images in the training set corresponding to the ground truth value of the current sample image category. y′This represents the proportion of sample images in category y′ of the training set. Illustratively, if training is performed with a fixed total number of training iterations T, the first preset threshold can be approximately set to x%T, assuming x% is 80%. For example, when the total number of training iterations T is 200, the first preset threshold is set to 160; when the total number of training iterations T is 40000, the first preset threshold is set to 32000. Illustratively, r is set to, for example, 0.05, and τ is set to, for example, 0.75 or 1. It should be understood that this is only for illustration, and implementers can set these values ​​according to their needs.

[0077] According to one embodiment of the present invention, when the current number of samples is greater than or equal to a first preset number threshold, the current training stage is determined to be the later stage, and the score adjustment term and reweighting term are configured as follows:

[0078] The multiplication score adjustment item in the score adjustment items is configured as follows:

[0079] β y =β y′ =1

[0080] The addition score adjustment item in the score adjustment items is configured as follows:

[0081] Δ y =τlogπ y

[0082] Δ y′ =τlogπ y′

[0083] The reweighting term is configured as follows:

[0084]

[0085]

[0086] Where ∝ represents proportional to, -v represents negative v to the power of 0, and v>0. For illustrative purposes, v can be set to values ​​such as 0.25, 0.5, or 0.75.

[0087] Step S34: Using the loss function configured according to the training stage, calculate the loss based on the classification results and corresponding labels of the sample images in the current batch, and update the parameters of the current model.

[0088] The technical solution of this embodiment can achieve at least the following technical effects: This embodiment sets the multiplicative score adjustment term and reweighting term of the loss function in stages, and adopts truncated score adjustment (TLA, i.e., α) in the early stage. y =1) To avoid reweighting term α y The conflict between β and the multiplication score adjustment term, in this scheme yβ y′ Still in the early stages regarding π y It increments, but is truncated to 1 later in the training process to avoid conflicting with α. y Conflict; in addition, settings Make α y and Alignment It is a term in the upper bound of fine-grained generalization error, where π y Let α represent the proportion of sample images in the training set corresponding to the ground truth value of the current sample image. y Adopted and A similar approach can more effectively obtain a tighter upper bound on the generalization error, thereby better ensuring the model's generalization performance. In existing technical solutions, α y Since it is not designed based on the upper bound of the generalization error, it cannot guarantee better generalization performance; therefore, combining these improvements can improve the generalization performance of the model after imbalanced learning.

[0089] According to one embodiment of the present invention, after the sample images of the current batch are input into the model, the sub-loss corresponding to each sample image is calculated according to the loss function configured according to the training stage, and then the average is calculated to obtain the loss. The gradient is calculated based on the loss and the parameters of the current model are updated by backpropagation.

[0090] For example, the loss can be calculated using the following formula:

[0091]

[0092] in, Indicates batch Based on the m sample images, according to the loss function L VS (f(x),y) calculates the mean of the sub-loss corresponding to each sample image.

[0093] For example, the parameters are updated according to the following formula:

[0094]

[0095] Where Θ represents the model parameters and η represents the learning rate. This represents the gradient of the model parameters.

[0096] According to one embodiment of the present invention, the learning rate when updating the parameters of the model is dynamically changed, set to 0.1 for training in batch [1, A], 0.01 for training in batch (A, B], and 0.001 for training in batch (B, C], where A < B < C.

[0097] According to another embodiment of the present invention, the learning rate when updating the model parameters can also be fixed, with a fixed value defined by the implementer used in both the early and later stages.

[0098] Step S35: Determine whether the training requirements have been met. If yes, end the training; otherwise, proceed to step S31.

[0099] According to one embodiment of the present invention, the training requirement can be evaluated from the total number of training iterations. For example, it can be determined whether the number of sampling iterations has reached a second preset threshold. If so, the training ends; otherwise, proceed to step S31.

[0100] According to another embodiment of the present invention, the training requirement can be evaluated by whether the model has converged. For example, after the number of sampling exceeds a first preset number threshold, it is determined whether the model has been trained to convergence (e.g., whether the change in loss is less than or equal to a preset magnitude threshold several times in a row). If yes, the training ends; otherwise, proceed to step S31.

[0101] According to an example of the present invention, an illustrative algorithm flow is shown in Algorithm 1 below.

[0102] Algorithm 1: Illustrative Algorithm Flow

[0103]

[0104]

[0105] According to an embodiment of the present invention, an image classification method is provided, the method comprising: acquiring an image to be classified; inputting the image into a trained model obtained by an imbalanced learning method for image classification models as described in the foregoing embodiments for image classification. For example, the image can be input into the trained model, and the image can be output as a score in each category; or, the image can be output as the category with the highest score (e.g., horse), or the image can be output as the category with the highest score and the highest score (e.g., dog (90%)).

[0106] The imbalanced learning method for image classification models proposed in the aforementioned embodiments is a theoretically sound approach. This will be explained below through both theoretical explanations and comparative experiments.

[0107] (I) Explanation of the theoretical nature

[0108] This section aims to explain the theoretical properties of loss modification methods in imbalanced learning and propose a theoretically guaranteed imbalanced learning method. Specifically, the loss modification method optimizes a predefined loss function L having the form of the aforementioned embodiments. VS (f(x),y), which will not be elaborated here. The inventors noted that existing theories could not simultaneously explain α. y ,βy ,Δ y How to improve the generalization performance of model f, i.e., its performance on the test set; and how to explain some experimental phenomena, such as the effectiveness of Deferred Re-Weighting (DRW) and the incompatibility between VS loss and DRW. To address these issues, the inventors employ a localization technique to extend the classical Lipschitz continuity and further construct a fine-grained upper bound on the generalization error, describing α in a unified manner. y ,β y ,Δ y The inventors explained the theoretical properties of the loss modifier and the experimental phenomena described above. Based on this upper bound, they proposed a theoretically sound imbalance learning method that improves the performance of the loss modification method.

[0109] This invention constructs a fine-grained upper bound for the generalization error of loss modification methods, and based on this, proposes an algorithm to improve the generalization performance of loss modification methods, comprising the following steps:

[0110] S1, (Analysis of generalization error in unbalanced learning induced by data correlation compression) A localization technique is used to extend the classical Lipschitz continuity. Based on this, a new data correlation compression technique is proposed to obtain a fine-grained upper bound on the generalization error.

[0111] S2, (Upper bound of generalization error for VS loss) The fine-grained upper bound of generalization error is applied to VS loss to analyze the theoretical properties of loss modification methods and explain the experimental results of DRW and VS loss.

[0112] S3, (Generalization Property-Induced Imbalanced Learning Algorithm): Based on the theoretical properties of loss modification methods, a theoretically guaranteed imbalanced learning method is proposed.

[0113] Indicatively, in this invention, step S1 includes:

[0114] S11. Define the fundamental lemma of generalization analysis. Specifically, given a set of functions... and loss function in, Let C be the set of real numbers, C be the number of classes, and M be the upper bound of the loss function. Then, for any δ∈(0,1), in the dataset... There exists at least a 1-δ probability such that for any The following equation holds true:

[0115]

[0116] in, For the model's generalization performance; π C :=N C / N,N C The number of samples in the category with the fewest samples, and N is the total number of samples; For the model on the training set Error on; For describing the set of functions The measure of complexity generally includes The left side of the inequality represents the risk of f on the balanced test set, and the goal of this invention is to minimize this term. The right side of the inequality is an upper bound on the left side; in other words, minimizing the right side also minimizes the left side. In the right side, Φ(L,δ) is a term that can generally be minimized, where... To mitigate the risks associated with imbalanced training sets, This is a constant term that decays rapidly as the number of training set samples N increases; while This is an item that needs attention. Generally, we analyze its decay rate with respect to the number of samples and the number of categories. The following invention focuses on the upper bound of this item.

[0117] S12. Define local Lipschitz continuity and data-related compression techniques. Specifically, if the loss function L(f,y) satisfies local Lipschitz continuity and has a constant... Then for any have:

[0118] |L(f,y)-L(f′,y)|≤μ y ·‖f(x)-f′(x)‖,

[0119] Where, ‖·‖ represents the 2-modulus of the vector, Indicates training set A subset of the sample labels (i.e., the labels of the sample images) where all samples are y; L(f,y) represents the output of the loss function L given the model f and the labels y of the sample images, i.e., l VS (f(x),y); L(f′,y) represents the output of the loss function L given the model f′ and the labels y of the sample images, where f′, like f, belongs to the set f(x) represents the score output by function f given sample x, and f′(x) represents the score output by function f′ given sample x. Furthermore, if the loss function L(f,y) satisfies local Lipschitz continuity and has a constant... Then the following inequality holds:

[0120]

[0121] Where, π y This indicates that the y-th class sample is in the training set. The proportion of the total number.

[0122] S13. Combining the fundamental lemma of generalization analysis in S11 and the data correlation compression technique in S12, we obtain the upper bound of the fine-grained generalization error. Specifically, given a set of functions... And the loss function L for local Lipschitz continuity, then for any δ∈(0,1), in the dataset There exists at least a 1-δ probability such that for any The following equation holds true:

[0123]

[0124] Where, π C This indicates that the C-class sample is in the training set. The proportion of the total number.

[0125] In this invention, step S2 includes:

[0126] S21. Analyze the local Lipschitz continuity of the VS loss. Specifically, the VS loss is locally Lipschitz continuous and has a constant.

[0127]

[0128] in, softmax(·) is the Softmax function, B y (f) is a positive real number that is positively correlated with the model's performance on the y-th class of the training set. α y Indicates the reweighted term, Δ y β represents the adjustment term for the addition score. y The multiplicative score adjustment term representing the true value of the category, β y′ This represents the adjustment term for the multiplication score of category y′.

[0129] S22. Incorporate the local Lipschitz continuity of the VS loss into the fine-grained upper bound of the generalization error in S13. Specifically, given a set of functions... and VS loss L VS Then for any δ∈(0,1), in the training set There exists at least a 1-δ probability such that for any The following equation holds true:

[0130]

[0131] in, This represents the model's generalization performance on a balanced dataset. The final theoretical results of this invention, and the following theoretical properties, are all derived from the analysis of the second term on the right-hand side of this inequality.

[0132] S23. Based on the upper bound of the fine-grained generalization error of VS loss in S22, analyze the theoretical properties of the loss modification method and explain the experimental results of DRW and VS loss. Specifically, the following theoretical properties exist:

[0133] The reason why reweighting and score adjustments are necessary is that: and B y (f) The generalization bound is also imbalanced across categories. This is achieved by defining the generalization bound for categories with different generalization bounds. and B y (f) assigns different weights to the categories; reweighting and score adjustment can both result in tighter generalization bounds. In this process, α y The main improvement was a rebalancing of generalization performance between categories, i.e. And β y and Δ y Then focus on adjusting the B between categories y The imbalance of (f).

[0134] The reason why DRW is necessary is that, as pointed out in references [4] and [5], weighted minority classes lead to optimization difficulties and instability, especially when the distribution is extremely imbalanced. To solve this problem, reference [5] proposes a DRW method that sets α at the beginning and end of the training phases respectively. y =1 and While this method brings significant performance gains, it still lacks a theoretical explanation. The fine-grained upper bound of the generalization error in the VS loss of S22 can provide some theoretical explanation. Specifically, although the weighted loss can improve the model's performance on the minority class, it is detrimental to further improvement on the performance of the majority class, leading to a lower B in the majority class. y (f) is relatively small. Therefore, although weighting the minority class can achieve balance in the initial stage of training. This item, but may lead to a more unbalanced B. y (f) thus fails to improve the generalization bound. In contrast, in DRW, α is set at the initial stage of training. y =1, encouraging the model to focus on the majority class, thus achieving a more balanced B after weighting the minority class. y (f) thus yields a tighter generalization bound, thereby explaining the effectiveness of DRW.

[0135] Are reweighting and score adjustment fully compatible? On the one hand, they are not fully compatible, due to the reweighting term α. y It's about π y Decreasing, while the multiplication score adjustment term β y It's about π y Increasing. Therefore, Will weaken α y The effect. On the other hand, α yAdjustment item Δ for addition score y They are compatible because both terms can induce tighter generalization bounds.

[0136] In this invention, step S2, based on the theoretical properties of the loss modification method obtained in S23, improves the existing loss modification method and proposes the following algorithm, which mainly includes three points: First, the comprehensive use of reweighting, score adjustment, and DRW is crucial for improving generalization performance. Second, the inventors propose truncated score adjustment (TLA) to avoid α. y and β y The conflict between them. In this scheme, β y It will still be relevant to π in the initial stages of training. y It increments, but is truncated to 1 at the end of the training phase. Third, set... Make α y and Alignment is performed and named Aligned DRW (ADRW). The entire learning process is as described in Algorithm 1 above.

[0137] (II) Comparative Experiment

[0138] The inventors conducted comparative experiments on two image classification datasets, CIFAR10 and CIFAR100. These datasets contain 10 and 100 classes respectively, with the original datasets containing 50,000 training samples and 10,000 validation samples. To generate imbalanced data, two types of imbalanced distributions were considered for resampling the original training set: long-tailed imbalance (LT), where the number of training samples in each class decays exponentially; and step imbalance (Step), where half of the training samples in each class are resampled to a certain proportion. Let ρ = P1 / P C This represents the training set imbalance rate after resampling. For each class of imbalanced data, it reports the average accuracy with 5 random seeds when ρ=100. Comparison methods include: CE. [8] LDAM [5] VS [7] and its relationship with DRW [5] Furthermore, ResNet-32 was used as the backbone network and trained for 200 epochs using stochastic gradient descent, with an initial learning rate of 0.1, which decayed to 0.01 and 0.001 in epochs 160 and 180, respectively.

[0139] The literature [8] is: Ye HJ, Chen HY, Zhan DC, et al. Identifying and compensating for feature deviation in imbalanced deep learning [J]. arXiv preprint arXiv: 2001.01385, 2020.

[0140] After the experiment, the model accuracy under different losses on the CIFAR-10 and CIFAR-100 datasets is shown in Table 1. Rows 3-8 correspond to the six existing methods, and rows 9-12 correspond to the method of this invention. It can be found that: (1) the method proposed in this invention is better than the existing methods compared, which verifies the effectiveness of the proposed method; (2) VS and DRW have better performance, that is, both reweighting and score adjustment can bring performance gains, which is consistent with the theory of this invention; (3) VS+DRW has worse performance than VS, that is, reweighting and multiplicative score adjustment are incompatible, which is also consistent with the theory of this invention.

[0141] Table 1

[0142]

[0143]

[0144] Note: VS+TLA+ADRW means using a preset loss function (VS) + truncation logic for the early and late stages (TLA, i.e., the early stage α). y =1, later β y =β y′ =1)+α y Set as (ADRW). As can be seen, using VS+TLA+ADRW simultaneously yields the best results, with the underlined part being the next best.

[0145] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0146] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0147] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0148] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An imbalanced learning method for image classification models, characterized in that, include: Obtain an image classification training set with imbalanced sample sizes for each category, which includes multiple sample images and labels, where the labels indicate the ground truth values ​​for the corresponding categories of the sample images; Obtain an image classification model based on a neural network, and initialize the parameters of the feature extractor and classifier in the model; Based on the initialized model, the model is repeatedly trained iteratively in the following manner to obtain the trained model: Sample a batch of image samples from the training set and input them into the current model; The current model uses a feature extractor to extract image features from the input sample image, and uses a classifier in the current model to determine the classification result of the input sample image based on the image features. The classification result includes the score of the corresponding image in each category. The current training phase (early or late) is determined based on the number of samples taken. Different configurations are applied to the score adjustment and reweighting terms in the preset loss function for the early and late phases, respectively, resulting in a loss function configured according to the training phase. The preset loss function is expressed as follows: in, Image classification model For the input sample image The output classification results, Represents sample image The tag, Indicates a reweighted term. express For sample images The score output under its category truth value, The multiplication score adjustment term represents the true value of the category. Additive score adjustment item representing the true value of the category; This refers to a category, that category The image classification model is responsible for classifying the data. Any one of the categories; Indicates category Multiplication score adjustment item express For sample images In category The output score, Indicates category Adjustment items for addition scores; Using the loss function configured according to the training stage, the loss is calculated based on the classification results and corresponding labels of the sample images in the current batch, and the parameters of the current model are updated.

2. The method according to claim 1, characterized in that, When the current number of samples is less than a first preset threshold, the current training phase is determined to be the early stage, and the score adjustment term and reweighting term are configured as follows: The multiplication score adjustment item in the score adjustment items is configured as follows: The addition score adjustment item in the score adjustment items is configured as follows: The reweighting term is configured as follows: in, This represents the number of sample images in the training set corresponding to the ground truth value of the current sample image's category. This represents the number of sample images in the category with the most sample images in the training set. express Power of 1 , Indicates the categories in the training set The number of sample images below , This represents the proportion of sample images in the training set that correspond to the ground truth value of the current sample image's category. Indicates the categories in the training set The proportion of sample images below.

3. The method according to claim 2, characterized in that, When the current number of samples is greater than or equal to a first preset threshold, the current training stage is determined to be the later stage, and the score adjustment term and reweighting term are configured as follows: The multiplication score adjustment item in the score adjustment items is configured as follows: The addition score adjustment item in the score adjustment items is configured as follows: The reweighting term is configured as follows: in, Indicates proportional to, Indicates negative Power of 1 .

4. The method according to any one of claims 1-3, characterized in that, When initializing the parameters of the feature extractor and classifier in the model, a random parameter initialization method is adopted. The sample images of the first batch are input into the initialized model, and the sample images of subsequent batches are input into the model after the parameters are updated based on the sample images of the previous batch.

5. The method according to claim 4, characterized in that, After the sample images of the current batch are input into the model, the sub-loss corresponding to each sample image is calculated according to the loss function configured according to the training stage, and then the average is calculated to obtain the loss. The gradient is calculated based on the loss and the parameters of the current model are updated by backpropagation.

6. The method according to claim 4, characterized in that, The learning rate is dynamically adjusted when updating the parameters of the current model. It is set to 0.1 for training in batch [1, A], 0.01 for training in batch (A, B], and 0.01 for training in batch (B, C]. 0.001, where A B C.

7. An image classification method, characterized in that, The method includes: Obtain the image to be classified; The image is input into a trained model obtained by the method according to any one of claims 1-6 for image classification.

8. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

Images