Active-class incremental image classification method and device for achieving class-balanced sampling

Abstract

The invention discloses an active-class incremental image classification method and device for achieving class-balanced sampling, and relates to the field of image classification for incremental tasks. The invention solves the problem that the existing class-incremental image classification model based on active learning for unlabeled image collections exhibits serious class imbalance when performing sample selection. The present invention adopts the improved greedy sampling algorithm designed by the present invention to select a batch of valuable samples, so that the distribution obeyed by the selected samples is close to the distribution of all samples, thereby achieving class balance in sampling. The improved greedy sampling algorithm determines whether each sample is worthy of selection by calculating the degree of similarity between the distribution of the selected samples and the distribution of all samples in the cluster. The present invention is mainly used to achieve image classification for incremental tasks.

Description

Technical Field

[0001] The present invention relates to the field of image classification for incremental tasks. Background Art

[0002] The incremental class image classification problem is a machine learning problem that aims to ensure that a model can effectively classify all known categories while continuously receiving data of new categories. In this problem, training and test data are systematically divided into several "incremental tasks," and the model learns on each task sequentially. Each task contains training images of several categories, and the category space of the training images of any two tasks does not overlap. When the model is training on task t, the training data from tasks 1 to t-1 cannot be used. After the model is trained on task t, it is expected to be able to classify images of all categories from tasks 1 to t.

[0003] In the prior art, the training data for each task in the class-incremental image classification problem consists of a large number of labeled images. For details, see application number 202311772256.X, titled "A Small-Sample Class-Incremental Image Classification Method Based on Cue Word Fine-Tuning and Feature Replay." To reduce the labeling cost of training data for each task, the active-class incremental image classification problem is proposed. In this problem, the training images for each task are initially unlabeled. The goal is to use the model to select the most valuable training images and feed them back to humans for labeling. The model is then trained using the human-labeled images.

[0004] However, when applying existing active learning methods (such as Entropy and CoreSet) to class-incremental image classification models based on deep learning models and fine-tuning visual cues to select samples from unlabeled image collections, the selected samples exhibit severe class imbalance. Specifically, class-incremental image classification models, such as L2P or DualPromp models, exhibit severe class imbalance. Furthermore, using these class-imbalanced samples to train classification models can compromise model performance, a problem that urgently needs to be addressed. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem that the existing unlabeled image collection presents serious class imbalance when selecting samples in the class-incremental image classification model based on active learning; the present invention provides an active-class-incremental image classification method and device for achieving class-balanced sampling.

[0006] An active-class incremental image classification method with balanced class sampling is implemented, the method comprising the following steps:

[0007] S1. Obtain unlabeled training sets for all tasks. The unlabeled training sets corresponding to each task include unlabeled images of multiple categories. Each image is used as a sample, and the image categories of different tasks do not overlap.

[0008] S2: The class-incremental image classification model loads the given model parameters and locks them. The initial value of t is 1. At this time, the prompt word parameters of the class-incremental image classification model are randomly initialized under the current t-th task.

[0009] The incremental image classification model consists of a deep learning model and a visual cue word fine-tuning unit, where the model parameters are used to constrain the deep learning model and the cue word parameters are used to constrain the visual cue word fine-tuning unit.

[0010] S3. Use the deep learning model to extract the visual features of each sample in the unlabeled training set corresponding to the current task to obtain a feature set;

[0011] S4. Use the clustering algorithm to cluster the samples in the feature set into several clusters, and use the improved greedy sampling algorithm to select valuable samples from each cluster until all clusters are traversed. The selection principle of the improved greedy sampling algorithm is: determine the number of samples K to be selected in each cluster c , and ensure that the selected K c The Gaussian distribution of the samples is related to the K c The KL divergence between the Gaussian distributions of all samples in the cluster where the sample is located is the smallest; among them, the selected K c All samples are regarded as valuable samples; K c is the number of samples selected from the cth cluster;

[0012] S5. Determine whether the total number of valuable samples selected corresponding to the current t-th task exceeds the preset total number of samples B. If the result is yes, randomly retain B valuable samples, and after the B valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6. If the result is no, after all the selected valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6.

[0013] S6. Let t = t + 1, and determine whether t is the last task. If yes, complete the training of the class-incremental image classification model and execute step S7. If no, use the prompt word parameters of the class-incremental image classification model of the previous task to initialize the prompt word parameters of the current task and execute step S3.

[0014] S7. Use the trained class-incremental image classification model to classify the image to be classified.

[0015] Preferably, in step S4, the number of samples K selected from each cluster is determined c The implementation is:

[0016]

[0017] Among them, M c is the cth cluster G c The total number of samples contained in N t is the total number of samples in the unlabeled training set corresponding to the t-th task, This is a round-up operation.

[0018] Preferably, in step S4, K is selected c The implementation of the sample is:

[0019] Step S41: Assume that the cth cluster G c All samples in follow Gaussian distribution

[0020] Among them, μ c Cluster G c The mean vector of is a real number space of dimension D×1, Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ;

[0021] i is cluster G c The eigenvector corresponding to the i-th sample in , D is an integer, M c Cluster G c The total number of samples included in ;

[0022] Step S42: From cluster G c The implementation method of selecting the first sample is:

[0023]

[0024] in, Cluster G c The corresponding set of valuable samples, The initial state is an empty set, f selected1 for The first sample in ; ∪ means taking the union;

[0025] Step S43: From cluster G c After selecting the mth sample, add the sample to In, until The number of samples reaches K c ; m=2,3……K c ;

[0026] The implementation method of selecting the mth sample is:

[0027]

[0028] Among them, f j is the difference set The feature vector of the jth sample in ;

[0029] for The Gaussian distribution obeyed by all samples in the constructed set;

[0030] D KL (·|·) means calculating two distributions and KL divergence between selectedm for The mth sample in .

[0031] Preferably, in step S41,

[0032]

[0033] in, Gaussian distribution The d-th row and d-th column element on the diagonal of the covariance diagonal matrix, f id and μ cd f i and μ c The dth row element in .

[0034] Preferably,

[0035]

[0036] in,

[0037] Gaussian distribution The mean vector of Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ;

[0038] Gaussian distribution The d-th row and d-th column elements on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row and d-th column elements on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row element of the mean vector, μ cd Gaussian distribution The d-th row element of the mean vector.

[0039] Preferably, the clustering algorithm is implemented using the K-means algorithm.

[0040] Preferably, the class-incremental image classification model is an L2P model or a DualPrompt model.

[0041] Preferably, the deep learning model is a deep learning model based on an attention mechanism.

[0042] Preferably, the given model parameters are parameters that have been trained for the deep learning model.

[0043] An active-class incremental image classification device that implements class-balanced sampling includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the active-class incremental image classification method that implements class-balanced sampling.

[0044] The beneficial effects brought by the present invention are:

[0045] In a given task, a deep learning model with given model parameters is first used to extract features for each unlabeled sample. Secondly, a clustering algorithm is used to cluster these features into several clusters, each cluster including feature vectors of multiple samples. For each cluster, the improved greedy sampling algorithm designed by the present invention is used to select a batch of samples. The improved greedy sampling algorithm determines whether each sample is worthy of being selected by calculating the distribution of the selected samples and the similarity of the distribution of all samples in the cluster. After the samples of each group are selected, the selected samples of each group are collected as the final set of selected samples. They are manually annotated with category labels, and the visual cue word fine-tuning unit in the class incremental image classification model is trained under the condition of locking the model parameters to determine the cue word parameters.

[0046] The present invention ensures that the KL divergence between the Gaussian distribution obeyed by the selected samples and the Gaussian distribution obeyed by all samples in the cluster is minimized by improving the greedy sampling algorithm, so that the distribution obeyed by the selected samples is close to the distribution of all samples, thereby achieving category balance in sampling.

[0047] The class balance of training samples affects the performance of the trained model. A more balanced class of training samples leads to higher performance, and vice versa. An improved greedy sampling algorithm can effectively select class-balanced samples for model training, thereby improving the model's classification capabilities. This method is applicable to incremental learning methods such as pre-training the Vision Transformer (ViT) and visual prompt tuning, and is suitable for cases where the overall class balance of unlabeled datasets is present. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the active-class incremental image classification method for achieving class-balanced sampling according to the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0051] During the conception of this invention, we proposed an active-class incremental image classification method that achieves class-balanced sampling. This method can effectively select class-balanced samples for model training in each task, thereby improving the model's classification ability. This method is applicable to the incremental learning method of the deep learning model based on the attention mechanism (Transformer, abbreviated as Viusal Prompt Tuning) and is suitable for situations where the unlabeled dataset is overall class-balanced.

[0052] Specifically, in a given task, the deep learning model in the pre-trained class incremental image classification model is first used to extract features for each unlabeled sample. Secondly, the K-means algorithm is used to cluster these features into several "groups", namely: clusters. For each group, the improved greedy sampling algorithm designed by the present invention is used to select a batch of samples. The greedy sampling algorithm determines whether each sample is worthy of being selected by calculating the distribution of the selected samples and the degree of similarity of the distribution of all samples in the group. After the samples of each group are selected, the selected samples of each group are collected as the final set of selected samples.

[0053] Based on the above inventive concept, the specific implementation methods provided by the present invention are as follows:

[0054] Specific implementation method 1. Figure 1 This embodiment describes an active-class incremental image classification method for achieving class-balanced sampling, which includes the following steps:

[0055] S1. Obtain unlabeled training sets for all tasks. The unlabeled training sets corresponding to each task include unlabeled images of multiple categories. Each image is used as a sample, and the image categories of different tasks do not overlap.

[0056] Specifically, the samples in the unlabeled training set corresponding to each task can be real-world data or business data in a production environment, or public images.

[0057] S2: The class-incremental image classification model loads the given model parameters and locks them. The initial value of t is 1. At this time, the prompt word parameters of the class-incremental image classification model are randomly initialized under the current t-th task.

[0058] The incremental image classification model is a deep learning model based on learnable visual cue words. It is specifically composed of a deep learning model and visual cue word fine-tuning. The model parameters are used to constrain the deep learning model, and the cue word parameters are used to constrain the visual cue word fine-tuning.

[0059] In specific applications, the incremental image classification model can be an L2P model or a DualPrompt model.

[0060] As an example, the deep learning model may be a deep learning model based on an attention mechanism (ViT).

[0061] S3. Use the deep learning model to extract the visual features of each sample in the unlabeled training set corresponding to the current task to obtain a feature set;

[0062] S4. Use the clustering algorithm to cluster the samples in the feature set into several clusters, and use the improved greedy sampling algorithm to select valuable samples from each cluster until all clusters are traversed. The selection principle of the improved greedy sampling algorithm is: determine the number of samples K to be selected in each cluster c , and ensure that the selected K c The Gaussian distribution of the samples is related to the K c The KL divergence between the Gaussian distributions of all samples in the cluster where the sample is located is the smallest; among them, the selected K c All samples are regarded as valuable samples; K c is the number of samples selected from the cth cluster;

[0063] The clustering algorithm clusters samples into several clusters. The number of categories in each cluster is unknown, so it provides an effective data basis for improving the greedy sampling algorithm to balance the selection of valuable samples.

[0064] The above steps first determine the number of samples K selected in each cluster c , and then use the improved greedy sampling algorithm to determine whether each sample is worth selecting by calculating the similarity between the distribution of the selected samples and the distribution of all samples in the cluster.

[0065] Empirically, K c The value of G c The ratio of the number of samples in the training set to the total number of samples in the unlabeled training set under the current task is determined; specifically, as an optimal way to determine the number of samples K selected in each cluster c The implementation is:

[0066]

[0067] Among them, M c is the cth cluster G c The total number of samples contained in N t is the total number of samples in the unlabeled training set corresponding to the t-th task, This is a round-up operation.

[0068] Furthermore, as a preferred method, in step S4, K is selected c The implementation of the sample is:

[0069] Step S41: Assume that the cth cluster G c All samples in follow Gaussian distribution

[0070] Among them, μ c Cluster G c The mean vector of is a real number space of dimension D×1, Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ;

[0071] f i Cluster G c The eigenvector corresponding to the i-th sample in , D is an integer, M c Cluster G c The total number of samples included in ;

[0072] Furthermore, μ is given in detail. c and The specific expression of

[0073]

[0074] in, Gaussian distribution The d-th row and d-th column element on the diagonal of the covariance diagonal matrix, f id and μ cd f i and μ c The dth row element in .

[0075] In this preferred method, μ c and The specific expression of effectively describes the distribution of samples with less computational effort.

[0076] Step S42: From cluster G c The implementation method of selecting the first sample is:

[0077]

[0078] in, Cluster G c The corresponding set of valuable samples, The initial state is an empty set, F selected1 for The first sample in ; ∪ means taking the union;

[0079] Step S43: From cluster G c After selecting the mth sample, add the sample to In, until The number of samples reaches K c ; m=2,3……K c ;

[0080] The implementation method of selecting the mth sample is:

[0081]

[0082] Among them, f j is the difference set The feature vector of the jth sample in {f j} is due to f j The set of components;

[0083] for The Gaussian distribution obeyed by all samples in the constructed set;

[0084] D KL (·|·) means calculating two distributions and KL divergence between selectedm for The mth sample in .

[0085] The degree of similarity between the distribution of the selected samples and the distribution of all samples in the cluster is used to determine whether each sample is worthy of being selected.

[0086] By ensuring that the KL divergence between the Gaussian distribution of the selected sample and the Gaussian distribution of the entire cluster is minimized, the distribution of the selected sample can be made close to the distribution of the entire cluster, thereby achieving class balance in the sampling. This method is applicable to situations where the unlabeled dataset exhibits class balance as a whole.

[0087] More specific Defined as a Gaussian distribution And gives the Gaussian distribution The specific expression of:

[0088]

[0089] Among them, Gaussian distribution Likewise The Gaussian distribution obeyed by all samples in the constructed set;

[0090] Gaussian distribution The mean vector of Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ;

[0091] Gaussian distribution The d-th row and d-th column elements on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row and d-th column elements on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row element of the mean vector, μ cd Gaussian distribution The d-th row element of the mean vector.

[0092] The above process specifically gives two distributions and The specific expression of the KL divergence between the two distributions accurately calculates the difference between them.

[0093] S5. Determine whether the total number of valuable samples selected corresponding to the current t-th task exceeds the preset total number of samples B. If the result is yes, randomly retain B valuable samples, and after the B valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6. If the result is no, after all the selected valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6.

[0094] Specifically, the process of labeling the categories of all the selected valuable samples is to manually label them and then return them to the model;

[0095] S6. Let t = t + 1, and determine whether t is the last task. If yes, complete the training of the class-incremental image classification model and execute step S7. If no, use the prompt word parameters of the class-incremental image classification model of the previous task to initialize the prompt word parameters of the current task and execute step S3.

[0096] S7. Use the trained class-incremental image classification model to classify the image to be classified.

[0097] In specific applications, the clustering algorithm is implemented using the K-means algorithm, and the given model parameters are the parameters trained for the deep learning model.

[0098] The process of training the class-incremental image classification model using the labeled samples can be achieved through existing technology, and the given model parameters are locked during the training process, wherein the given model parameters are parameters trained for the deep learning model. The process of training the class-incremental image classification model using the labeled samples is mainly to train the visual cue word fine-tuning unit in the class-incremental image classification model. Since the given model parameters have been locked, it mainly trains the cue word parameters that determine the deep learning model.

[0099] The active-class incremental image classification method for achieving class-balanced sampling described in this embodiment can effectively select class-balanced samples for model training through an improved greedy sampling algorithm, thereby improving the classification ability of the model. This method is suitable for the class-incremental learning method of deep learning models and visual prompt word fine-tuning (Viusal Prompt Tuning) after training, and is suitable for situations where the unlabeled data set presents a class balance as a whole. An improved greedy sampling algorithm is used to select a batch of samples. The improved greedy sampling algorithm determines whether each sample is worthy of being selected by calculating the distribution of the selected samples and the degree of similarity of the distribution of all samples in this cluster. This avoids the phenomenon that the samples selected by the model present a serious class imbalance.

[0100] Specific embodiment 2: An active-class incremental image classification device that implements class-balanced sampling includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the active-class incremental image classification method that implements class-balanced sampling as described above.

[0101] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. An active-class incremental image classification method for achieving class-balanced sampling, characterized in that: The method comprises the following steps: S1. Obtain unlabeled training sets for all tasks. The unlabeled training sets corresponding to each task include unlabeled images of multiple categories. Each image is used as a sample, and the image categories of different tasks do not overlap. S2: The class-incremental image classification model loads the given model parameters and locks them. The initial value of t is 1. At this time, the prompt word parameters of the class-incremental image classification model are randomly initialized under the current t-th task. The incremental image classification model consists of a deep learning model and a visual cue word fine-tuning unit, where the model parameters are used to constrain the deep learning model and the cue word parameters are used to constrain the visual cue word fine-tuning unit. S3. Use the deep learning model to extract the visual features of each sample in the unlabeled training set corresponding to the current task to obtain a feature set; S4. Use the clustering algorithm to cluster the samples in the feature set into several clusters, and use the improved greedy sampling algorithm to select valuable samples from each cluster until all clusters are traversed. The selection principle of the improved greedy sampling algorithm is: determine the number of samples K to be selected in each cluster c , and ensure that the selected K c The Gaussian distribution of the samples is related to the K c The KL divergence between the Gaussian distributions of all samples in the cluster where the sample is located is the smallest; among them, the selected K c All samples are regarded as valuable samples; K c is the number of samples selected from the cth cluster; S5. Determine whether the total number of valuable samples selected corresponding to the current t-th task exceeds the preset total number of samples B. If the result is yes, randomly retain B valuable samples, and after the B valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6. If the result is no, after all the selected valuable samples are labeled by category, continue to lock the given model parameters, and use the labeled samples to train the visual cue word fine-tuning unit in the class incremental image classification model, so as to determine the cue word parameters of the class incremental image classification model, and execute step S6. S6. Let t = t + 1, and determine whether t is the last task. If yes, complete the training of the class-incremental image classification model and execute step S7. If no, use the prompt word parameters of the class-incremental image classification model of the previous task to initialize the prompt word parameters of the current task and execute step S3. S7. Use the trained class-incremental image classification model to classify the image to be classified.

2. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: In step S4, the number of samples K selected from each cluster is determined c The implementation is: Among them, M c is the cth cluster G c The total number of samples contained in N t is the total number of samples in the unlabeled training set corresponding to the t-th task, This is a round-up operation.

3. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: In step S4, K is selected c The implementation of the sample is: Step S41: Assume that the cth cluster G c All samples in the y-order Gaussian distribution Among them, μ c Cluster G c The mean vector of is a real number space of dimension D×1, Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ; f i Cluster G c The eigenvector corresponding to the i-th sample in , D is an integer, M c Cluster G c The total number of samples included in ; Step S42: From cluster G c The implementation method of selecting the first sample is: in, Cluster G c The corresponding set of valuable samples, The initial state is an empty set, f selected1 for The first sample in ; ∪ means taking the union; Step S43: From cluster G c After selecting the mth sample, add the sample to In, until The number of samples reaches K c ; m=2,3……K c ; The implementation method of selecting the mth sample is: Among them, f j is the difference set The feature vector of the jth sample in ; for The Gaussian distribution obeyed by all samples in the constructed set; D KL (·|·) means calculating two distributions and KL divergence between selectedm for The mth sample in .

4. The active-class incremental image classification method for achieving class-balanced sampling according to claim 3, characterized in that: In step S41, in, Gaussian distribution The d-th row and d-th column element on the diagonal of the covariance diagonal matrix, f id and μ cd f i and μ c The dth row element in .

5. The active-class incremental image classification method for achieving class-balanced sampling according to claim 3, characterized in that: in, Gaussian distribution The mean vector of Gaussian distribution The diagonal element vector of the covariance diagonal matrix of ; Gaussian distribution The d-th row and d-th column element on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row and d-th column element on the diagonal of the covariance diagonal matrix, Gaussian distribution The d-th row element of the mean vector, μ cd Gaussian distribution The d-th row element of the mean vector.

6. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: The clustering algorithm is implemented using the K-means algorithm.

7. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: The class-incremental image classification model is either the L2P model or the DualPrompt model.

8. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: The deep learning model is a deep learning model based on the attention mechanism.

9. The active-class incremental image classification method for achieving class-balanced sampling according to claim 1, characterized in that: The given model parameters are the parameters that have been trained for the deep learning model.

10. An active-class incremental image classification apparatus for achieving class-balanced sampling, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, wherein: The processor executes the computer program to implement the active-class incremental image classification method for achieving class-balanced sampling according to any one of claims 1 to 9.

Images