A small sample classification method based on conversion network and self-supervision

Abstract

The present invention discloses a small sample classification method based on a conversion network and self-supervision. A conversion network module is added on the basis of a general classification model, different noises are added for feature enhancement, and feature embedding with discrimination and diversity is synthesized, so that the trained model can be better applied to small sample downstream tasks. Specifically, the following steps are included: obtaining an image data set for training a feature extractor and a conversion network module; sending the image data set into the network, using a feature enhancement method to obtain a feature embedding with discrimination and diversity, and combining self-supervised learning to train a feature extractor and a conversion network module, the optimization target is the sum of several cross entropy losses and KL divergences; obtaining a trained feature extractor and a conversion network module, and applying it to a small sample classification task. The present invention has good performance on four small sample classification task benchmarks (miniImageNet, tieredImageNet, CIFAR‑FS and Caltech‑UCSD), proving its effectiveness and superiority in performance.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a small sample classification method adding a conversion network and self-supervision. Background Art

[0002] Few-shot learning aims to identify the target category by learning only a small number of samples for each category. To accomplish this task, many existing methods use base classes to train models, each of which contains a large number of labeled samples, and then apply the trained model to the test task. Based on the data transferred from the base classes, existing few-shot learning methods can be roughly divided into three categories: meta-learning-based methods; metric learning-based methods; and data augmentation-based methods.

[0003] Meta-learning based methods attempt to learn a meta-learner that can adjust the optimization algorithm so that the meta-learner can quickly adapt to small sample tasks;

[0004] The metric learning-based method refers to learning a transferable distance metric function to evaluate the similarity between samples;

[0005] The data enhancement method refers to enhancing the data by using general image transformation techniques or generative adversarial networks. However, this method lacks the features required for small sample tasks, so its performance is always unsatisfactory.

[0006] The classification problem in small sample learning mainly refers to the C-way K-shot problem, which means: in the training stage, C categories are randomly selected from the training set, K samples of each category (a total of C×K data) are used as the support set input of the model, and then Q samples are extracted from the remaining data in these C categories as the query set of the model, that is, the model is required to learn how to distinguish these C categories from C×K data. Summary of the invention

[0007] The present invention proposes a small sample classification method that adds a conversion network and self-supervision to better adapt to small sample downstream tasks. This method adds a conversion network module, which consists of a pair of encoders and decoders, and the output is a synthesized feature embedding. This method uses a simple feature synthesis technology to perturb the feature space and synthesize feature embeddings with discrimination and diversity. This is achieved by allowing the synthesized feature embedding to be correctly classified into the category where the original feature embedding is located, and at the same time, it is divided into different subclasses according to the different interferences added. In addition, in the process of ensuring diversity, self-supervised learning is used. This is exactly the characteristic required for small sample tasks.

[0008] To achieve the above object, the technical solution of the present invention is:

[0009] A small sample classification method with a conversion network and self-supervision includes the following steps:

[0010] S1, obtain an image dataset for training a feature extractor and a conversion network module;

[0011] S2, feeding the image dataset into the network, using feature enhancement methods to obtain discriminative and diverse feature embeddings, and combining self-supervised learning to train feature extractors and conversion network modules;

[0012] S3. Use the trained model for small sample classification tasks.

[0013] Furthermore, in step S1, given the base class Where n is the total number of images in the dataset, x i and i Respectively represent the i-th image and its corresponding class label, y i ∈{1,...,C}, C represents the total number of categories, and each category contains multiple images.

[0014] Furthermore, step S2 specifically includes:

[0015] S21. When training a deep neural network, batch processing is used. First, a batch of image samples is randomly sampled from the image dataset. The batch size N bs given in advance;

[0016] S22. Send the batch image samples in B to the model composed of the backbone network and the classifier to obtain their predicted probabilities. The optimization goal of the model using cross-entropy (CE) loss is

[0017]

[0018] Among them, f and g represent the feature extractor and classifier respectively, Θ is the parameter set, and L ce represents the CE loss, R represents the regularization term of the parameter set, and λ is a hyperparameter.

[0019] S23. In order to ensure the distinguishability of the synthetic feature embeddings, they are fed into the classification network of the original visual features to make the predicted categories consistent with the categories to which the original visual features belong. The classification of synthetic feature embeddings is

[0020]

[0021] Where t is the number of additional synthetic feature embeddings, c j represents the feature of the jth Gaussian distribution noise, T represents the conversion network module, y ij is the synthetic feature T(f(xi ), c j ), which is consistent with the original visual feature f(x i ), Θ represents the parameter set of the entire model.

[0022] S24. In order to ensure the diversity of synthetic feature embedding, the features synthesized with different noises are classified into different subcategories. The original visual features and synthetic feature embeddings are sent to a classifier different from the above classifier, and the output is different categories.

[0023]

[0024] Among them, l ij are the self-supervised class labels manually annotated according to the different distributions of noise, and h represents the self-supervised classifier.

[0025] S25. Use real visual features to regularize synthetic feature embedding in label space so that synthetic feature embedding retains the inter-class relationship of real visual features.

[0026]

[0027] Where KL represents the Kullback Leibler divergence, x ij is category y i The real sample in f(x ij ) as a synthetic feature embedding T(f(x i ), c j ) supervisor, without optimization.

[0028] S26, the overall optimization goal is

[0029] L all =L1+L2+αL3+βL4

[0030] Among them, α and β are hyperparameters.

[0031] S27. According to the obtained total loss function, a stochastic gradient descent optimizer with momentum and a back propagation algorithm are used to train the deep neural network.

[0032] S28. Repeat steps S21 to S27 until the model converges.

[0033] Furthermore, step S3 specifically includes:

[0034] S31. Given a C-way K-shot classification task, the support set is S. For each support sample x u , first obtain the final feature representation through the feature extractor and conversion network module

[0035]

[0036] S32. Calculate the visual prototype of each category

[0037]

[0038] Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set.

[0039] S33. For the test sample x in the query set u , the probability that it belongs to category c is

[0040]

[0041] Where d is the similarity measurement function. Finally, the probability of the test sample belonging to N categories is used to predict which category it belongs to, and the category with the highest probability is the predicted category.

[0042] The small sample classification method of the present invention with the addition of a conversion network and self-supervision has the following advantages:

[0043] First, this method directly synthesizes visual features instead of input data, and introduces SSL supervision to ensure the discriminability and diversity of the synthesized feature embedding;

[0044] Second, this method demonstrates that synthetic feature embedding can provide additional patterns for feature representation, making the model better suited for downstream few-shot tasks;

[0045] The small sample classification method of the present invention, which adds a conversion network and self-supervision, performs well on four small sample classification task benchmarks (miniImageNet, tieredImageNet, CIFAR-FS and Caltech-UCSD), proving the effectiveness and superiority of the present invention in performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flow chart of the small sample classification method with the addition of conversion network and self-supervision of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] On the contrary, the present invention covers any alternatives, modifications, equivalents and arrangements falling within the spirit and scope of the present invention as defined by the claims.

[0049] refer to Figure 1 In a preferred embodiment of the present invention, a small sample classification method with a conversion network and self-supervision is added, comprising the following steps:

[0050] First, obtain the image dataset used to train the feature extractor and conversion network modules.

[0051] Specifically, given the base class Where n is the total number of images in the dataset, x i and i Respectively represent the i-th image and its corresponding class label, y i ∈{1,...,C}, C represents the total number of categories, and each category contains multiple images.

[0052] Then, the image dataset is fed into the network, and feature enhancement methods are used to obtain distinguishing and diverse feature embeddings, and self-supervised learning is combined to train the feature extractor and the conversion network module. The specific steps include the following:

[0053] The first step is to use batch processing when training deep neural networks. First, a batch of image samples is randomly sampled from the image dataset. The batch size N bs Given in advance.

[0054] In the second step, the batch image samples in B are fed into the model consisting of the backbone network and the classifier to obtain their predicted probabilities. The optimization goal of the model using cross entropy loss is

[0055]

[0056] Among them, f and g represent the feature extractor and classifier respectively, Θ is the parameter set, and L ce represents the CE loss, R represents the regularization term of the parameter set, and λ is a hyperparameter.

[0057] In the third step, in order to ensure the discriminability of the synthetic feature embeddings, they are fed into the classification network of the original visual features to make the predicted categories consistent with the categories to which the original visual features belong. The classification of synthetic feature embeddings is

[0058]

[0059] Where t is the number of additional synthetic feature embeddings, c j represents the feature of the jth Gaussian distribution noise, T represents the conversion network module, y ij is the synthetic feature T(f(x i ), c j ), which is consistent with the original visual feature f(x i), Θ represents the parameter set of the entire model.

[0060] In the fourth step, in order to ensure the diversity of synthetic feature embedding, the features synthesized with different noises are classified into different subcategories. The original visual features and synthetic feature embeddings are sent to a classifier different from the above classifier, and the output is different categories.

[0061]

[0062] Among them, l ij are the self-supervised class labels manually annotated according to the different distributions of noise, and h represents the self-supervised classifier.

[0063] The fifth step is to use the real visual features to regularize the synthetic feature embedding in the label space so that the synthetic feature embedding retains the inter-class relationship of the real visual features.

[0064]

[0065] Where KL represents the Kullback Leibler divergence, x ij is category y i The real sample in f(x ij ) as a synthetic feature embedding T(f(x i ), c j ) supervisor, without optimization.

[0066] In the sixth step, the overall optimization goal is

[0067] L all =L1+L2+αL3+βL4

[0068] Among them, α and β are hyperparameters.

[0069] Step 7: Based on the total loss function obtained, the stochastic gradient descent optimizer with momentum and the back propagation algorithm are used to train the deep neural network.

[0070] Finally, repeat the above steps until the model converges.

[0071] Finally, the trained model is used for the few-shot classification task.

[0072] Given a C-way K-shot classification task, the support set is S. For each support sample x u , first obtain the final feature representation through the feature extractor and conversion network module

[0073]

[0074] Then, calculate the visual prototype of each category

[0075]

[0076] Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set.

[0077] Furthermore, for the test sample x in the query set u , the probability that it belongs to category c is

[0078]

[0079] Where d is the similarity measurement function. Finally, the probability of the test sample belonging to N categories is used to predict which category it belongs to, and the category with the highest probability is the predicted category.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small sample classification method with a conversion network and self-supervision, characterized in that: The following steps are involved: S1, obtain an image dataset for training a feature extractor and a conversion network module; S2. Send the image dataset into the network, use the feature enhancement method to obtain discriminative and diverse feature embedding, and combine self-supervised learning to train the feature extractor and conversion network module, including: S21, batch processing is used during training. First, a batch of image samples is randomly sampled from the image dataset. The batch size N bs given in advance; S22. Send the batch image samples in B to the model composed of the backbone network and the classifier to obtain their predicted probabilities; the optimization goal of the model using cross entropy loss is Among them, f and g represent the feature extractor and classifier respectively, Θ is the parameter set, and L ce represents CE loss, R represents the regularization term of the parameter set, and λ is a hyperparameter; S23. In order to ensure the distinguishability of the synthetic feature embedding, they are sent to the classification network of the original visual features to make the predicted category consistent with the category to which the original visual features belong; the classification of the synthetic feature embedding is Where t is the number of additional synthetic feature embeddings, c j represents the feature of the jth Gaussian distribution noise, T represents the conversion network module, y ij is the synthetic feature T(f(x i ),c j ), which is consistent with the original visual feature f(x i ) have the same class label, Θ represents the parameter set of the entire model; S24. In order to ensure the diversity of synthetic feature embedding, the features synthesized with different noises are classified into different subcategories. The original visual features and synthetic features are embedded into a classifier different from the above classifier, and the output is different categories. Among them, l ij is the self-supervised class label manually annotated according to the different distributions of noise, and h represents the self-supervised classifier; S25. Use real visual features to regularize synthetic feature embedding in label space so that synthetic feature embedding retains the inter-class relationship of real visual features. Where KL represents the Kullback Leibler divergence, x ij is category y i The real sample in f(x ij ) as a synthetic feature embedding T(f(x i ),c j ) supervisor, no optimization is performed; S26, the overall optimization goal is L all =L1+L2+αL3+βL4 Among them, α and β are hyperparameters; S27. According to the obtained total loss function, a stochastic gradient descent optimizer with momentum and a back propagation algorithm are used to train the deep neural network. S28, repeat steps S21 to S27 until the model converges; S3. Use the trained model for small sample classification tasks.

2. The small sample classification method with a conversion network and self-supervision as claimed in claim 1, characterized in that: In step S1, given the base class Where n is the total number of images in the dataset, x i and i Respectively represent the i-th image and its corresponding class label, y i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple images.

3. The small sample classification method with a conversion network and self-supervision as claimed in any one of claims 1 to 2, characterized in that: Step S3 specifically includes: S31. Given a C-way K-shot classification task, the support set is S; for each support sample x u , first obtain the final feature representation through the feature extractor and conversion network module S32. Calculate the visual prototype of each category Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set; S33. For the test sample x in the query set u , the probability that it belongs to category c is Where d is the cosine similarity measurement function; finally, the probability of the test sample belonging to N categories is used to predict which category it belongs to, and the category with the highest probability is the predicted category.

Images