A small sample classification method based on twin knowledge distillation and self-supervised learning

Abstract

The present invention discloses a small sample classification (small sample) method based on twin knowledge distillation and self-supervised learning, which combines twin knowledge distillation with self-supervised learning for downstream small sample classification tasks. Specifically, the following steps are included: obtaining an image dataset for training a twin knowledge distillation model; using the image dataset to train a predefined twin knowledge distillation network from scratch, and the optimization target is a loss function that combines twin knowledge distillation with self-supervision; obtaining a trained feature extractor F, and applying it to a small sample classification task. The present invention has performed well on three small sample classification task bases (miniImageNet, tieredImageNet, and CIFAR‑FS), proving the effectiveness and superiority of the present invention in performance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to knowledge distillation technology and small sample classification tasks. Background Art

[0002] Few-shot learning (Few-shot learning for short) means that after a machine learning model has learned a large amount of data in a certain category, it only needs a small number of samples to quickly learn new categories. The classification problem in few-shot learning mainly refers to the N-way K-shot problem, which means: in the training phase, N categories are randomly selected from the training set, K samples of each category (a total of N×K data) are used as the support set input of the model, and then Q samples are extracted from the remaining data in these N categories as the problem set of the model, that is, the model is required to learn how to distinguish these N categories from N×K data.

[0003] Current small sample methods can be divided into three categories: gradient-based methods; data augmentation methods; and metric-based methods.

[0004] Gradient-based methods mainly refer to meta-learning-based methods, which means that in the training phase, the model is trained by learning a series of N-way K-shot problems, and then the model is used to solve new small sample tasks.

[0005] Data augmentation methods address the scarcity of datasets by generating additional datasets of the target category. These methods use a modified model learned from the base class or directly use adversarial generative networks to generate data from a small number of samples of the target category.

[0006] The metric-based method is the best method in small sample tasks. It aims to learn a distance metric function that can compare the similarities between different samples. The solution of the present invention is proposed based on this method. Summary of the invention

[0007] The present invention aims to improve the quality of feature representation extracted by the feature extractor, thereby improving the performance of the model on small sample downstream classification tasks, while not increasing the complexity of the network. It mainly retains the distinguishing information between feature representations during training to smooth the feature representation so that the model can obtain better generalization ability on the target category.

[0008] To achieve this goal, the present invention proposes a twin knowledge distillation network, which regularizes the classification prediction of a pair of input samples in the same category through the same network to keep them consistent, and adds self-supervised learning on this basis, thereby enhancing the generalization of the feature extractor. At the same time, the model pays more attention to difficult samples, further improving the generalization ability of the model.

[0009] The technical solution adopted by the present invention is:

[0010] A small sample classification method based on twin knowledge distillation and self-supervised learning includes the following steps:

[0011] S1. Obtain an image dataset for training the twin knowledge distillation network model and testing small sample classification tasks;

[0012] S2. Establish a twin knowledge distillation network model consisting of a feature extraction network and a classifier network, and train the twin knowledge distillation network model using a method combining twin knowledge distillation and self-supervised learning;

[0013] S3. Apply the trained twin knowledge distillation network model to the small sample classification task.

[0014] Furthermore, in step S1, the image dataset obtained for training the twin knowledge distillation network model includes C base base classes, each of which includes a large number of labeled samples that meet the preset data volume; the image datasets obtained for testing small sample classification tasks include C target test categories, each of which includes a small number of samples; and the base class and the test category do not intersect in the label space,

[0015] Furthermore, step S2 specifically includes:

[0016] S21. Randomly sample a batch of image samples from the image dataset used to train the twin knowledge distillation network. The batch size N bs given in advance;

[0017] S22. For a sample x in the base class, its category prediction vector is obtained through the feature extraction network and the classifier network, that is, z = [z 1 , ..., z i , ..., z m ], and then pass it through the softmax classifier to get the predicted probability that it belongs to the i-th class

[0018]

[0019] Among them, τ>0 is the temperature scaling parameter, which controls the smoothness of the normalized probability distribution of the softmax output; M is the number of categories;

[0020] For multi-classification tasks, the objective loss function is

[0021] L CE (x) = H(y, P(x))

[0022] Where H represents the cross entropy loss, P(x) is the predicted probability distribution of input x, and y is the true class label;

[0023] S23. Take a pair of samples of the same category as the input of the twin knowledge distillation network model to obtain their respective probability distributions P S (x i ) and P S (x j ), let P T (x)=(P S (x i )+P S (x j )) / 2 as a teacher, to distill the two student distributions respectively; at the same time, the two student probability distributions are also supervised by the true label of the category

[0024] L TwinKd (x i , x j )=H(αy+βP T (x), P S (x i ))+H(αy+βP T (x), P S (x j )),

[0025] Among them, α and β are two hyperparameters that balance teacher supervision and category true label supervision, and α+β=1; P T (x) No gradient propagation is performed;

[0026] S24. Use 2D image rotation in self-supervised learning to enhance data. The rotation angles are R = {90°, 180°, 270°}. Since the sample itself can be regarded as 0° rotation, the combined rotation set can be expressed as R′ = {0°, 90°, 180°, 270°}. For an input sample, first create its three rotated images. in Indicates that the sample x i Rotate by angle r; then use feature extractor F to extract their feature representations, and then regularize the feature representations of the input sample pairs and their enhanced samples

[0027]

[0028] The classifier network G maps the feature representation into the label space to predict the label, and then distills the knowledge from the average probability value to the rotated sample; the corresponding self-supervised knowledge distillation is defined as

[0029]

[0030] Where P T (x) is the predicted probability P of the input sample S (x i ) and P S (x j )’s mean;

[0031] At the same time, the logit vector G(F(x)) extracted from the input sample and its rotation sample is used by the rotation classifier to predict their rotation angle label r; therefore, the self-supervised loss formula is

[0032]

[0033] Where Q(x) = [q 1 (x), ..., q R (x)] is the rotation prediction probability vector, which is obtained by

[0034]

[0035] Where u represents the sample x rotation vector;

[0036] In summary, the entire optimization goal of this network is

[0037]

[0038] S26. According to the obtained optimization target, use a stochastic gradient descent optimizer with momentum and a back propagation algorithm to train the twin knowledge distillation network model;

[0039] S27. Repeat steps S21 to S26 until the model converges.

[0040] Furthermore, step S3 specifically includes:

[0041] S31. Given an N-way K-shot classification task, the support set is S. First, the visual prototypes of each category are calculated through the feature extraction network.

[0042]

[0043] Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set;

[0044] S32. For the test sample x in the query set t , the probability that it belongs to category c is

[0045]

[0046] Where d is the similarity measurement function; finally, the probability of the test sample belonging to N categories is used to predict which category it belongs to, and the category with the highest probability is the predicted category.

[0047] The small sample classification method based on twin knowledge distillation and self-supervised learning of the present invention has the following advantages:

[0048] First, an effective and easy-to-implement framework TwinKD is proposed to solve small sample tasks. TwinKD adds KL divergence to the optimization objective and distills input image pairs of the same category through the same network to produce consistent feature representations, so that the probability distribution of each sample can be smoothed by the same sample to obtain better generalization. The advantage of TwinKD is that it does not require a pre-trained model, so it is more efficient and saves more computing resources.

[0049] Secondly, it is theoretically proven that TwinKD pays more attention to difficult samples and learns richer feature representations by enhancing difficult sample data, which is reflected in the addition of SSL to the TwinKD framework. In this way, the model will pay more attention to difficult samples, further improving the generalization ability of the model, and it is more flexible and efficient than the existing KD.

[0050] The small sample classification method based on twin knowledge distillation and self-supervised learning of the present invention has performed well on three small sample classification task benchmarks (miniImageNet, tieredImageNet and CIFAR-FS), proving the effectiveness and superiority of the present invention in performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the small sample classification method based on twin knowledge distillation and self-supervised learning of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0054] refer to Figure 1In a preferred embodiment of the present invention, a small sample classification method based on twin knowledge distillation and self-supervised learning includes the following steps:

[0055] First, obtain the image dataset for training the twin knowledge distillation network model and testing the small sample classification task.

[0056] Among them, the datasets obtained for training the twin knowledge distillation network model total C base There are C base classes, each of which has a large number of labeled samples. target There are only a few samples in each test category. The base class and the test category are disjoint in the label space, that is, The specific symbol is expressed as {(x 1 ,y 1 ), (x 2 ,y 2 ),...,(x N ,y N )}, where N is the total number of images in the dataset, x i and i Respectively represent the i-th image and its corresponding class label, y i ∈{1,...,M}, M represents the total number of categories.

[0057] Then, a twin knowledge distillation network model consisting of a feature extraction network and a classifier network is established, and the twin knowledge distillation network model is trained using a method combining twin knowledge distillation with self-supervised learning. Specifically, the following steps are included:

[0058] The first step is to use batch processing when training deep neural networks. First, a batch of image samples is randomly sampled from the image dataset. The batch size N bs Given in advance.

[0059] In the second step, for a sample x in the base class, its logit vector is obtained through the feature extraction network and the fully connected layer, that is, z = [z 1 , ..., z i , ..., z m ], and then pass it through the softmax classifier to get the predicted probability that it belongs to the i-th class

[0060]

[0061] Among them, τ>0 is the temperature scaling parameter, which controls the smoothness of the normalized probability distribution of the softmax output; M is the number of categories.

[0062] For classification tasks, the objective loss function is

[0063] L CE (x) = H(y, P(x))

[0064] Where H represents the cross entropy loss, P(x) is the predicted probability distribution of input x, and y is the true class label.

[0065] In the third step, the present invention uses the twin knowledge distillation mode (TwinKd for short) to distill the same category sample pairs input into the same model. Specifically, the present invention only trains one model, takes a pair of samples of the same category as input, and then passes them through the same network to obtain their respective probability distributions P S (x i ) and P S (x j ), let P T (x)=(P S (x i )+P S (x j )) / 2 as a teacher to distill the two student distributions. Of course, at the same time, the two student probability distributions must also be supervised by the true label of the category

[0066] L TwinKd (x i , x j )=H(αy+βP T (x), P S (x i ))+H(ay+βP T (x), P S (x j ))

[0067] Among them, α and β are two hyperparameters that balance teacher supervision and class true label supervision, and α+β=1. T (x) No gradient propagation is performed.

[0068] Step 4: Fix the hyperparameter β and optimize the target L TwinKd For the logit vector z i The gradient of

[0069]

[0070] By deduction, we can get

[0071]

[0072] where γ i Represents the probability of incorrect classification.

[0073] When the sample x i When it is a sample that is difficult to classify, that is, γ iBig and Small, then the scaling factor Large means that the model will pay more attention to difficult samples, and vice versa.

[0074] In the fifth step, the present invention uses a self-supervised learning (SSL) strategy to perform data enhancement on difficult samples, that is, adding SSL to TwinKd to diversify the feature representation.

[0075] The data is augmented using 2D image rotation in SSL, with rotation angles R = {90°, 180°, 270°}. Since the sample itself can be regarded as a 0° rotation, the combined rotation set can be expressed as R′ = {0°, 90°, 180°, 270°}. For an input sample, first create its three rotated images in Indicates that the sample x i Rotate by angle r. Then use feature extractor F to extract their feature representations, and then regularize the feature representations of the input sample pairs and their enhanced samples

[0076]

[0077] The classifier network G maps the feature representation into the label space to predict the label, and then distills the knowledge from the average probability value to the rotated samples. The corresponding self-supervised knowledge distillation is defined as

[0078]

[0079] Where P T (x) is the predicted probability P of the input sample S (x i ) and P S (x j )’s mean.

[0080] At the same time, the logit vector G(F(x)) extracted from the input sample and its rotation sample is used by the rotation classifier to predict their rotation angle label r. Therefore, the self-supervised loss formula is

[0081]

[0082] Where Q(x) = [q 1 (x), ..., q R (x)] is the rotation prediction probability vector, which is obtained by

[0083]

[0084] Where u represents the sample x rotation vector.

[0085] In summary, the entire optimization goal of this network is

[0086]

[0087] In the sixth step, based on the obtained total loss function, the stochastic gradient descent optimizer with momentum and the back-propagation algorithm are used to train the deep neural network.

[0088] Finally, repeat the above steps until the model converges.

[0089] Finally, the trained twin knowledge distillation network model is applied to the small sample classification task.

[0090] Given an N-way K-shot classification task, the support set is S. First, the visual prototypes of each category are calculated through the feature extraction network.

[0091]

[0092] Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set.

[0093] For the test sample x in the problem set t , the probability that it belongs to category c is

[0094]

[0095] Where d is a similarity measurement function, and the cosine similarity function is used in the present invention. Finally, the category to which the test sample belongs is predicted based on the probability of the test sample belonging to the N categories, and the category with the highest probability is the predicted category.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small sample classification method based on twin knowledge distillation and self-supervised learning, characterized in that: The following steps are involved: S1. Obtain an image dataset for training the twin knowledge distillation network model and testing small sample classification tasks; S2. Establishing a twin knowledge distillation network model consisting of a feature extraction network and a classifier network, and training the twin knowledge distillation network model using a method combining twin knowledge distillation with self-supervised learning; specifically including: S21. Randomly sample a batch of image samples from the image dataset used to train the twin knowledge distillation network. The batch size N bs given in advance; S22. For a sample x in the base class, its category prediction vector is obtained through the feature extraction network and the classifier network, that is, z = [z1,…,z i ,…,z m ], and then pass it through the softmax classifier to get the predicted probability that it belongs to the i-th class Among them, τ>0 is the temperature scaling parameter, which controls the smoothness of the normalized probability distribution of the softmax output; M is the number of categories; For multi-classification tasks, the objective loss function is L CE (x)=H(y,P(x)) Where H represents the cross entropy loss, P(x) is the predicted probability distribution of input x, and y is the true class label; S23. Take a pair of samples of the same category as the input of the twin knowledge distillation network model to obtain their respective probability distributions P S (x i ) and P S (x j ), let P T (x)=(P S (x i )+P S (x j )) / 2 as a teacher to distill the two student distributions respectively; at the same time, the two student probability distributions are also supervised by the true label of the category L TwinKd (x i ,x j )=H(αy+βP T (x),P S (x i ))+H(αy+βP T (x),P S (x j )), Among them, α and β are two hyperparameters that balance teacher supervision and category true label supervision, and α+β=1; P T (x) No gradient propagation is performed; S24. Use 2D image rotation in self-supervised learning to enhance the data. The rotation angles are R = {90°, 180°, 270°}. Since the sample itself can be regarded as 0° rotation, the combined rotation set can be expressed as R ′ ={0°,90°,180°,270°}; For an input sample, first create its three rotated images in Indicates that the sample x i Rotate by angle r; then use feature extractor F to extract their feature representations, and then regularize the feature representations of the input sample pairs and their enhanced samples The classifier network G maps the feature representation into the label space to predict the label, and then distills the knowledge from the average probability value to the rotated sample; the corresponding self-supervised knowledge distillation is defined as Where P T (x) is the predicted probability P of the input sample S (x i ) and P S (x j )’s mean; At the same time, the logit vector G(F(x)) extracted from the input sample and its rotation sample is used by the rotation classifier to predict their rotation angle label r; therefore, the self-supervised loss formula is where Q(x) = [q1(x),…,q R (x)] is the rotation prediction probability vector, which is obtained by Where u represents the sample x rotation vector; In summary, the entire optimization goal of this network is S26. According to the obtained optimization target, use a stochastic gradient descent optimizer with momentum and a back propagation algorithm to train the twin knowledge distillation network model; S27, repeat steps S21 to S26 until the model converges; S3. Apply the trained twin knowledge distillation network model to the small sample classification task.

2. The small sample classification method based on twin knowledge distillation and self-supervised learning according to claim 1, characterized in that: In step S1, the image dataset obtained for training the twin knowledge distillation network model includes C base base classes, each of which includes a large number of labeled samples that meet the preset data volume; the image datasets obtained for testing small sample classification tasks include C target test categories, each of which includes a small number of samples; and the base class and the test category do not intersect in the label space, 3. The small sample classification method based on twin knowledge distillation and self-supervised learning according to any one of claims 1-2, characterized in that: Step S3 specifically includes: S31. Given an N-way K-shot classification task, the support set is S. First, the visual prototypes of each category are calculated through the feature extraction network. Among them, c represents a category, S c and |S c | is the support set of category c and the number of samples in the support set; S32. For the test sample x in the query set t , the probability that it belongs to category c is Where d is the similarity measurement function; finally, the probability of the test sample belonging to N categories is used to predict which category it belongs to, and the category with the highest probability is the predicted category.

Images