Knowledge distillation method and device for small sample learning, and storage medium

Abstract

The application provides a kind of small sample learning-oriented knowledge distillation method, device and storage medium, relating to the computer vision application field of deep learning.The application specifically includes three kinds of methods:distillation method based on auxiliary classifier;Distillation method based on image local features;Distillation method based on image internal relationship.New knowledge distillation method is limited in small sample learning task for traditional distillation, starting from the calculation method of distillation loss to improve, use image intermediate layer features or the output of auxiliary classifier to assist the knowledge transfer between teacher and student models, more suitable for the characteristics of small sample learning containing a large number of random tasks.The application applies new distillation method to small sample learning task, and obtains competitive experimental results on related data sets.In particular, in the distillation experiment using light weight setting, the improvement effect of the proposed method on the performance of student model is more significant.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision application of deep learning, and particularly relates to a knowledge distillation method and device for small sample learning and a storage medium. BACKGROUND

[0002] The knowledge distillation method is based on a "teacher-student model" training paradigm, and a pre-trained teacher model is used to assist in training a new student model. The application of the knowledge distillation technology only requires that the teacher and the student can produce similar outputs, and there is no mandatory requirement for the specific internal structure of the two. According to the different ways of generating the student model, the knowledge distillation can be roughly divided into two categories: "self-distillation" and "lightweight distillation". In the self-distillation task, the student model adopts the same network structure as the teacher model, and at this time, it is required that the performance of the student model after training can exceed that of the teacher model. In the lightweight distillation task, the parameter size of the student model is much smaller than that of the teacher model, and at this time, it is not mandatory to require that the performance of the student model after training exceeds that of the teacher, as long as it is better than the control performance of the teacher without distillation training, which can prove the effectiveness of the distillation method.

[0003] The traditional knowledge distillation method has limitations in dealing with small sample learning tasks, which may lead to instability caused by small sample sub-tasks containing only a small number of categories and random combination of category order to the distillation task. SUMMARY

[0004] The present application relates to the field of computer vision application of deep learning, and particularly relates to a knowledge distillation method and device for small sample learning and a storage medium.

[0005] In the first aspect, a knowledge distillation method for small sample learning is provided, and the steps are as follows:

[0006] Step (1): The small sample image dataset is divided into training set, validation set and test set which do not overlap in category space;

[0007] Step (2): Pre-train the teacher model based on the conventional classification task. Perform channel pruning on the teacher model network to obtain a lightweight student model;

[0008] Step (3): Small sample training is performed in random sub-task mode, and each time only a fixed number of categories are extracted to construct an "N-way K-shot" small sample classification task, and the extracted image samples are divided into support set and query set;

[0009] Step (4): After the image samples in each task are uniformly preprocessed, two groups of image features are obtained through the feature extraction networks of the teacher and the student model, respectively;

[0010] Step (5): The student model completes the small sample metric classification based on its own output, and calculates the classification loss;

[0011] Step (6): According to the selected distillation method, the distillation loss is calculated based on the outputs of the teacher model and the student model;

[0012] Step (7): The weighted sum of the classification loss and the distillation loss is used as the overall backpropagation update for the student model.

[0013] In further implementation details, the dataset used in step (1) is mainly MiniImageNet and TieredImageNet. The MiniImageNet dataset contains 100 image classes, each containing 600 images, of which 64 classes are used for training, 16 classes are used for validation during training, and another 20 classes are used for final testing. The TieredImageNet dataset has a relatively larger data capacity, with 351, 97, and 160 image classes in the training-validation-testing sets, respectively.

[0014] In further implementation details, step (2) uses channel pruning on the complete feature extraction network (deep convolutional neural network structure) used by the teacher model. Except for the initial input and final output channels, the number of I / O channels at each layer is reduced by 50%, and the overall network parameter quantity is reduced to about 1 / 3 of the original teacher model. The pruned lightweight network parameters are reinitialized and used as the student model for training from scratch. In this invention, a twelve-layer residual network (Resnet12) is used as the feature extraction network of the teacher model, and the lightweight network obtained by channel pruning on the basis of the lightweight network (denoted as Slim-Resnet12) is used as the feature extraction network of the student model.

[0015] In further implementation details, step (3) constructs the "N-way K-shot" small sample subtask, which extracts a fixed number of classes and a fixed number of image samples for each class each time. The specific settings are as follows: for the 5way-5shot subtask, 5 classes are extracted, each containing 1 support set sample and 15 query set samples; for the 5way-5shot subtask, 5 classes are extracted, each containing 5 support set samples and 15 query set samples. The class combination contained in each subtask is completely random.

[0016] In further implementation details, in step (4), the preprocessing of the image includes uniform scaling to 84*84 pixels, random data augmentation transformation, and standardization based on the statistical prior of the pixel mean and variance of the dataset. The feature extraction networks of the teacher model and the student model are both deep neural network structures composed of convolution-pooling-activation function layers. According to whether the feature extraction network contains a global pooling layer at the end, the obtained features can be divided into global features (each image produces a unique feature vector) or local features (each image produces a batch of n>1 feature vectors).

[0017] In further implementation details, in step (5), after the student model extracts the local features of the image, the local features are input to the small sample metric classifier. The features from the support set samples are used to represent the feature centers of the classes contained in the current subtask, and the metric classifier judges the similarity between each query set sample and each feature center, and the most similar class is taken as the predicted classification result of the query set sample. The small sample metric classifier used in the present application is a deep nearest neighbor neural network (abbreviated as DN4), which is a classifier based on image local features.

[0018] In further implementation details, the distillation loss calculation of step (6) is different from the traditional distillation method which directly constrains the KL divergence of the small sample classification probability. The new small sample distillation method uses the intermediate layer features (or the relationship between features) extracted from the image or the fixed classification probability generated by the auxiliary classifier for distillation constraint. Thus, the instability caused by the random combination of the small sample subtask containing only a small number of classes and the class order is avoided, providing a relatively constant update direction for the training of the student model. The specific calculation method of the distillation loss of each method is as follows:

[0019] Auxiliary classifier-based distillation: after the student model obtains the local features, in addition to completing the small sample metric classification, the features are passed through a global mean pooling layer and an additional auxiliary linear classifier to obtain fixed classification probabilities S in the class space of the entire training set. The same dimension classification probability T generated by the teacher model is calculated according to the KL divergence to obtain the distillation loss. As shown in the following formula (1):

[0020] (1)

[0021] Image local feature-based distillation: the teacher model and the student model obtain two groups of local features for the same sample, each containing n feature vectors of the same dimension. The KL divergence is calculated between the feature vectors corresponding to the same position, and then summed to obtain the final distillation loss, as shown in the following formula (2):

[0022] (2)

[0023] Distillation based on intra-image relationship: after the teacher model and the student model obtain two sets of local features respectively for the same sample, cosine similarity is calculated between each pair of features, and two n*n similarity matrices are obtained, denoted as R t and R s . The mean square error of the two similarity matrices is used as the distillation loss, as shown in the following formula (3):

[0024] (3)

[0025] In a second aspect, a knowledge distillation device for small sample learning is provided, and the knowledge distillation device comprises:

[0026] A first module is configured to divide a small sample image dataset into a training set, a validation set and a test set which do not overlap in a class space;

[0027] A second module is configured to pre-train a teacher model based on a conventional classification task, and perform channel pruning on the teacher model network to obtain a lightweight student model;

[0028] A third module is configured to perform small sample training in a random sub-task mode, and each time a fixed number of classes are extracted to construct an N-way K-shot small sample classification task, and the extracted image samples are divided into a support set and a query set;

[0029] A fourth module is configured to obtain two sets of image features through the feature extraction networks of the teacher and the student model after the image samples included in each task are uniformly preprocessed;

[0030] A fifth module is configured to complete small sample metric classification based on the output of the student model, and calculate a classification loss;

[0031] A sixth module is configured to calculate a distillation loss based on the outputs of the teacher model and the student model according to a selected distillation method;

[0032] A seventh module is configured to perform weighted summation of the classification loss and the distillation loss, and update the student model as a whole.

[0033] In a third aspect, a computer readable storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction makes an electronic device execute the operations of the knowledge distillation method for small sample learning according to the first aspect when the electronic device runs.

[0034] Advantages:

[0035] (1) The present application proposes three knowledge distillation methods for small sample learning: an auxiliary classifier-based distillation method (ACD), an image local feature-based distillation method (LFD), and an image internal relationship-based distillation method (IRD).

[0036] (2) Compared with traditional knowledge distillation, the method proposed in the present application has more significant performance improvement for student models in small sample learning tasks.

[0037] (3) The method proposed in the present application also has good effect in small sample learning tasks under light weight setting. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the classification of knowledge distillation methods and the intention of the present application.

[0039] Figure 2 is a comparison diagram of a conventional classification task and a small sample classification task.

[0040] Figure 3 is a diagram of the DN4 small sample metric classifier used in the experiment.

[0041] Figure 4 is a flowchart of the auxiliary classifier-based distillation method (ACD) proposed in the present application.

[0042] Figure 5 is a flowchart of the image local feature-based distillation method (LFD) proposed in the present application.

[0043] Figure 6 is a flowchart of the image internal relationship-based distillation method (IRD) proposed in the present application. DETAILED DESCRIPTION

[0044] In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present application. However, it is obvious to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, some technical features known in the art are not described in order not to obscure the present application.

[0045] The present application aims to improve the working mode of knowledge distillation from the method level, so that it is more suitable for image classification tasks under small sample setting. The traditional knowledge distillation method based on classification probability has obvious limitations in small sample learning tasks, such as Figure 2As shown, the training process of small sample learning is based on a large number of randomly extracted "N-way K-shot" subtasks, each subtask contains a small number of categories, and the order of the categories is randomly combined. If the distillation loss is directly calculated based on the classification probability of each small sample task, the gradient update of the student model generated by the distillation loss through back propagation is only applicable to the current subtask category combination, and is not necessarily beneficial to other category combinations randomly generated in the subsequent training process. In this way, the update of the distillation loss to the student model becomes a kind of adjustment for the random subtask, and lacks a constant update target, which makes it more difficult for the student model to approach the performance of the teacher model.

[0046] Therefore, the ideal small sample learning knowledge distillation method must solve the following problems:

[0047] (1) It is necessary to explore a sufficient amount of potential knowledge under the condition of limited sample size to maintain the effectiveness of distillation;

[0048] (2) The calculation of the distillation loss needs to be independent of the categories of the small sample task to provide a relatively constant update direction for the update of the student model.

[0049] The three small sample knowledge distillation methods proposed in the application will solve the above problems from different angles.

[0050] ① Auxiliary classifier distillation (ACD) based on auxiliary classifier: this scheme first retains the linear classifier at the end of the teacher model used for pre-training based on the conventional classification task, and also adds an auxiliary linear classifier to the student model, and the target class is equal to the length of the entire training set sample space. At this time, the classification process of the student model in the distillation training process includes two independent branches: the first branch normally performs small sample measurement classification to obtain small sample classification probability and calculate classification loss; the second branch uniformly passes all sample features through the auxiliary classifier to obtain fixed classification probability on the entire training set category, and calculates the distillation loss with the same dimension classification probability generated by the teacher model classifier. In the design of this method, the auxiliary classifier does not directly participate in the small sample classification task, but through the back propagation of the distillation loss to the student model, the ability of feature extraction is enhanced, thereby indirectly improving the performance of small sample learning. Since the teacher model and the student model both generate fixed classification probability on the entire training set category, the defect of small number of categories and random combination in the small sample random task is avoided, and a relatively constant target is provided for the distillation learning of the student model.

[0051] ② Local Feature Distillation (LFD): Each sample image is fed into the feature extraction network of the teacher and student model respectively, and two sequences of n feature vectors are obtained. The output features of the teacher model are denoted as [t1, t2, …, t n ], and the output features of the student model are denoted as [s1, s2, …, s n ]. For any i, t i and s i are local feature vectors corresponding to the same position of the same image, and their theoretical distributions should also be close to each other. Therefore, after each local feature vector is activated by SoftMax, the difference is calculated for each position one by one, and the specific distillation loss definition is still the KL divergence. Finally, the loss values calculated by all feature pairs are added together as the final distillation loss. In this way, local feature distillation avoids the relevance between the distillation loss and the small sample task categories in the traditional distillation method, and instead pursues the fitting of the teacher at the intermediate layer feature distribution level, which is more suitable for the large number of random sub-tasks generated in the small sample training process. The use of spatial local features ensures that sufficient feature pairs can be obtained even in the case of limited sample size in the small sample setting, ensuring the reliability of the distillation effect.

[0052] ③ Internal Relational Distillation (IRD): Each sample image is fed into the feature extraction network of the teacher and student model respectively, and two sequences of n feature vectors are obtained. Between the features obtained by the teacher model, the cosine similarity is calculated between each pair, and a similarity matrix R t of n*n is obtained. The features obtained by the student model are also processed similarly, and a similarity matrix R s is obtained. Each cell in the similarity matrix reflects the relationship between two regions inside the image. Therefore, the mean square error of the two similarity matrices can be calculated as the distillation loss. Under this distillation scheme, the student does not need to fit the teacher in the absolute size of the feature distribution, but only needs to generate a relative relationship prediction close to the teacher model, which can be considered as effective distillation. Compared with the distillation method based on the relationship between samples, this scheme explores the relationship between features inside the image, and is more suitable for the scenario of small sample task with small sample size.

[0053] The knowledge distillation method for small sample learning tasks proposed by the present application specifically includes the following application steps:

[0054] Step (1): The small sample image dataset is divided into a training set, a validation set, and a test set that do not overlap in the class space;

[0055] Step (2): Pre-training the teacher model based on a regular classification task. Perform channel pruning on the teacher model network to obtain a lightweight student model.

[0056] Step (3): Small sample training using random sub-task mode. Extract a fixed number of categories each time to construct an "N-way K-shot" small sample classification task. The extracted image samples are divided into support set and query set.

[0057] Step (4): After uniform preprocessing, the image samples in each task are respectively passed through the feature extraction networks of the teacher and student models to obtain two groups of image features.

[0058] Step (5): The student model completes small sample metric classification based on its own output and calculates the classification loss.

[0059] Step (6): According to the selected distillation method, calculate the distillation loss based on the outputs of the teacher and student models.

[0060] Step (7): Weighted sum of classification loss and distillation loss as the overall backpropagation update of the student model.

[0061] 1. KL divergence loss representation

[0062] KL divergence is the most commonly used distillation loss function, which is used to measure the difference between two groups of probability distributions of the same dimension. The formula is shown in equation (1). S and T are the classification probabilities generated by the student and teacher models, respectively. x represents an independent event, which in the classification recognition task means the current sample belongs to each class. P S (x) and P T (x) represent the prediction probabilities of the student and teacher models for event x, respectively. Finally, the entropy values for all class events x are summed to obtain the KL divergence value between S and T.

[0063] (1)

[0064] In a regular classification task, the lengths of S and T are equal to the total number of classes in the training set, and each bit corresponds to a fixed class. In the "N-way K-shot" task under the small sample setting, the length of S and T is only N, i.e. the number of classes included in each random sub-task, and the true class corresponding to each bit is also completely random between different sub-tasks.

[0065] 2. Distillation method based on auxiliary classifier representation

[0066] As Figure 3As shown, the linear classifier used by the teacher model in the pre-training stage is retained, and an auxiliary linear classifier of the same dimension is added to the student model. After the input image samples are passed through the feature extraction networks of the teacher model and the student model in parallel, the student model performs small-sample metric classification and regular linear classification on the auxiliary classifier based on the obtained features. The classification loss is calculated normally based on the "N-way" classification probability obtained from the small-sample classification. The fixed classification probability in the entire training set category space generated by the auxiliary classifier is calculated based on the KL divergence with the same-dimensional probability generated by the teacher model to obtain the distillation loss based on the auxiliary classifier. The formula is expressed as Equation (2):

[0067] (2)

[0068] 3. Distillation method based on local image features

[0069] like Figure 4 As shown, the input image samples are processed by the feature extraction networks of the teacher model and the student model, respectively, to obtain two sets of local features of the same dimension, denoted as [t1, t2, …, t]. n ] and [s1, s2, …, s n ]. Among them, the t with the same number i and s i The receptive field corresponds to the same region of the image. After the two sets of features are activated by the SoftMax function, the degree of difference is calculated for each location (the specific definition is still based on the KL divergence loss). Finally, the results are summed to obtain the distillation loss based on the local features of the image. The formula is expressed as in equation (3):

[0070] (3)

[0071] 4. Distillation method based on image internal relationships

[0072] like Figure 5 As shown, the input image samples are processed by the feature extraction networks of the teacher model and the student model, respectively, to obtain two sets of local features of the same dimension, denoted as [t1, t2, …, t]. n ] and [s1, s2, …, s n The cosine similarity is calculated between each pair of features within each feature group, resulting in two n*n similarity matrices R. t and R s Each similarity value in the matrix reflects the relationship between two regions within the image. The mean squared error between the two similarity matrices is calculated to obtain the distillation loss based on the internal relationship of the image. The formula is expressed as in equation (4):

[0073] (4)

[0074] To evaluate the performance of the three small sample distillation methods proposed in the present application, comparative experiments were carried out on the image data sets MiniImageNet and TieredImageNet commonly used in the field of small sample learning. Small sample training and testing are based on the random subtask mode, each "N-way K-shot" subtask contains a fixed number of N classes, and each class contains a fixed number of K support set samples and a number of query set samples. The experiment adopts a lightweight distillation setting, and the teacher model feature extraction network is a complete Resnet12, and the student model feature extraction network is a lightweight processed Slim-Resnet12. The small sample metric classifier is DN4, and its metric classification principle is as shown in Figure 3 The conventional classification loss is calculated based on the cross entropy between the classification probability and the true label. The distillation loss is calculated according to the selected distillation method, and has different calculation methods:

[0075] (1) Traditional knowledge distillation method (KD): the teacher model removes the linear classifier and global pooling layer used in the pre-training stage, and a DN4 small sample metric classifier is added at the end. The samples contained in each small sample subtask are respectively passed through the teacher and student models to generate two groups of small sample probability distributions of the same dimension, and the distillation loss is calculated using the KL divergence.

[0076] (2) Distillation method based on auxiliary classifier (ACD): as shown in Figure 4 The teacher model retains the linear classifier and global pooling layer used in the pre-training stage. The student model additionally adds a linear classifier. The random samples contained in each subtask are respectively passed through the feature extraction networks and linear classifiers of the teacher and student models to obtain fixed classification probabilities in the entire training set class space, and the distillation loss is calculated according to formula (2).

[0077] (3) Distillation method based on image local features (LFD): as shown in Figure 5 The teacher model removes the linear classifier and global pooling layer, and retains the intermediate layer features (local features) of each batch of input image samples as the result. The same dimension local features obtained by the teacher model and the student model are one-to-one corresponding according to the position, and the distillation loss is calculated according to formula (3).

[0078] (4) Distillation method based on image internal relationship (IRD): as shown in Figure 6 The teacher model removes the linear classifier and global pooling layer, and retains the intermediate layer features (local features) of each batch of input image samples as the result. The teacher model and the student model calculate the similarity in the respective obtained feature sets, and the two groups of similarity matrices of the same dimension are calculated according to formula (4).

[0079] Table 1 and Table 2 show the comparison results of applying various distillation methods and no-distillation training on MiniImageNet and TieredImageNet datasets under the lightweight distillation setting. Distillation loss weights are used to balance the importance of distillation during training. The distillation loss weights filled in the table are the optimal values obtained by actual debugging of various distillation methods. The final test set accuracy is the statistical mean of the accuracy of the lightweight student model on 10,000 random sub-tasks after training.

[0080] Table 1 MiniImageNet dataset small sample distillation results

[0081]

[0082] Table 2 TieredImageNet dataset small sample distillation results

[0083]

[0084] The comparison results show that the performance of various distillation methods is better than that of the no-distillation control group. Compared with traditional distillation methods, the three small sample learning-oriented distillation methods proposed in this paper have more significant effects. Taking the 5-way 1-shot task as an example, on the MiniImageNet dataset, the accuracy of the auxiliary classifier distillation method is improved by 8.70% compared with the no-distillation control, the accuracy of the local feature distillation method is improved by 7.69%, and the accuracy of the internal relationship distillation method is improved by 7.42%. On the TieredImageNet dataset, the accuracy of the auxiliary classifier distillation method is improved by 6.92%, the accuracy of the local feature distillation method is improved by 6.53%, and the accuracy of the internal relationship distillation method is improved by 6.88%. The results prove the effectiveness of the methods proposed in this invention in small sample distillation tasks.

[0085] Table 3 shows the results of small sample distillation experiments using the local feature distillation scheme on the MiniImageNet dataset under the self-distillation setting. At this time, the teacher model and the student model both use a complete Resnet12 as the feature extraction network. The teacher model results in the table are obtained by small sample training alone.

[0086] Table 3 MiniImageNet dataset results under self-distillation setting

[0087]

[0088] The results show that the student model precision obtained by the local feature distillation scheme under the self-distillation setting even exceeds the teacher model itself, with an increase of more than one percentage point. Further, it is proved that the distillation method proposed in this paper can be used as an auxiliary means in small sample learning tasks, and further explore the potential performance of the model without introducing additional data.

[0089] Table 4 shows the performance comparison of the small sample distillation method proposed in this paper and other small sample learning methods. In the table, the results based on the complete Resnet12 network and the lightweight Slim-Resnet12 are shown respectively, and the optimal results of the two settings are highlighted in bold. The distillation method proposed in this paper has the best performance under the lightweight setting.

[0090] Table 4 Performance comparison of the method in this paper and other small sample learning methods

[0091]

[0092] As described above, although the present application has been shown and described with reference to specific preferred embodiments, it is not to be construed as being in any way limited to the precise forms of the above described embodiments, and should not be interpreted in the scope of the appended claims. Various changes can be made to the forms and details of the present application without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A knowledge distillation method for small sample learning, characterized in that, Comprising the following steps: Step S1, dividing the small sample image data set into training set, validation set and test set which do not overlap in category space; Step S2, pre-training the teacher model based on the conventional classification task; performing channel pruning on the teacher model network to obtain a lightweight student model; specifically comprising: S2-1, performing channel pruning on the complete feature extraction network used by the teacher model, reducing the number of I / O channels of each layer by 50% except for the initial input and final output channels, and finally reducing the overall network parameter quantity to 1 / 3 of the original teacher model; S2-2, reinitializing the pruned lightweight network parameters as the student model to start training from scratch; using a twelve-layer residual network as the feature extraction network of the teacher model, and using the lightweight network obtained by channel pruning on this basis as the feature extraction network of the student model; Step S3, using a random sub-task mode for small sample training, extracting a fixed number of categories each time to construct an "N-way K-shot" small sample classification task, and dividing the extracted image samples into support set and query set; Step S4, after the image samples contained in each task are uniformly preprocessed, two groups of image features are obtained through the feature extraction networks of the teacher and student models respectively; Step S5, the student model completes small sample metric classification based on its own output and calculates the classification loss; Step S6, according to the selected distillation method, the distillation loss is calculated based on the outputs of the teacher model and the student model; the distillation loss calculation uses the image intermediate layer features or the relationship between the features or the fixed classification probability generated by the auxiliary classifier to perform distillation constraint; The specific calculation methods of the distillation losses of each method are as follows: Auxiliary classifier-based distillation: after the student model obtains local features, in addition to completing small sample metric classification, the features are passed through a global mean pooling layer and an additional auxiliary linear classifier to obtain fixed classification probabilities S in the entire training set category space, and the same dimension classification probabilities T generated by the teacher model are used to calculate the distillation loss according to the KL divergence: ； Image local feature-based distillation: the teacher model and the student model obtain two groups of local features for the same sample, each containing n feature vectors of the same dimension; the KL divergence is calculated between the feature vectors corresponding to the same position, and then the sum is obtained as the final distillation loss: ； In the formula, With respectively represent the local feature vectors generated by the student model and the teacher model, both have the same dimension and correspond to the same local region in the same input image; Image internal relationship-based distillation: after the teacher model and the student model obtain two groups of local features for the same sample, the cosine similarity is calculated between each other in the obtained features to obtain two n*n similarity matrices, and the mean square error of the two similarity matrices is used as the distillation loss: ； wherein, and respectively represent the similarity matrices of the same dimension produced by the teacher model and the student model, and represent the local feature vectors produced by the teacher model and the student model for the same image region i, and represent the local feature vectors produced by the teacher model and the student model for the same image region j; Step S7, the weighted sum of the classification loss and the distillation loss is used as the overall back propagation update for the student model.

2. The knowledge distillation method for small sample learning according to claim 1, wherein, The small sample image data set in step S1 includes MiniImageNet data set and TieredImageNet data set; The MiniImageNet data set contains N image categories, and each image category contains M images; Among the N image categories, a image categories are used for training, b categories are used for validation during the training process, and c categories are used for final testing; The TieredImageNet dataset contains P image categories, and each of the image categories contains Q images respectively. Among the P image categories, x image categories are used for training, y categories are used for validation during the training process, and z categories are used for final testing.

3. The knowledge distillation method for small sample learning according to claim 1, wherein, In step S3, only a fixed number of categories are extracted, and each category contains a fixed number of image samples in the process of constructing the "N-way K-shot" small sample subtask.

4. The knowledge distillation method for small sample learning according to claim 3, wherein, For the 5way-5shot subtask, 5 categories are extracted, each containing 1 support set sample and 15 query set samples; for the 5way-5shot subtask, 5 categories are extracted, each containing 5 support set samples and 15 query set samples; the category combination contained in each subtask is random.

5. The knowledge distillation method for small sample learning according to claim 1, wherein, In step S4, the preprocessing of the image includes uniform scaling to 84*84 pixels, random data augmentation transformation, and standardization based on the statistical prior of the mean and variance of the dataset pixels; The feature extraction network of the teacher model and the student model is a deep neural network structure composed of convolution-pooling-activation function layers; according to whether the feature extraction network contains a global pooling layer at the end, the obtained features are divided into global features or local features; Among them, the global feature represents that each image generates a unique feature vector; The local feature represents that each image generates a batch of n>1 feature vectors.

6. The method of claim 1, wherein the method is a small sample learning oriented knowledge distillation method. In step S5, after the student model extracts the local features of the image, it inputs them to the small sample metric classifier. The features from the support set samples are used to represent the feature centers of the categories contained in the current subtask, and the metric classifier judges the similarity between each query set sample and each feature center, and the closest category is taken as the predicted classification result of the query set sample.

7. A small sample learning oriented knowledge distillation apparatus, configured to perform the small sample learning oriented knowledge distillation method according to any one of claims 1 to 6. It comprises: A first module for dividing a small sample image dataset into a training set, a validation set, and a test set that do not overlap in category space; A second module for pre-training a teacher model based on a conventional classification task; Performing channel pruning on the teacher model network to obtain a lightweight student model; A third module for small sample training in a random subtask mode, which extracts a fixed number of categories to construct an "N-way K-shot" small sample classification task each time, and the extracted image samples are divided into a support set and a query set; A fourth module for obtaining two groups of image features through the feature extraction networks of the teacher and student models after the image samples contained in each task are uniformly preprocessed; A fifth module for completing small sample metric classification based on the output of the student model and calculating the classification loss; A sixth module for calculating the distillation loss based on the outputs of the teacher model and the student model according to the selected distillation method; A seventh module for weighting and summing the classification loss and the distillation loss to update the student model.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, and the executable instruction runs on the electronic device to make the electronic device execute the operations of the small sample learning knowledge distillation method according to any one of claims 1-6.

Images