Collaborative picture classification method based on knowledge distillation

Abstract

The invention discloses a collaborative picture classification method based on knowledge distillation, and the method comprises the steps: adding several quitted branches to several middle layers of an original neural network, and each branch is composed of an attention module and a classifier; then, the knowledge distillation part selects another model with a different framework, in the pre-training stage, the model of the knowledge distillation part is trained to obtain a better model export parameter, and then in the training stage, the soft label generated by the model of the knowledge distillation part is used for providing rich external knowledge for the self-distillation model. The model of the self-distillation part, the teacher model and the student model are arranged in one model, and a deeper classifier is utilized to provide soft label knowledge for a shallower classifier; in the reasoning stage, all additional attention modules and classifiers are abandoned, only one classifier is left, and the deployed model does not have additional attention modules and redundant classifiers, so that no additional parameter quantity exists.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge distillation, and in particular relates to a collaborative image classification method based on knowledge distillation. Background Art

[0002] In image classification tasks, due to the huge model size of deep learning, many deep network models are very dependent on computing and storage resources, making it almost impossible to deploy these models in some practical scenarios, because many IoT and mobile devices are devices with low computing and storage resources. For these problems, it is necessary to make high-performance models meet the requirements of low performance and low power consumption, while at the same time ensuring that the performance of the model is not significantly reduced. Model lightweighting is needed to solve the above problems. There are four mainstream methods for model lightweighting: model pruning, quantization, low-rank decomposition, and knowledge distillation.

[0003] Knowledge distillation is a teacher and student model structure. Knowledge distillation usually involves distilling knowledge from a trained teacher model. The student model acquires the knowledge in the teacher model through knowledge distillation training. Knowledge distillation is a method in which the student model improves the ability of the student model by imitating the "dark knowledge" of the similarities between the output classes of the teacher model. The teacher model can provide soft targets to the student model to provide more information than the corresponding hard targets to improve the ability of the student model. Soft targets can guide the direction of optimization of the student model. The student model is usually much smaller than the teacher model, making it easier to deploy on mobile devices, which can solve the difficulties in model deployment.

[0004] Distilling the teacher model to the student model is a challenging task: 1) The selection of the teacher model is difficult. The selection of the teacher model has a great impact on the ability of the student model. The best teacher model is not necessarily suitable for the currently selected student model. 2) The student model always encounters the problem of accuracy degradation in the inference stage.

[0005] There are two traditional knowledge distillation methods: logit distillation and feature distillation. Logit distillation is to transfer the knowledge in the teacher model (large) to the student model (small) at the logit level by minimizing the KL-Divergence between the teacher and student predicted logits. Feature distillation transfers knowledge by matching the feature distribution and logit output of the middle layer between the teacher model and the student model. To use the feature distribution of the middle layer, the architecture of the teacher model (large) and the student model must be similar, but logit distillation does not have this limitation. Another distillation method is self-distillation. The method adopted by self-distillation is that the teacher model and the student model are one model, and this model is used to guide itself to transfer knowledge and perform knowledge distillation. Because the knowledge transfer of self-distillation is within a model, it inevitably leads to the phenomenon of knowledge scarcity in self-distillation and lack of rich external knowledge. Summary of the invention

[0006] The present invention aims at solving the problems in the prior art and proposes the following technical solutions:

[0007] A collaborative image classification method based on knowledge distillation, the classification method comprising the following steps:

[0008] S1. Training the knowledge distillation model: First, train the knowledge distillation model to obtain a better model export parameter. Then, during the training phase, use the soft labels generated by the knowledge distillation model to enrich the external knowledge of the self-distillation model.

[0009] S2, knowledge distillation and self-distillation collaborative training: add several exit branches to several intermediate layers of the original neural network, and each branch consists of an attention module and a classifier. The model of the self-distillation part, the teacher model and the student model are in one model, and the deeper classifier will be used to provide soft label knowledge to the shallower classifier;

[0010] S3, model reasoning and deployment: Based on the trained results, select the corresponding model block according to your needs, discard all additional attention modules and classifiers, and leave only one classifier.

[0011] The present invention is an image classification method based on traditional knowledge distillation and self-distillation. The self-distillation part of the method does not change the structure of the trunk layer, but adds several exit branches to several intermediate layers of the original neural network, and the branches are composed of an attention module and a classifier. Then the knowledge distillation part selects another model with a different architecture. In the preparatory training stage, the model of the knowledge distillation part is first trained to a better model to derive parameters, and then in the training stage, the soft labels generated by the model of the knowledge distillation part are used to enrich the external knowledge of the self-distillation model. In the model of the self-distillation part, the teacher model and the student model are in one model, and the deeper classifier will be used to provide soft label knowledge to the shallower classifier. In the inference stage, all additional attention modules and classifiers will be discarded, leaving only one classifier. The deployed model has no additional attention modules and redundant classifiers, so there will be no additional parameters.

[0012] As a preferred embodiment of the above technical solution, the knowledge distillation model in S1 adopts a densely connected network (DenseNet).

[0013] As a preferred embodiment of the above technical solution, the Densenet expands the function as follows:

[0014] x→[x, f1(x), f2([x, f1(x)]),...].

[0015] As a preferred embodiment of the above technical solution, the self-distillation part in S2 adopts a residual network (ResNet) model, and the self-distillation is a change and addition to the ResNet model.

[0016] As a preferred embodiment of the above technical solution, the outputs of the multiple classifiers are subjected to a softmax function.

[0017] As a preferred embodiment of the above technical solution, the model is exported to ONNX (Open Neural Network Exchange) format during deployment, and its ONNX format file is imported into the corresponding mobile hardware end that supports ONNX format during deployment.

[0018] The beneficial effects of the present invention are:

[0019] 1. The self-distillation method of the present invention allows the compressed model to be deployed on the mobile terminal, overcoming the limitations of low computing resources and low memory resources of mobile terminal devices;

[0020] 2. The self-distillation method of the present invention can overcome the difficulty in selecting the teacher model. The teacher model and the student model are in one model, allowing the student model to fully utilize the knowledge of the teacher model;

[0021] 3. The self-distillation method of the present invention uses an additional traditional knowledge distillation method to collaborate with the self-distillation method to compress the model, thereby providing the self-distillation model with rich external knowledge, which can make the model compression effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Shown is a flow chart of the collaborative method of knowledge distillation and self-distillation in the present invention;

[0023] Figure 2 What is shown is the block structure diagram of ResNet in the present invention;

[0024] Figure 3 What is shown is the block structure diagram of DenseNet in the present invention;

[0025] Figure 4 Shown is a diagram of a network model for collaborative training of knowledge distillation and self-distillation in the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments.

[0027] A collaborative image classification method based on knowledge distillation, the classification method comprising the following steps:

[0028] S1. Training the knowledge distillation model: First, train the knowledge distillation model to obtain a better model export parameter. Then, during the training phase, use the soft labels generated by the knowledge distillation model to enrich the external knowledge of the self-distillation model.

[0029] S2, knowledge distillation and self-distillation collaborative training: add several exit branches to several intermediate layers of the original neural network, and each branch consists of an attention module and a classifier. The model of the self-distillation part, the teacher model and the student model are in one model, and the deeper classifier will be used to provide soft label knowledge to the shallower classifier;

[0030] S3, model reasoning and deployment: Based on the trained results, select the corresponding model block according to your needs, discard all additional attention modules and classifiers, and leave only one classifier.

[0031] The main steps of the method of the present invention are divided into three parts: training the knowledge distillation partial model, collaborative training of knowledge distillation and self-distillation, and model reasoning and deployment.

[0032] In S1, the knowledge distillation model can choose a dense network (DenseNet) because it is similar to and different from the residual network (ResNet). The residual network is based on blocks, and each block is connected by a series of layers and shortcuts. The shortcut connects the input and output of the residual network module together. Figure 2 , the dense network is also based on blocks, and each layer has output connections. Figure 3 The key difference between ResNet and DenseNet is that DenseNet outputs connections, while ResNet simply adds, which can be distinguished by formulas. Because each dense block of a dense network will increase the number of channels, adding more will make the number of channels too large. Using transition layers to control the complexity of the model, using 1×1 convolutional layers to reduce the number of channels, and then using an average pooling layer with a stride of 2 to reduce the half-height and width can reduce the complexity of the model.

[0033] ResNet expands the function to:

[0034] f(x)=x+g(x)

[0035] Densenet expands the function into:

[0036] x→[x, f1(x), f2([x, f1(x)]),...]

[0037] Then use DenseNet training to select better parameters, export the parameters and wait for the next step.

[0038] In S2, the self-distillation part uses the ResNet model, and self-distillation is a change and addition to the ResNet model. Self-distillation does not change the main structure of the network model, but adds several exit branches to some middle layers of the network. These branches are composed of an attention module and a classifier. Knowledge distillation and self-distillation collaborate to train the network.

[0039] like Figure 2 , Figure 3 Shown

[0040] Classifier1, classifier2, and classifier3 can all be used separately during inference, and other redundant parts can be directly deleted. Classifier1 is equivalent to being supervised by classifier2 and classifier3 plus the supervision of secondary classifier4, and classifier2 is equivalent to being supervised by classifier3 plus the supervision of secondaryclassifier4. In the collaborative training method of knowledge distillation and self-distillation, classifier1 is equivalent to receiving three supervisions, and three hyperparameters are required to supervise them. Classifier2 is equivalent to receiving two supervisions, and two hyperparameters are required to supervise it.

[0041] Each classifier output needs to pass through the softmax function. An example is as follows: For a training sample from the xth class, the classification probability can be expressed as p = [p1, p2, ..., px, ..., pc] ∈ R 1×C , where pi is the probability of the i-th class, C is the number of classes, and each element in p is obtained through the softmax function as follows:

[0042]

[0043] Where zi represents the logit of the i-th class, and T represents the distillation temperature, which is a hyperparameter and is usually greater than 1. When T = 1.0, the output will be reduced to a normal softmax output, minimizing the KL divergence between the student and teacher model outputs to achieve distillation.

[0044]

[0045] L kD represents the loss of knowledge distillation, and Represent the probability values ​​of the jth class output by the teacher and the student respectively.

[0046] Each classifier has its own distillation loss. Classifier1, classifier2, classifier3, and secondary classifier4 are represented as C1KD, C2KD, C3KD, and C4KD, respectively. The total loss of classifier1, classifier2, and classifier3 is represented as SC1KD, SC2KD, and SC3KD. For classifier3, it receives a knowledge transfer from secondary classifier4. Secondary classifier4 is equivalent to the teacher of classifier3, and classifier3 is equivalent to the student. Two hyperparameters will be introduced as the weight of this transfer. For classifier3, it turns the classic knowledge distillation loss into the weighted sum of two independent parts, which can be written as the following formula:

[0047] SC3KD=αC3KD+βC4KD

[0048] For classifier2, it receives two knowledge transfers, from classifier3 and secondaryclassifier4. Classifier3 and secondary classifier4 are equivalent to the teachers of classifier2, and classifier2 is equivalent to the student. Two hyperparameters η and λ are introduced as the weights of the knowledge transfer between the two.

[0049] SC1KD=C1KD+λSC2KD

[0050] The above contents constitute the traditional knowledge distillation and self-distillation collaborative training method.

[0051] In S3, based on the training results, you can choose the corresponding model block according to your needs. The accuracy of the model composed of classifier1 is the worst compared to the others, but its inference speed is the best. The accuracy of the model composed of classifier2 is balanced compared to the others, and the inference speed is also balanced. The accuracy of the model composed of classifier2 is the best, and the inference speed is slower than the others. When deploying the model, other redundant classifiers need to be removed. For example, when deploying classifier1, classifier2 and classifier3 of the self-distillation part need to be removed, so that no redundant parameters are added during the inference process.

[0052] When deploying, the model is exported to ONNX (Open Neural Network Exchange) format, which is a format used to describe computational graphs in a standard way. It has been connected to a variety of deep learning frameworks and a variety of inference engines for deployment. When deploying, you only need to import the ONNX format file into the corresponding mobile hardware that supports the ONNX format, and you can perform image recognition on devices with low computing resources and low memory resources such as mobile hardware.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them.

Claims

1. A collaborative image classification method based on knowledge distillation, characterized in that: The classification method comprises the following steps: S1. Training the knowledge distillation model: First, train the knowledge distillation model to obtain a better model export parameter. Then, during the training phase, use the soft labels generated by the knowledge distillation model to enrich the external knowledge of the self-distillation model. S2, knowledge distillation and self-distillation collaborative training: add several exit branches to several intermediate layers of the original neural network, and each branch consists of an attention module and a classifier. The model of the self-distillation part, the teacher model and the student model are in one model, and the deeper classifier will be used to provide soft label knowledge to the shallower classifier; S3, model reasoning and deployment: Based on the trained results, select the corresponding model block according to your needs, discard all additional attention modules and classifiers, and leave only one classifier.

2. A collaborative image classification method based on knowledge distillation according to claim 1, characterized in that: The knowledge distillation model in S1 adopts a densely connected network (DenseNet).

3. The collaborative image classification method based on knowledge distillation according to claim 2, characterized in that: The Densenet expands the function as follows: x→[x, f1(x), f2([x, f1(x)]),...].

4. The collaborative image classification method based on knowledge distillation according to claim 1, characterized in that: The self-distillation part in S2 adopts the residual network (ResNet) model, and the self-distillation is a change and addition to the ResNet model.

5. The collaborative image classification method based on knowledge distillation according to claim 4, characterized in that: The outputs of the multiple classifiers are passed through a softmax function.

6. The collaborative image classification method based on knowledge distillation according to claim 1, characterized in that: During the deployment, the model is exported to the ONNX (OpenNeural Network Exchange) format, and its ONNX format file is imported into the corresponding mobile hardware end that supports the ONNX format during deployment.

Images