A self-distillation classification method based on progressive knowledge transfer

Abstract

The application discloses a self-distillation classification method based on progressive knowledge transmission. A self-distillation phased learning mode is proposed, and shallow student networks are caused to learn from teacher networks progressively in a learning process, so that the classification performance of the student networks is improved. Firstly, after the training of the teacher network is completed, the parameters of the teacher network are frozen, and an early exit module of a student network is introduced to perform distillation operation. Secondly, when the student network is trained, the student network learns knowledge from a deep teacher network in stages.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural network self-distillation and image classification. In particular, it relates to a self-distillation classification method based on progressive knowledge transfer. BACKGROUND

[0002] In recent years, deep learning has developed rapidly and has been widely used in image recognition, target detection and other tasks. Although convolutional neural networks have achieved excellent results in various tasks, as the difficulty of the task increases, the structure of the convolutional neural network also becomes gradually complex to obtain good adaptability, but good adaptability is often at the cost of a large number of parameters. A large number of parameters will consume a large amount of disk storage space when storing the model, and bring a large amount of floating-point matrix multiplication when the model is forward inferred. The problems of space storage and floating-point calculation make it impossible for complex models to guarantee real-time inference results. Although there are now manufacturers developing high-speed computing devices, in the current rapid development of deep learning, the update cycle of hardware cannot meet the speed of neural network iteration. When complex networks are used on devices with limited resources, energy consumption, storage and delay are all deployment difficulties, and these problems are particularly prominent in the development of neural networks.

[0003] Innovations of the present application and differences from prior art

[0004] In the self-distillation framework, when a deep complex network is used as a teacher for a shallow student network, the complexity gap between the teacher network and the student network will result in an unsatisfactory knowledge transfer from the teacher to the student. Moreover, as the complexity of the teacher network increases, the classification performance of the shallow student network will gradually decrease. Therefore, the present application proposes a self-distillation phased learning method, which enables the shallow student network to gradually learn from the teacher network during the learning process, thereby improving the classification performance of the student network. First, after the teacher network is trained, its parameters are frozen, and then an early exit module of the student network is introduced for distillation operation. Second, when training the student network, it is divided into stages to learn knowledge from the deep teacher network. SUMMARY

[0005] Existing neural networks have a large number of parameters and high computational complexity, making it difficult to deploy them on devices with limited resources. To solve this problem, the present application uses a progressive approach to improve self-distillation classification. This method gradually trains the student network by continuously improving the complexity of the teacher network.

[0006] A self-distillation classification method based on progressive knowledge transfer mainly includes the following steps:

[0007] Step S1, preprocessing the training data set

[0008] Step S1.1, data augmentation using flipping and cropping

[0009] Step S1.2, shuffle the data and divide them into different batches. Each batch contains 128 data samples.

[0010] Step S2, training of the neural network

[0011] Step S2.1, phase 1 teacher network training process. The training set data is fed into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification exit. The teacher network is updated using the label, and the label loss is defined as The parameters of the network are updated using the stochastic gradient descent method (SGD).

[0012] Step S2.2, introduce the early exit student network, and construct the self-distillation structure. Freeze the parameters of the teacher network.

[0013] Step S2.3, phase 1 student network backpropagation process. The student network is supervised by the teacher network loss, which is defined as At the same time, the student network is subjected to the label loss The parameters of the network are updated using the stochastic gradient descent method (SGD).

[0014] Step S2.4, cycle S2.1-S2.3.

[0015] Step S2.5, phase 2 teacher network training process. The training set data is fed into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification exit. The teacher network is updated using the label, and the label loss is defined as The parameters of the network are updated using the stochastic gradient descent method (SGD).

[0016] Step S2.6, introduce the early exit student network, and construct the self-distillation structure. Freeze the parameters of the teacher network.

[0017] Step S2.7, phase 2 student network backpropagation process. The student network is supervised by the teacher network loss, which is defined as At the same time, the student network is subjected to the label loss The parameters of the network are updated using the stochastic gradient descent method (SGD).

[0018] Step S2.8, cycle S2.5-S2.7.

[0019] Step S2.9, phase 3 teacher network training process. The training set data is fed into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification exit. The teacher network is updated using the label, and the label loss is defined as The parameters of the network are updated using the stochastic gradient descent method (SGD).

[0020] Step S2.10, introduce early exit student network, network parameters come from stage 2. Construct self-distillation structure. Freeze teacher network parameters.

[0021] Step S2.11, stage 3 student network back propagation process. Student network is supervised by teacher network loss, defined as At the same time, student network is subject to label loss Update network parameters using stochastic gradient descent method SGD.

[0022] Step S2.12, loop S2.9-S2.11.

[0023] Step S2.13, put training set data into neural network, get prediction result for current sample when reaching teacher classification exit. Update teacher network using label, label loss is defined as Update network parameters using stochastic gradient descent method SGD.

[0024] Step S2.14, introduce early exit student network, network parameters come from stage 3. Construct self-distillation structure. Freeze teacher network parameters.

[0025] Step S2.15, stage 2 student network back propagation process. Student network is supervised by teacher network loss, defined as At the same time, student network is subject to label loss Update network parameters using stochastic gradient descent method SGD.

[0026] Step S2.16, loop S2.13-S2.15.

[0027] Step S3, prediction process

[0028] Step S3.1, prune neural network, only keep shallow branch prediction exit.

[0029] Step S3.2, input picture to be predicted into model, get classification result. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 Model structure involved in the present application.

[0031] Figure 2 Model processing flowchart involved in the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with specific embodiments and with reference to the drawings.

[0033] This embodiment uses Cifar100 dataset as image classification training dataset. Cifar100 dataset contains 60,000 training images, of which 50,000 images are training set and 10,000 images are test set, and a total of 10 categories.

[0034] Step S1, preprocessing training dataset

[0035] Step S1.1, using flip, cropping method to enhance the training data

[0036] Step S1.2, randomly shuffle the data and divide it into different batches. The number of data samples in each batch is 128.

[0037] Step S2, training of neural network

[0038] Step S2.1, phase 1. Set the learning rate of the neural network to 0.1.

[0039] Step S2.2, phase 1 teacher network training process. The training set data is sent into the neural network, and the prediction result for the current sample is obtained when it reaches the teacher classification exit. Update the teacher network using the label, and the label loss is defined as Update the parameters of the network using the stochastic gradient descent method SGD.

[0040] Step S2.3, introduce early exit student network, construct self-distillation structure. Freeze the parameters of the teacher network.

[0041] Step S2.4, phase 1 student network back propagation process. The student network is supervised by the teacher network loss, which is defined as At the same time, the student network is subject to label loss Update the parameters of the network using the stochastic gradient descent method SGD.

[0042] Step S2.5, cycle S2.1-S2.4, update round is 125.

[0043] Step S2.6, phase 2. Set the learning rate of the neural network to 0.01.

[0044] Step S2.7, phase 2 teacher network training process. The training set data is sent into the neural network, and the prediction result for the current sample is obtained when it reaches the teacher classification exit. Update the teacher network using the label, and the label loss is defined as Update the parameters of the network using the stochastic gradient descent method SGD.

[0045] Step S2.8, introduce early exit student network, network parameters come from phase 1. Construct self-distillation structure. Freeze the parameters of the teacher network.

[0046] Step S2.9, Phase 2 student network backpropagation process. The student network is supervised by the teacher network loss, defined as At the same time, the student network is supervised by the label loss The network parameters are updated using the stochastic gradient descent method, SGD.

[0047] Step S2.10, loop S2.6~S2.9, update round is 125.

[0048] Step S2.11, Phase 3. Set the neural network learning rate to 0.001.

[0049] Step S2.12, Phase 3 teacher network training process. The training set data is sent into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification exit. The teacher network is updated using the label, and the label loss is defined as The network parameters are updated using the stochastic gradient descent method, SGD.

[0050] Step S2.13, introduce early exit student network, network parameters from phase 2. Construct self-distillation structure. Freeze the teacher network parameters.

[0051] Step S2.14, Phase 3 student network backpropagation process. The student network is supervised by the teacher network loss, defined as At the same time, the student network is supervised by the label loss The network parameters are updated using the stochastic gradient descent method, SGD.

[0052] Step S2.15, loop S2.11~S2.14, update round is 125.

[0053] Step S2.16, Phase 4, teacher network training process.

[0054] Step S2.17, the training set data is sent into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification exit. The teacher network is updated using the label, and the label loss is defined as The network parameters are updated using the stochastic gradient descent method, SGD.

[0055] Step S2.18, introduce early exit student network, network parameters from phase 3. Construct self-distillation structure. Freeze the teacher network parameters.

[0056] Step S2.19, Phase 2 student network backpropagation process. The student network is supervised by the teacher network loss, defined as At the same time, the student network is supervised by the label loss The network parameters are updated using the stochastic gradient descent method, SGD.

[0057] Step S2.20, loop S2.16-S2.19, update round is 125.

[0058] Step S3, prediction process

[0059] Step S3.1, pruning the neural network, only keeping the shallow branch prediction exit.

[0060] Step S3.2, input the picture to be predicted into the model to obtain a classification result.

[0061] The present application considers self-distillation from the perspective of dynamic teacher networks. First, a shallow student network with unchanged complexity is defined, and then the depth of the main network is extended to construct the teacher network in the self-distillation structure. The influence on the student as the teacher network gradually becomes complex is verified.

[0062] The above is only a specific implementation in the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can understand and think of the transformation or replacement within the technical range disclosed by the present application, which should be covered in the inclusive scope of the present application, therefore, the inclusive scope of the present application should be subject to the protection scope of the patent right book.

Claims

1. A self-distillation classification method based on progressive knowledge transfer, characterized by The data preprocessing, model training and prediction processes are as follows: Step S1, preprocessing the training data set Step S1.1, using flipping and cropping to enhance the training data Step S1.2, randomly shuffling the data and dividing it into different batches, with 128 data samples in each batch; Step S2, training of the neural network Step S2.1, phase 1, setting the neural network learning rate to 0.1; Step S2.2, phase 1 teacher network training process; the training set data is sent into the neural network, and the prediction result for the current sample is obtained when reaching the teacher classification outlet; the teacher network is updated using the label, and the label loss is defined as The parameters of the network are updated using the stochastic gradient descent method SGD. Step S2.3, introducing the early exit student network; constructing the self-distillation structure; freezing the teacher network parameters; Step S2.4, phase 1 student network backpropagation process; the student network is supervised by the teacher network with a loss defined as At the same time, the student network is supervised by the label loss The parameters of the network are updated using the stochastic gradient descent method, SGD. Step S2.5, repeating S2.1-S2.4, with 125 update rounds; Step S2.6, phase 2, setting the neural network learning rate to 0.01; Step S2.7, phase 2 teacher network training process; send the training set data into the neural network, get the prediction result for the current sample when reaching the teacher classification exit; update the teacher network using the label, and the label loss is defined as Update the parameters of the network using the stochastic gradient descent method SGD. Step S2.8, introducing the early exit student network, with network parameters from phase 1; constructing the self-distillation structure; freezing the teacher network parameters; Step S2.9, phase 2 student network backpropagation process; the student network is supervised by the teacher network with a loss defined as At the same time, the student network is supervised by the label loss The parameters of the network are updated using the stochastic gradient descent method, SGD. Step S2.10, repeating S2.6-S2.9, with 125 update rounds; Step S2.11, phase 3, setting the neural network learning rate to 0.001; Step S2.12, phase 3 teacher network training process; send the training set data into the neural network, get the prediction result for the current sample when reaching the teacher classification exit; update the teacher network using the label, and the label loss is defined as Update the parameters of the network using the stochastic gradient descent method SGD. Step S2.13, introducing the early exit student network, with network parameters from phase 2; constructing the self-distillation structure; freezing the teacher network parameters; Step S2.14, phase 3 student network backpropagation process; the student network is supervised by the teacher network with a loss defined as At the same time, the student network is supervised by the label loss The parameters of the network are updated using the stochastic gradient descent method, SGD. Step S2.15, repeating S2.11-S2.14, with 125 update rounds; Step S2.16, phase 4, setting the neural network learning rate to 0.0001; Step S2.17, phase 4 teacher network training process; send the training set data into the neural network, get the prediction result for the current sample when reaching the teacher classification outlet; update the teacher network using the label, and the label loss is defined as Update the parameters of the network using the stochastic gradient descent method SGD; Step S2.18, introducing the early exit student network, with network parameters from phase 3; constructing the self-distillation structure; freezing the teacher network parameters; Step S2.19, phase 2 student network backpropagation process; the student network is supervised by the teacher network with a loss defined as At the same time, the student network is supervised by the label loss The parameters of the network are updated using the stochastic gradient descent method, SGD. Step S2.20, repeating S2.16-S2.19, with 125 update rounds; Step S3, prediction process Step S3.1, cropping the neural network, only keeping the shallow branch prediction exit; Step S3.2, inputting the picture to be predicted into the model to obtain the classification result.

Images