A knowledge distillation method and system based on multi-student discussion

Abstract

The application discloses a kind of knowledge distillation method and system based on multi-student discussion.The steps of the method include:1) selecting complex network ResNet32x4 as the teacher model of knowledge distillation, pretraining is carried out to the teacher model;2) knowledge distillation is carried out, and the distillation mode of single teacher and multiple students is adopted, the parameters of multiple small student networks are initialized and independently trained, and the knowledge from the teacher network is learned respectively;3) with the aid of discussion module, student networks are discussed with each other, the logits output of each student model is used as its input, a multi-layer convolutional neural network is used, the outputs of each student network are coupled together, and the final class prediction is output;4) the image to be classified is input into the student network, and the final image classification result is obtained after the discussion between students.The application greatly improves the accuracy of image classification, and improves the situation that the expression ability difference of teacher-student model in the field of knowledge distillation is large.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a knowledge distillation method and system based on multi-student discussion. BACKGROUND

[0002] With the improvement of computing power and the widespread appearance of large-scale data sets, deep models have achieved great success, especially in image and speech recognition tasks. However, most deep learning models contain a large number of parameters, and deep and wide models require a large amount of computing resources during training, and there is still a high storage and computing requirement when deploying the model. Therefore, in order to obtain faster computing speed, the compression of deep models has become a research hotspot in recent years. Among them, knowledge distillation is an effective method for model compression, which aims to compress complex models or model sets into smaller models for deployment. When the complex model is trained well, it can be used to guide the learning of the lightweight model, so as to be applied to real-time scenarios. In recent years, knowledge distillation has been widely applied in the field of natural language processing and computer vision.

[0003] Specifically, knowledge distillation (Knowledge Distillation) aims to transfer knowledge from a complex deep model (teacher model) to a lightweight model (student model). Generally speaking, the former has stronger learning and representation ability, and higher performance, while the latter has low computational complexity, which is convenient for deployment on edge devices. Hinton et al. first proposed the concept of knowledge distillation in 2015. In order to transfer the implicit knowledge between models, the target of the student model is set to minimize the Kullback-Leibler (KL) divergence of the teacher output and the student output. The student model improves its performance by imitating the soft target of the teacher model. The reason why this soft target distillation method is effective is that the teacher model provides rich information for the training of the student model by assigning different classes with relative probabilities.

[0004] The above soft target-based distillation belongs to an important method of traditional knowledge distillation, and in recent years, many different methods have appeared to try various forms of knowledge transfer. For example, by learning the intermediate representation of the teacher model, the problem solving process, the attention map, the structural relationship, the similarity of the activation map, etc. to promote the optimization process of the student network. However, all these methods extract knowledge from a single teacher model to the student network, which can lead to the single and even biased knowledge learned. In order to further improve the performance of the student network when deployed, some recent researches propose to use multiple teacher models during distillation. The simplest idea is to directly use the mean of multiple teacher soft targets as the guidance for the student network learning, and each teacher network is assigned the same weight. Further, the weighted average of multiple teacher models can be used to guide the training of the student network, where the weight is a hyperparameter that remains unchanged during the training process, and different teacher networks are assigned different weights. In order to further optimize the assignment of weights, some scholars propose to use reinforcement learning to dynamically assign weights to teacher models to optimize the performance of student models. In addition to research based on weights, some scholars propose a noise-based regularization method to simulate the learning of multiple teachers. In addition, the distillation architecture of multi-teacher teaching can also be combined with different tasks to solve different problems. For example, the domain adaptation task can be studied by associating each source domain with a teacher for future distillation learning. In the field of multi-task learning, multi-teacher learning can be applied to multi-task learning, where each teacher corresponds to a task.

[0005] In summary, knowledge distillation, as an effective deep neural network compression and acceleration technique, has been widely applied in different fields of artificial intelligence, including computer vision, speech recognition, natural language processing, and recommendation systems, etc. In the field of computer vision, knowledge distillation aims to provide efficient and effective teacher-student learning for various visual recognition tasks, so that lightweight student networks can be deployed on edge devices. However, due to the different sizes of teacher models and student models, there is a large gap in their representation capabilities, resulting in the performance of student models still being not high. SUMMARY

[0006] In order to overcome the problem of knowledge representation gap between teachers and students existing in the existing knowledge distillation method, the purpose of the present application is to provide a knowledge distillation method based on multi-student discussion, further improving the performance of student networks.

[0007] The present application firstly utilizes different student networks to generate different predictions for the same image, and then inputs these results into a discussor network, and after discussion, different network consensus is reached, and finally an accurate prediction for the category to which the image belongs is obtained. Specifically, in order to generate diversified and excellent student networks, firstly, a plurality of student networks are trained by means of knowledge distillation of a powerful teacher network, and then the prediction results generated by the plurality of student networks are input into a discussor (discussion module), and then the discussor is trained for image classification, so that the discussor also serves as an image classifier to output the final prediction result. Due to the addition of the discussion stage, the advantages of the students can be complementary, and the gap in representation ability between the single teacher and student networks is better solved, and finally more accurate classification results are obtained.

[0008] The technical solutions adopted by the present application are as follows:

[0009] A knowledge distillation method based on multiple student discussions, the steps of which include:

[0010] 1) Pre-training of the teacher network, the starting point of knowledge distillation is a good teacher network, generally speaking, a large-scale complex network needs to be selected as the teacher network, so as to better fit the complex data;

[0011] 2) Training of multiple student networks, a single teacher and multiple student distillation mode is adopted, the parameters of multiple small student networks are initialized respectively, and the knowledge from the teacher network is learned independently;

[0012] 3) Discussion among student networks, the discussion module completes the discussion function, taking the logits output of each student network as its input, adopting a multi-layer convolutional neural network, coupling the outputs of each student network together, and outputting the final category prediction.

[0013] 4) Inputting the image to be classified into the student network, and then obtaining the final classification result with higher accuracy through the discussion among the students.

[0014] Further, the stage 1) pre-training of the teacher network, a general knowledge distillation framework usually includes one or more large pre-trained teacher networks and a small student network. The teacher network is usually much larger than the student network, and needs to be pre-trained on a large data set in advance, and the present application preferably uses a ResNet32x4 network as the teacher network for pre-training on the CIFAR100 data set.

[0015] Further, the training of the stage 2) multiple student network, in the general knowledge distillation framework, usually contains a powerful teacher network and a smaller student network, the main idea is to train an effective student network under the guidance of the teacher network to obtain comparable accuracy. The supervision information from the teacher network, commonly known as the "knowledge" learned by the teacher network, can help the student network to imitate the behavior of the teacher network. But the traditional distillation mode only contains a single student network, which usually leads to poor deployment effect due to model overfitting and insufficient model expression ability. Therefore, the present application adopts the distillation mode of single teacher and multiple students, solves the problem of insufficient expression ability caused by the distillation architecture of single student, and reduces the performance gap between the teacher network and the student network.

[0016] Further, the stage 3) student network discussion, based on the above steps, a plurality of well-trained student networks are obtained, and these networks are usually diversified. In order to give full play to the advantages of each student network and make up for each other in the deployment stage, the present application further designs a discussion module. The discussion module takes the logits output of each student network as its input, adopts the architecture of convolutional neural network (CNN), couples the outputs of each student network together, utilizes the characteristics of convolutional layer, pooling layer and nonlinear activation contained in the CNN network, captures the advantages of each student network, and finally outputs more accurate class prediction.

[0017] Further, the stage 4) obtains the prediction result of the image to be classified. After the training of the above stages 2) and 3), the complete network architecture and the matching network parameters have been obtained. In order to obtain good image classification effect on the lightweight student network, the present application adopts the network structure combined with the distilled multiple student network and the discussion module in this stage, takes the image to be classified as the input, the image is input into the multiple student network to obtain diversified intermediate results, and then the intermediate results are input into the discussion module. The prediction result output by the discussion module is the final classification result. Experimental verification shows that the classification result obtained by the above process is better than other methods in performance.

[0018] Further, for the distillation process of each student network in the stage 2), the method of transferring dark knowledge in the present application is as follows: define the logits vector of the output of the last fully connected layer of the teacher deep network as z, let z i The input belonging to the i-th class probability p i can be estimated by the softmax function:

[0019]

[0020] Therefore, the prediction of the soft target obtained by the teacher network contains dark knowledge, which can be used as supervision information to transfer knowledge from the teacher network to the student network.

[0021] Further, a temperature factor T is introduced to control the importance of each soft target:

[0022]

[0023] Where a higher temperature will produce a softer probability distribution. Specifically, when T→∞, all classes share the same probability. When T→0, the soft target becomes a one-hot label, i.e., a hard target. The soft target from the teacher network and the real label are both very important for improving the performance of the student network, serving as the distillation loss and the student loss respectively to guide the optimization of the student network.

[0024] Preferably, the knowledge distillation method based on multi-student discussion of the present application mainly comprises the following steps:

[0025] 1) Using a conventional image classification dataset CIFAR100 as the training dataset, a complex network is selected as the teacher network for knowledge distillation, and the teacher network is pre-trained;

[0026] 2) Training of the multi-student network, different student networks are adopted, including ResNet20, ResNet32, ResNet8x4, and logits distillation form is adopted in the process of distillation of each student;

[0027] 3) Discussion of the student networks, the discussion module adopts a multi-layer convolutional neural network, specifically including a convolutional layer, a pooling layer, a fully connected layer, and a nonlinear activation function, the logits of multiple students are input into the discussion module to generate the classification result of the picture;

[0028] 4) The image to be classified is input into the trained student network, and the trained discussion module outputs the final classification result with higher accuracy.

[0029] Further, in the pre-training stage of the teacher network of step 1), a typical image classification loss, i.e., a cross entropy (Cross Entropy) loss, is used to train the ResNet32x4 teacher network, and the formal representation of the cross entropy loss is:

[0030]

[0031] Where p(x) represents the real label distribution in one-hot form of the image, and q(x) represents the distribution predicted by the teacher network.

[0032] Further, step 2) distillation of multiple student networks, in order to increase the diversity of student networks, the present application selects small student networks with different architectures, and the parameters of each network are initialized respectively. In the distillation process of student networks, each network learns knowledge from the teacher network respectively, so the training process is independent of each other. On the one hand, independent training makes the diversity among students further enhanced, on the other hand, independent training allows multiple student networks to train in parallel, greatly shortening the training time. Further, the above-mentioned teacher network is only one, and its knowledge is shared by student networks, which greatly simplifies the pre-training process of the teacher network in step 1), reduces the amount of calculation and the occupation of storage space. Further, the optimization process of each student network adopts the knowledge representation of the typical knowledge distillation method, that is, the student imitates the logits of the teacher, further, the logits refer to the output of the last layer in the deep neural network, which is regarded as the carrier of the knowledge from the teacher network. Therefore, the distillation loss of each student network is defined as the cross entropy between the teacher network and the student network, that is:

[0033]

[0034] Where z t and z s are logits of the teacher network and the student network respectively. Formula (4) makes the logits of the teacher network match the logits of the student network.

[0035] The classification loss of the student network itself is the cross entropy between the real label and the output result of the student network, and its formula is represented as:

[0036]

[0037] Where L S represents the classification loss of the student network itself, L CE represents the cross entropy loss between the output of the student network and the real label, and y is the vector of ground truth (real label), where only one element is 1, representing the ground truth label of the transfer training sample, and the other elements are 0. In the distillation loss and the classification loss of the student itself, the same logits of the student network are used, but the temperature coefficients are different. The temperature coefficient in the student classification loss is T = 1, and the temperature coefficient in the distillation loss is T = t, which is uniformly taken as 4 in the present application. Therefore, the final loss of each student in the multiple student knowledge distillation is the combination of the distillation loss and the classification loss:

[0038] L(x,W)=α*L D (p(z t ,T),p(z s ,T))+(1-α)*LS (y, p(z s , T)) (6)

[0039] where x is the training input on the training set in CIFAR100 dataset, W is the parameter of the student network, and alpha is the adjustment parameter for balancing the weight of the two.

[0040] Further, the step 4) is that the final input picture into the network is from the test dataset of CIFAR100, and the picture is mapped through multiple student networks and the discussion module, and finally the classification result is obtained.

[0041] The application also provides a knowledge distillation system based on multiple student discussions adopting the above method, which comprises:

[0042] a teacher network pre-training unit configured to pre-train a teacher network;

[0043] a multiple student network training unit configured to adopt a single-teacher and multiple-student distillation mode, and train multiple student networks by using the pre-trained teacher network for knowledge distillation;

[0044] a discussion module training unit configured to input image classification prediction results generated by the multiple student networks into the discussion module, and train the discussion module for image classification;

[0045] an image classification unit configured to input an image to be classified into the trained student network, and then into the trained discussion module, and output a final image classification result.

[0046] To sum up, the application designs a knowledge distillation method based on multiple student discussions, so that the student network obtained by distillation can produce more accurate classification results.

[0047] 1. The knowledge distillation mode of single teacher and multiple students is adopted, the number of distilled students is increased, and the selected student network architectures are different, which makes the intermediate result after distillation more diverse, and lays a foundation for the discussion module to make up for the shortcomings in the following;

[0048] 2. The application adopts a parallel training mode of each student network in the distillation process, each student network is independent and has no parameter sharing, so the total distillation time is the distillation time of the largest student network, which saves the time cost of model training to a certain extent;

[0049] 3、The design of the discussion module obviously improves the prediction performance. Experimental results show that the multi-student discussion distillation method of the present application always has better generalization ability than the classic teacher-guided one-to-one distillation method. Compared with other methods, greater student diversity and better discussion results are observed in the method of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of the knowledge distillation method based on multi-student discussion;

[0051] Figure 2 A single-teacher multi-student distillation training architecture diagram;

[0052] Figure 3 A multi-student discussion architecture diagram. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below through specific examples and drawings. The flowchart of the knowledge distillation method based on multi-student discussion of the present application is shown in Figure 1 , which is mainly divided into two stages of training stage and test stage.

[0054] The training stage is divided into three stages, and the steps are as follows:

[0055] 1) The first stage is to pre-train the teacher network, and ResNet32*4 network is selected as the teacher network, and CIFAR100 is used as the training data set.

[0056] The processing process of step 1) is: using all the data contained in the training set of CIFAR100 data set (100 image categories, each containing 500 32*32 size color maps) to train ResNet32*4 network to obtain a well-trained teacher network. The initialization of the above-mentioned teacher network adopts a random initialization strategy, and the training iterates a total of 240 epochs, the initial learning rate is 0.05, and then the learning rate is decayed by a factor of 0.1. The training process adopts the stochastic gradient descent method for parameter optimization, and the momentum is set to 0.9.

[0057] 2) The knowledge distillation stage adopts the single-teacher multi-student distillation mode, aiming to produce diversified intermediate results through distillation, and the training process is shown in Figure 2 .

[0058] The student network selection of step 2) requires that the size of the student network is small enough to be deployed on the edge device, and on the other hand, the student network should produce as diverse prediction results as possible so that the networks can learn from each other. Therefore, the student networks used by the present application are ResNet20, ResNet32 and ResNet8x4, all of which are much smaller in size than the teacher network. The structure of the student network is a stack of convolutional layers, BN layers and nonlinear activation layers of different scales, and at the end of the model, a mean pooling layer is connected to a fully connected layer as the output layer. To perform knowledge distillation, each student network calculates the cross-entropy loss between its last layer output and the last layer output of the teacher network as the distillation loss, and by minimizing the distillation loss, the student network learns the implicit knowledge from the teacher network output. Similar to step 1), the training data of the student network still uses the training set of the CIFAR100 dataset, and the parameter settings during training are the same as in step 1). In addition, since the objective function of the student network includes minimizing the distillation loss and the cross-entropy loss of classification, the weights of both are set to 1, and in order to facilitate comparison, the temperature coefficient of the distillation loss is set to 4 in the present application.

[0059] 3) Discussion module training stage, since the discussion module is also composed of a series of convolutional layers, pooling layers, fully connected layers and nonlinear activation functions, its parameters also need to be iteratively optimized according to the objective function, and the architecture of the discussion stage is as shown in Figure 3

[0060] The discussion module of step 3) is composed of two convolutional layers, one max pooling layer, a fully connected layer and a final fully connected output layer. The input of the first convolutional layer conv_1 is the result of concatenating the logits of the three student networks, with a size of 3x100, and the convolutional kernel size of conv_1 is 3x3. The convolutional kernel size of conv_2 is also set to 3x3, and the outputs of the subsequent two fully connected layers are set to 1024 and 100, respectively. Since the output of the discussion module is the final class prediction, a softmax function is used to calculate the class probability distribution. The objective function of the discussion module is defined as the cross-entropy loss between the model output and the label of the ground truth, and the optimized parameters only involve the discussion module and are not backpropagated to the student network. During training, the discussion module only needs 90 epochs to train to convergence due to its simple structure.

[0061] The steps of the test stage are as follows:

[0062] ​1) The test images are input in parallel into the trained multi-student network to obtain the logits of the last layer output of multiple student networks. All logits are then concatenated and used as input to the discussion module. The trained discussion module calculates and outputs the classification result corresponding to the test images. The test images used in this invention are from the CIFAR100 test dataset, which contains 100 images in 100 classes.

[0063] 2) Compare the final prediction results of the model with the labels of the test set itself, calculate the accuracy of the top 1 correct predictions, and use it as an evaluation of the classification performance of the model.

[0064] The knowledge distillation method based on multi-student discussion proposed in this invention has the following testing environment and experimental results:

[0065] (1) Test environment:

[0066] System environment: Ubuntu 16.04.5;

[0067] Hardware configuration: Memory: 24GB, GPU: NVIDIA Quadro P6000, Hard drive: 1TB.

[0068] (2) Experimental data:

[0069] Training data: The CIFAR100 dataset was used for training and testing respectively. The model was trained until it stabilized and the performance no longer improved.

[0070] Test data: CIFAR100 test dataset.

[0071] Assessment method: Online assessment.

[0072] (3) Experimental results:

[0073] The experimental results of this invention are compared with those of the traditional knowledge distillation method. The traditional knowledge distillation method uses a single-teacher, single-student distillation model, with the teacher network being ResNet32x4 and the student network being ResNet3. The specific knowledge representation also minimizes the difference in logits between the two. The test comparison results are shown in Table 1:

[0074] Table 1. Comparison of test results of the present invention

[0075] Method comparison Accuracy Conventional knowledge distillation methods 72.22 The present method 76.77

[0076] Based on the same inventive concept, another embodiment of the present invention provides a knowledge distillation system based on multi-student discussion, comprising:

[0077] The teacher network pre-training unit is used to pre-train teacher networks;

[0078] a plurality of student networks are trained by using the pre-trained teacher network to distill knowledge of the plurality of student networks in a single-teacher and multi-student distillation mode;

[0079] a discussion module training unit configured to input image classification prediction results generated by the plurality of student networks into the discussion module, and train the discussion module for image classification;

[0080] an image classification unit configured to input an image to be classified into the trained student network and the trained discussion module, and output a final image classification result.

[0081] The specific implementation process of each module can be found in the description of the method of the present application above.

[0082] Based on the same inventive concept, another embodiment of the present application provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present application.

[0083] Based on the same inventive concept, another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the present application.

[0084] The above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and the ordinary skilled in the art can modify or equivalently replace the technical solutions of the present application without departing from the spirit and scope of the present application, and the protection scope of the present application should be subject to the description of the claims.

Claims

1. A knowledge distillation method based on multi-student discussion, characterized in that, Includes the following steps: Pre-training of teachers' networks; A single-teacher, multi-student distillation model is adopted, using a pre-trained teacher network to train multiple student networks for knowledge distillation; the selected student networks have different architectures, resulting in diverse intermediate results after distillation; during the distillation process, each student network is trained in parallel, with each student network operating independently and without parameter sharing. The image classification prediction results generated by multiple student networks are input into the discussion module to train the discussion module for image classification. The image to be classified is input into the trained student network. Different student networks generate different predictions for the same image. The image is then processed by the trained discussion module. Through discussion, the different student networks reach a consensus and output the final image classification result. In the single-teacher, multi-student distillation mode, the distillation process for each student network includes: defining the logits vector of the output of the last fully connected layer of the teacher network as z, and letting z... i Let p represent the logit of the i-th class. Then, the probability p of the input belonging to the i-th class is... i It is estimated by the softmax function: Accordingly, the predictions of soft objectives obtained by the teacher network contain dark knowledge, which can be used as supervisory information to transfer knowledge from the teacher network to multiple student networks; and a temperature factor T is introduced to control the importance of each soft objective. Higher temperatures produce a softer probability distribution; when T→∞, all classes share the same probability; when T→0, soft targets become one-hot labels, i.e., hard targets; soft targets from the teacher network and real labels serve as distillation loss and student loss, respectively, to jointly guide the optimization of the student network. The distillation loss for each student network is defined as the cross-entropy between the matching teacher and student networks, i.e.: Where z t and z s These are the logits of the teacher network and the student network, respectively. This formula ensures that the logits of the teacher network match the logits of the student network. The classification loss of the student network itself is the cross-entropy between the true labels and the output of the student network, and its formula is expressed as: Where y is a vector of ground truth labels, with only one element being 1, representing the ground truth label of the transferred training sample, and the other elements being 0; the same logits of the student network are used in both the distillation loss and the student network's own classification loss, but the temperature coefficients are different. The temperature coefficient in the student network's classification loss is T = 1, and the temperature coefficient in the distillation loss is T = t. In multi-student knowledge distillation, the final loss of each student network is the joint of the distillation loss and the classification loss: L(x,W)=α*L D (p(z t ,T),p(z s ,T))+(1-α)*L S (y,p(z s ,T)) Where x is the training input on the training set in the CIFAR100 dataset, W is the parameter of the student network, and α is the adjustment parameter used to balance the weights of the two. The discussion module takes the logits output of each student network as its input, and uses the architecture of a CNN network to couple the outputs of each student network together. By utilizing the convolutional layers, pooling layers and non-linear activation characteristics of the CNN network, it captures the dominant class of each student network and finally outputs an accurate class prediction.

2. The method according to claim 1, characterized in that, The pre-training of the teacher network involves using a ResNet32x4 network as the teacher network and pre-training it on the CIFAR100 dataset.

3. The method according to claim 1, characterized in that, The multiple student networks include ResNet20, ResNet32, and ResNet8x4, and logits distillation is used in the individual distillation process of each student network.

4. A knowledge distillation system based on multi-student discussion employing the method described in any one of claims 1 to 3, characterized in that, include: The teacher network pre-training unit is used to pre-train teacher networks; The multi-student network training unit is used to train multiple student networks to perform knowledge distillation using a pre-trained teacher network in a single-teacher, multi-student distillation mode. The discussion module training unit is used to input the image classification prediction results generated by multiple student networks into the discussion module to train the discussion module for image classification. The image classification unit is used to input the image to be classified into the trained student network, and then through the trained discussion module to output the final image classification result.

5. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 3.

Images