An image classification model compression method based on deep Bregman divergence knowledge distillation

Abstract

The application relates to an image classification model compression method based on knowledge distillation of deep Bregman divergence. The method based on deep Bregman divergence is generally used for measuring the similarity of teacher and student models in the horizontal direction, and is now extended to a kind of measurement of the teacher or student network under different distributions. The student and teacher networks share parameters under different distributions, and the knowledge sensitivity of the teacher network to image classification is deeply mined, so that longitudinal learning is realized. The method firstly designs a universal feature migration knowledge distillation strategy, and then connects a Bregman loss module: in different deep Bregman networks, the Bregman divergence matrices of the teacher and the student under different distributions are respectively extracted, the difference between the two is calculated, the parameters of the student network are optimized through back propagation by using the distillation loss and the Bregman loss, and the classification accuracy of the student network model is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model compression, and particularly relates to an image classification model compression method based on deep Bregman divergence knowledge distillation. BACKGROUND

[0002] With the rise and development of deep neural networks, deep learning has opened up new paths in computer vision, natural language processing, recommendation systems and other fields, and has made rapid progress, especially in the supervised learning field. Even some results have almost exceeded human performance. However, supervised learning relies too much on large data sets: first, the acquisition of data sets sometimes involves private data that is not open to the public, second, the authenticity of the data, and most importantly, the annotation of the data set itself is very labor-intensive, material-intensive and capital-intensive, which greatly limits the development of artificial intelligence.

[0003] Although deep neural networks have gradually shown excellent performance in image classification tasks. However, deep learning is usually accompanied by a large number of network weights and parameters, complex models and a large amount of computational consumption, which is not suitable for small and micro electronic devices. Therefore, in order to facilitate deployment on mobile terminals, it is necessary to compress the model while maintaining considerable performance, so as to promote the development of neural networks in terminal inference, interpretability, training, optimization and other aspects.

[0004] Knowledge distillation uses a complex deep network model to transfer knowledge to a small network, which not only enables cross-domain and cross-modal learning, avoiding direct exposure of sensitive private data, but also reuses existing model resources to guide the new training phase. Knowledge distillation technology is one of the important technologies of model compression and is widely used.

[0005] Current model compression techniques include pruning, quantization, knowledge distillation, parameter sharing, and parameter matrix approximation. Among them, knowledge distillation is applied to various computer vision tasks, and the main method is to use a well-performing mature teacher network to guide the learning process of a student network. For the student network, it is a process of imitation, and what to imitate and how to imitate are two core problems of knowledge distillation. This process is usually achieved by reusing teacher network weights or measuring the difference between final outputs or hidden features as auxiliary loss. This forced approximation training is prone to overfitting in the student network, resulting in poor generalization ability of the network. Moreover, the accuracy of the trained student network model can be higher.

[0006] Therefore, a compression method is needed that can maintain performance after model compression and enable the student network to have a certain generalization ability. We propose an image classification model compression method based on deep Bregman divergence knowledge distillation. SUMMARY

[0007] Based on the higher requirement of the accuracy of the student network model after compression, and the generalization ability of the student network model, the application provides a deep Bregman divergence-based knowledge distillation image classification model compression method, and the technical scheme is as follows:

[0008] The deep Bregman divergence-based knowledge distillation image classification model compression method provided by the application, wherein the teacher network model is a Resnet series or a Vgg series, and the data set is CIFAR10 or CIFAR100.

[0009] Firstly, the teacher network model is trained, and the parameters are set as follows: the learning rate is 0.05, the training round is 240, the batch size is 64, the learning rate decay ratio is 0.1, the learning rate decay round is 150, 180 and 210, the weight decay is 0.0005, the momentun is set to 0.9, the data set is loaded and the model is initialized, wherein the loss function is cross entropy loss, and the weight parameters of each layer of the teacher model are saved after training;

[0010] Secondly, the student network is trained: the data set is loaded, the parameters of each layer of the teacher and student model Resnet are loaded, there are 3 blocks, each block is composed of 6 residual modules,

[0011] Each block module is connected by two relu and convolution operations; the feature maps of the three aligned middle layers of the teacher and student network models (that is, aligned once after each block) are extracted as the input of the deep Bregman network;

[0012] The output after sequentially passing through the three groups of block modules is subjected to bn operation, relu activation and average pooling, and finally a fully connected layer is connected to obtain the output y_t of the final teacher and student network model and y_s, we use the method of "letting the output y_s of the student model approach the output y_t of the teacher model" to set the loss function, so as to achieve "the parameters of the student network model approach the parameters of each layer of the teacher network model", and then realize "the classification accuracy of the student network model can be comparable to the performance of the teacher completed model", for this purpose, we set the distillation loss L_Breg;

[0013] The distillation loss function involves two kinds of labels: the soft label corresponds to the probability distribution of the teacher model output, that is, the result of the softmax activation of the output layer of the teacher model after T temperature distillation, that is, the following q tThe hard label is the original one-hot label, that is, labels in the formula L_kd, that is, the real label of image classification. The specific distillation loss function is as follows, wherein a is a coefficient, which is set to 0.9, ce is a cross-entropy loss function, y_s represents a feature map of an output layer of a student network, and q S is a result of the output layer of the student model after T temperature distillation and softmax activation, as follows:

[0014]

[0015] L_kd=(1-α)*ce(y_s,labels)+α*l_kl(q s ,q t )

[0016] In the above formula, T represents a distillation temperature, which is set to 4; the formula l_kl is related to the kl divergence formula as follows:

[0017]

[0018]

[0019] Wherein N is the number of data;

[0020] The distillation learning of the output layer is as described above; in addition, we also have knowledge learning of the intermediate layer, because we have three groups of blocks, so we can obtain the results after each block as the intermediate layer, learn it through the Bregman divergence, and set the deep Bregman network;

[0021] We implement the deep Bregman network on the multilayer perceptron, which contains two subnetworks. Each subnetwork is a multilayer perceptron, which contains 128, 32, and 1 hidden nodes, respectively activates the linear layer, and then connects a batch normalization layer, and finally splices, transposes, and unifies the dimensions of the batch normalization results O1 and O2 of the two subnetworks to obtain the Bregman divergence matrix d, and then performs batch processing normalization. Each subnetwork has its independent weight set, and the teacher and student networks share a weight parameter of the Bregman network;

[0022] Feature maps of the intermediate layers of the teacher and student network models The feature maps of the intermediate layers of the teacher and student network models (i represents the order of alignment) pass through two subnetworks Z1 and Z2 in the deep Bregman network, respectively, so as to obtain the deep Bregman divergence of the teacher and student network models under different distributions, and synthesize divergence matrices d t and d s ;

[0023] The loss value of the teacher divergence matrix and the corresponding student divergence matrix is calculated, and a plurality of losses are summed, that is, the loss value of the teacher divergence matrix and the corresponding student divergence matrix is calculated by the following formula:

[0024]

[0025] In the above formula respectively represent the Bregman divergence matrix of the student-teacher network under different Bregman network distributions, and the Bregman loss L_breg between the teacher and student network models is obtained by using the Euclidean norm;

[0026] In addition, the number of alignment for different networks may be different, and the network characteristics need to be considered specifically, for example, in the Resnet series of network models, alignment for 3 times can significantly improve the classification accuracy, and in the Vgg series of models, 5 times of alignment may be needed to achieve better experimental performance.

[0027] Finally, the Bregman loss is multiplied by the weight coefficient and added to the distillation loss as the final loss, as follows:

[0028] loss=L_kd+β*L_breg

[0029] Where beta is an adjustable coefficient; the parameters of the student network model are updated using SGD, and the above steps are repeated to train the student network model to converge quickly, and finally a student network model with fewer layers and parameters than the teacher model and better performance than the teacher network model can be obtained, realizing effective compression of the image classification model. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the overall flowchart of the present application

[0031] Figure 2 is the deep Bregman divergence flowchart in the present application

[0032] Figure 3 is the setting of the deep Bregman network in the present application

[0033] Figure 4 is the knowledge distillation flowchart in the present application DETAILED DESCRIPTION

[0034] The embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are only used to explain the present application and cannot be interpreted as a limitation on the present application;

[0035] The basic idea of the application is to guide the classification accuracy of the student network model with the preserved and trained teacher network model with high accuracy, extract the output of the teacher model and the student model and the deep Bregman divergence matrix under different distributions respectively, calculate the distillation loss and the deep Bregman loss, update and optimize the student network model by combining the two, realize model compression and performance improvement;

[0036] Please refer to Figure 1 The application provides a deep Bregman divergence-based knowledge distillation image classification model compression method, which applies a deep Bregman network for distillation learning in the middle layer and applies a knowledge distillation method for learning a teacher model in the output layer, the teacher model is Resnet56, the student model is Resnet20, and the data set is CIFAR100, including the following steps:

[0037] Step 1, train the teacher network model with a data set with a divided training set and test set, set the parameters: learning rate is 0.05, training rounds are 240, batch size is 64, learning rate decay ratio is 0.1, learning rate decay rounds are 150, 180 and 210, weight decay is 0.0005, momentun is set to 0.9, load the data set and initialize the model, wherein the loss function is cross-entropy loss, save the weight parameters of each layer of the teacher model after training;

[0038] Step 2, refer to Figure 4 , train the student network: load the data set, load the parameters of each layer of the teacher and student models Resnet, a total of 3 blocks, each block is composed of 6 residual modules, each residual module has two convolution layers, and each block module is connected by two relu and convolution operations; refer to Figure 2 , extract the feature maps of the three aligned middle layers of the teacher and student network models respectively (that is, align once after each block), as the input of the deep Bregman network;

[0039] Step 3, refer to Figure 4 , perform bn operation, relu activation and average pooling on the output after passing through the three block modules in succession, and finally connect a fully connected layer to obtain the final output y_t and y_s of the teacher and student network models, we use the method of "letting the output y_s of the student model approximate the output y_t of the teacher model" to set the loss function, so as to achieve "the parameters of the student network model are close to the parameters of each layer of the teacher network model", and then realize "the classification accuracy of the student network model can be comparable to the performance of the teacher model", for this purpose, we set the distillation loss L_Breg;

[0040] Two kinds of labels are involved in the distillation loss function: the soft label corresponds to the probability distribution output by the teacher model, that is, the result of the softmax activation of the teacher model output layer after T temperature distillation, that is, q t , and the hard label is the original one-hot label, that is, the labels in the L_kd formula, that is, the real label of image classification. The specific distillation loss function is as follows, where a is a coefficient, set to 0.9, ce is a cross-entropy loss function, y_s represents the feature map of the output layer of the student network, q S is the result of the softmax activation of the student model output layer after T temperature distillation, as follows:

[0041]

[0042] L_kd=(1-α)*ce(y_s,labels)+α*l_kl(q s ,q t )

[0043] In the above formula, T represents the distillation temperature, which is set to 4; the l_kl formula is related to the kl divergence formula as follows:

[0044]

[0045] Where N is the number of data;

[0046] Step 4, as mentioned above, the distillation learning of the output layer; In addition, we also have knowledge learning of the intermediate layer, refer to Figure 2 Because we have three groups of blocks, we can get the results after each block as the intermediate layer, learn it through Bregman divergence, and set the deep Bregman network;

[0047] Step 5, refer to Figure 3 We implement a deep Bregman network on a multilayer perceptron, which contains two subnetworks. Each subnetwork is a multilayer perceptron containing 128, 32, and 1 hidden nodes, respectively, activating the linear layer therein, and then connecting a batch normalization layer to each. Finally, the batch normalization results O1 and O2 of the two subnetworks are spliced, transposed, and unified in dimension to obtain the Bregman divergence matrix d, and then batch processing normalization is performed. Each subnetwork has its own independent weight set, and the teacher and student networks share a Bregman network weight parameter;

[0048] Step 6, refer to Figure 3 The feature maps of the intermediate layers of the teacher and student network models (i represents the order of alignment) through two sub-networks Z1 and Z2 in the deep Bregman network respectively, so as to obtain the deep Bregman divergences of the teacher and student network models under different distributions, and to synthesize Bregman divergence matrices respectively and (i represents the order of alignment).

[0049] (1) First alignment: record the intermediate results of input through the first block of the teacher and student models respectively input into the Bregman network of net0, and after passing through two sub-networks Z1 and Z2, d 0 , and the Bregman divergence matrices of the student and teacher models are distinguished.

[0050] (2) Second alignment: record the intermediate results of input through the second block of the teacher and student models respectively input into the Bregman network of net1, and after passing through two sub-networks Z3 and Z4, d 1 , and the Bregman divergence matrices of the student and teacher models are distinguished.

[0051] (3) Third alignment: record the intermediate results of input through the third block of the teacher and student models respectively input into the Bregman network of net2, and after passing through two sub-networks Z5 and Z6, d 2 , and the Bregman divergence matrices of the student and teacher models are distinguished.

[0052] The sub-networks Z1, Z2, Z3, Z4, Z5 and Z6 are the same, but because the information features contained in the intermediate layers of each alignment are different, the Bregman network shared by the teacher and student in the first alignment is net0, the Bregman network shared by the teacher and student in the second alignment is net1, and the Bregman network shared by the teacher and student in the third alignment is net2, but their internal structures are the same, as described in step 5.

[0053] Step 7, calculate the loss value of the teacher Bregman divergence matrix and the corresponding student Bregman divergence matrix, and sum multiple losses: that is, the loss value of the teacher Bregman divergence matrix and the corresponding student Bregman divergence matrix is calculated by the following formula:

[0054]

[0055] In the above formula respectively represent the Bregman divergence matrix of the student-teacher network under different Bregman network distributions, and the Bregman loss L_breg between the teacher and student network models is obtained by using the Euclidean norm;

[0056] Step 8, finally let the Bregman loss multiply the weight coefficient and the distillation loss add as the final loss, and then back propagation is carried out on the student network to optimize the network parameters, as follows:

[0057] loss=L_kd+β*L_breg

[0058] Wherein, beta is an adjustable coefficient, and the outputs L_kd and L_breg are adjusted according to the size of L_kd, so that the term beta*L_breg can play the role of a regular term, and after tuning, when beta is 0.01, the effect is best, and the accuracy rate reaches 70.8;

[0059] In summary, the image classification model compression method based on the deep Bregman divergence knowledge distillation disclosed by the application can make the student network learn the Bregman divergence of the teacher network to different distributions, combined with knowledge distillation, so as to improve the efficiency of the knowledge distillation method and further improve the performance of the student network.

[0060] It should be emphasized that the embodiments described in the application are illustrative rather than limiting, and therefore the application includes but is not limited to the embodiments described in the specific embodiments, and any other embodiments derived by those skilled in the art according to the technical solutions of the application also belong to the protection scope of the application.

Claims

1. A method for compressing image classification models based on knowledge distillation using deep Bregman divergence, characterized in that, include: Step 1: Train the teacher network model using the dataset that has been divided into training and test sets. Set parameters: Load the dataset and initialize the model. Step 2: After training, save the weight parameters of each layer of the teacher model; Step 3, train the student network: Load the dataset, load the parameters of each layer of the teacher and student ResNet models, and extract the feature maps of the three aligned intermediate layers of the teacher and student network models as input to the deep Bregman network; Step 4: Record the final logit outputs y_t and y_s of the teacher and student network models. Use the method of "making the output y_s of the student model approximate the output y_t of the teacher model" to set the loss function, so as to achieve "the parameters of the student network model are close to the parameters of the teacher network model", and thus achieve "the classification accuracy of the student network model can be comparable to the performance of the teacher network model". For this purpose, set the distillation loss L_Breg. The final logit outputs y_t and y_s of the teacher and student network model in step 4 are described in detail below: The output after passing through three consecutive block modules is subjected to batch normalization, ReLU activation, and average pooling, and finally connected to a fully connected layer to obtain the final outputs y_t and y_s of the teacher and student network models. The distillation loss L_kd in step 4 is described in detail below: The distillation loss function involves two types of labels: the soft label corresponds to the probability distribution of the teacher model's output, which is the result of the teacher model's output layer being activated by softmax after distillation at temperature T, as shown in the q below. t The hard label is the original one-hot label, which is the labels in the L_kd formula, i.e., the true labels for image classification. The specific distillation loss function is as follows, where α is a coefficient, set to 0.9, ce is the cross-entropy loss function, y_s represents the feature map of the student network output layer, and q S The result of the student model's output layer being activated by softmax after distillation at temperature T is as follows: L_kd=(1-α)*ce(y_s,labels)+α*l_kl(q s ,q t ) In the above formula, T represents the distillation temperature, which is set to 4; the relationship between the l_kl formula and the kl divergence formula is as follows: Where N is the number of data; Step 5: Extract the feature maps of the intermediate layers of the teacher and student network models. By inputting two subnetworks Z1 and Z2 from the deep Bregman network, the deep Bregman divergence of the teacher and student network models under different distributions is obtained, and the Bregman divergence matrix is ​​synthesized. and i represents the alignment order; Step 6: Calculate the loss value of the teacher's Bregman scatter matrix and the corresponding student's Bregman scatter matrix, sum the multiple losses, and calculate the loss value L_breg of the teacher's Bregman scatter matrix and the corresponding student's Bregman scatter matrix. Step 7: Finally, multiply the Bregman loss by the weight coefficient and sum it with the distillation loss to obtain the final loss, and then backpropagate to optimize the network parameters of the student network.

2. The image classification model compression method based on knowledge distillation using deep Bregman divergence according to claim 1, characterized in that: The teacher / student network models are ResNet and VGG series networks, and the datasets are CIFAR10 and CIFAR100.

3. The image classification model compression method based on knowledge distillation using deep Bregman divergence according to claim 1, characterized in that: The specific description of the model parameter settings in step 1 is as follows: The learning rate is 0.05, the number of training epochs is 240, the batch size is 64, the learning rate decay rate is 0.1, the learning rate decay epochs are 150, 180, and 210, the weight decay is 0.0005, the momentun is set to 0.9, and the loss function is cross-entropy loss.

4. The image classification model compression method based on knowledge distillation using deep Bregman divergence according to claim 1, characterized in that: The student model in step 3 is described in detail below: Load the parameters of each layer of the ResNet model for teachers and students. There are 3 blocks in total. Each block consists of 6 residual modules. Each residual module has two convolutional layers. Each block module is composed of two ReLU convolution operations connected together.

5. The image classification model compression method based on knowledge distillation using deep Bregman divergence according to claim 1, characterized in that: in, The specific setup for the deep Bregman network in step 5 is described below: A deep Bregman network was implemented on a multilayer perceptron, containing two subnetworks. Each subnetwork is a multilayer perceptron with 128, 32, and 1 hidden nodes, respectively. The linear layers within each subnetwork are activated, and then each subnetwork is connected to a batch normalization layer. Finally, the batch normalization results O1 and O2 of the two subnetworks are concatenated, transposed, and have their dimensions unified to obtain the Bregman divergence matrix d, which is then batch normalized. Each subnetwork has its own independent set of weights, while the teacher and student networks share the weight parameters of a single Bregman network.

6. The image classification model compression method based on knowledge distillation using deep Bregman divergence according to claim 1, characterized in that: in, The L_breg loss in step 6 is described in detail below: In the above formula Let L_breg represent the Bregman divergence matrices of the student and teacher networks under different Bregman network distributions, respectively. The Bregman loss L_breg between the teacher and student network models is obtained using Euclidean paradigm.

7. The image classification model compression method based on knowledge distillation of deep Bregman divergence according to claim 1, characterized in that: in, The final loss in step 7 is described in detail below: Loss=L_kd+β*L_breg Where β is an adjustable coefficient with a value of 0.01, and the outputs are L_kd and L_breg.

Images