A lightweight compression method of a deep convolutional network

Abstract

The application relates to a lightweight compression method of a deep convolutional network, comprising the following steps: performing simplification on the output of a first network model, performing pruning on the backbone network part of the first network model to obtain a second network model; the first network model is a trained deep convolutional network model; performing fine-tuning training on the second network model based on knowledge distillation and multi-resolution images; performing model pruning on the second network model to generate a third network model, and performing fine-tuning training on the third network model based on feature space alignment and knowledge distillation, and taking the trained third network model as a final model. The application can obtain a network model with low-resolution input, simplified output and lightweight structure while guaranteeing the detection accuracy and robustness of the network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a lightweight compression method of deep convolutional network. BACKGROUND

[0002] Although the network model based on deep learning has a significant advantage in detection accuracy compared with the traditional image processing-based method, it faces problems such as too large parameter quantity and too slow operation speed when deployed in mobile terminal or chip. Therefore, lightweight compression of deep convolutional network model has important application value.

[0003] At present, knowledge distillation is a commonly used deep convolutional network compression method, which uses a network with complex structure but better prediction performance as a teacher network to train a student network with simple structure, so that the student network finally achieves similar accuracy and robustness to the teacher network. The current mainstream knowledge distillation algorithm mainly faces the teacher and student networks with the same output and input, and focuses on how to design the target loss function to achieve the purpose of knowledge transfer from the teacher network to the student network. When the deep convolutional neural network is deployed in mobile terminal or chip terminal, it is constrained by computing power, and in addition to compressing the weight parameters of the network backbone, the model output often needs to be simplified and the resolution of the model input image needs to be reduced. Therefore, directly using the existing knowledge distillation technology cannot achieve good knowledge transfer. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a lightweight compression method of deep convolutional network, which can obtain a network model with low-resolution input, simplified output and lightweight structure while ensuring the detection accuracy and robustness of the network.

[0005] The technical scheme adopted by the present application to solve the technical problem is: providing a lightweight compression method of deep convolutional network, comprising the following steps:

[0006] (1) simplifying the output of the first network model and pruning the backbone network part of the first network model to obtain a second network model; the first network model is a trained deep convolutional network model;

[0007] (2) fine-tuning the second network model based on knowledge distillation and multi-resolution image;

[0008] (3) model pruning the second network model to generate a third network model, and fine-tuning the third network model based on feature space alignment and knowledge distillation, and taking the trained third network model as the final model.

[0009] The step (1) specifically comprises:

[0010] (11) retaining a backbone network part responsible for image feature extraction and a necessary detection output branch part in the first network model;

[0011] (12) pruning the backbone network part to delete redundant network model parameters, to obtain the second network model.

[0012] The step (2) specifically comprises:

[0013] (21) setting the first network model as a first teacher network and setting the second network model as a first student network;

[0014] (22) training the first student network by using the first teacher network, and mixing using images of a training set with true values and images without true value annotations in each round of training;

[0015] (23) calculating a first target loss function to drive the first student network to perform back propagation and adjust the weight values of each layer of the first student network.

[0016] In the step (22), when the first student network is trained by using the first teacher network, the input of the first teacher network is an original image with a height of H and a width of W; the input of the first student network is the original image and a processed image which are the same as the input of the first teacher network and are distributed according to a preset proportion; the processed image is an image with a height of H and a width of W which is generated by performing N times down-sampling on the image input into the first teacher network and then performing a padding operation.

[0017] In the step (23), the mathematical expression of the first target loss function E is: E = μ1 * E1 + μ2 * E2 hard E1 is a target loss function formed by the output y of the first student network and the true value y corresponding to the training set image; E2 is a target loss function formed by the output y of the first student network and the output of the first teacher network; wherein y1 represents the output of the first teacher network when the input is the training set image with the true value, y2 represents the output of the first teacher network when the input is the image without the true value annotation, s soft s μ1 and μ2 are both weighting coefficients.

[0018] The step (3) specifically comprises:

[0019] (31) performing model pruning on the second network model to generate a third network model;

[0020] ​​​(32) taking the second network model as a second teacher model, and taking the third network model as a second student model;

[0021] (33) training the second student network by using the second teacher network, and mixing the training set images with true values and images without true value labels in each round of training;

[0022] (34) selecting corresponding feature maps in the second teacher network and the second student network, respectively performing up-sampling processing on the feature maps of the second student network by using a pixel reorganization mechanism, and performing down-sampling processing on the feature maps of the second teacher network to obtain two groups of feature map sets with corresponding sizes;

[0023] (35) calculating, for each feature map in the two groups of obtained feature map sets, a feature map containing spatial and channel upper and lower dependency information;

[0024] (36) calculating a second target loss function, driving the second student network to perform back propagation, and adjusting the weight values of each layer of the second student network.

[0025] In the step (33), when the second student network is trained by using the second teacher network, the input of the second teacher network is the original image and the processed image distributed according to a preset proportion; the processed image is an image with a height of H and a width of W generated by performing N times down-sampling on the original image and then performing a padding operation; and the input of the second student network is an image obtained by performing N times down-sampling on the original image.

[0026] The step (34) specifically includes:

[0027] selecting K feature maps T = {T1, T2,..., T i ,...,T K} from a backbone network part of the second teacher network, and selecting K feature maps S = {S1, S2,..., S i ,...,S K} from a backbone network part of the second student network; wherein the feature maps selected from the second teacher network are N times larger than the feature maps selected from the second student network in height and width, and have the same number of channels;

[0028] performing convolution operation on the feature map S i by using a convolution with a size of M1×M1, a number of channels of C1×N 2 , and a step of 1 to generate a feature map S′ i with a size of and a number of channels of C1×N 2 ; and performing pixel rearrangement on the feature map S′ i to generate a feature map S″ with a size of Feature map S″ with C1 channels i Using convolutional subnetworks to process feature map S″ i Perform a convolution operation to generate a product of size 1000. The number of channels is C i Feature map

[0029] Using a size of M2×M2 and a number of channels of C2×(1 / N) 2 A convolution pair with a stride of 1 is used to convolve the feature map T. i Perform a convolution operation to generate a product of size [size missing]. The number of channels is C2×(1 / N) 2 Feature map T′ i ; For feature map T′ i Pixel rearrangement generates a size of Feature map T″ with C2 channels i Using convolutional subnetworks to process feature map T″ i Perform a convolution operation to generate a product of size [size missing]. The number of channels is C i Feature map

[0030] Convolution is used on feature map T respectively i and feature map S i Perform a convolution operation with a stride of 1 to obtain the feature map. and feature map

[0031] Based on feature map Feature map Feature map and feature map Two sets of size-aligned feature maps {T} are obtained ^ ,S *} and {S ^ ,T *},in,

[0032] Step (35) specifically includes:

[0033] feature map Feature map Feature map and feature map Reshape them into dimension C respectively i ×W i Feature map Feature map Feature map and feature map

[0034] feature maps feature maps feature maps and feature maps are respectively reshaped into feature maps with dimensions of W i × C i feature maps feature maps and feature maps

[0035] based on the feature maps feature maps feature maps and feature maps normalized feature maps containing feature space context dependency information are respectively calculated feature maps feature maps and feature maps based on the feature maps feature maps feature maps and feature maps normalized feature maps containing feature channel context dependency information are respectively calculated feature maps feature maps and feature maps wherein H(x) = softmax(x T x).

[0036] The second target loss function E * in the step (36) is E * = E' + E feature , wherein, E h ' ard is a target loss function formed by the output y s ' of the second student network and the true value y corresponding to the training set image; E s ' oft is a target loss function formed by the output y s ' of the second student network and the output of the second teacher network; wherein y1' represents the output of the second teacher network when the input is the training set image with true value, and y'2 represents the output of the second teacher network when the input is the image without true value annotation, μ1 and μ2 are both weighting coefficients; δ1, δ2, δ3, δ4 are all weight coefficients, and L(·) is L1 norm or L2 norm.

[0037] Advantages

[0038] ​Compared with the prior art, the present application has the following advantages and positive effects: the present application uses progressive model pruning and knowledge distillation technology to compress the structure, output and input image resolution of the model, obtain a network model with low resolution input, simplified output and lightweight structure, and ensure the detection accuracy and robustness of the network. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of the lightweight compression method of the deep convolutional network of the embodiment of the present application;

[0040] Figure 2 is a model compression schematic diagram of the deep convolutional network of the embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the claims attached hereto.

[0042] The embodiment of the present application relates to a lightweight compression method of a deep convolutional network, which uses progressive model pruning and knowledge distillation technology to obtain a network model with low resolution input, simplified output and lightweight structure, and ensures the detection accuracy and robustness of the network. The overall process is shown in Figure 1 , the model compression process is shown in Figure 2 , and specifically includes the following main steps:

[0043] Step 1, based on the trained deep convolutional network model (referred to as the first network model), the output is simplified and the main network part is pruned to obtain a new lighter network model (referred to as the second network model). The specific sub-steps are as follows:

[0044] 1) According to the actual requirements and computing power of the mobile end or chip end application, the main network responsible for image feature extraction and the necessary detection output branch are retained, and the unnecessary network detection output branch is removed. For example, in a gesture detection application, the model first extracts the feature of the input image information through the main network, and then detects the two-dimensional position, category, key point information and pose information in the three-dimensional world of the gesture target in the image through different detection branches. When the model is actually deployed on the mobile end or chip end, the output branch of the key point information and the pose information can be removed according to the application requirements and the computing power, and although the more detailed information is lost, the basic gesture recognition function can still be ensured in the actual application.

[0045] 2) subsequently pruning the backbone network part of the first network model, deleting redundant network model parameters, obtaining a second network model.

[0046] Step 2, after model pruning, the accuracy will generally decrease, and the input of the first network model is a large resolution image, so the second network model obtained after pruning the first network model has the problem of poor compatibility for low resolution images. In order to improve the accuracy of the second network model and enhance the compatibility of the network for low resolution images, the second network model obtained in step 1 is fine-tuned based on knowledge distillation and multi-resolution image. The specific sub-steps are as follows:

[0047] 1) Set the first network model as the teacher network and the second network model as the student network, and the input image of the two networks is H high and W wide.

[0048] 2) In order to increase the generalization and robustness of the second network model, more image data other than the training data set is added in knowledge distillation, that is, during each round of training, the training set images with true values and images without true value labels are mixedly used.

[0049] ① The input of the first network model is the original image with a height of H and a width of W.

[0050] ② The input of the second network model is selected between the same image as the input of the first network model and the processed image at a certain ratio. Among them, the processed image is generated by N times down-sampling the image input into the first network model, and then padding operation is performed to generate an image with a height of H and a width of W.

[0051] 3) Calculate the target loss function E to drive the second network model to perform back propagation and adjust the weights of each layer of the network. Among them, the target loss function E is respectively weighted according to the different input images of the network, and the specific mathematical expression of the target loss function E is:

[0052]

[0053] Among them, E hard is the target loss function composed of the output y s of the student network and the true value y corresponding to the data set picture; E soft is the target loss function composed of the output of the student network and the output of the teacher network, wherein y1 is the output of the teacher network when the input image is the image in the training data set, and y2 is the output of the teacher network when the input image is the image without true value label; μ1 and μ2 are both weighting coefficients, preferably μ1 is greater than μ2.

[0054] Step 3, model pruning is performed on the second network model to generate a third network model, and the third network model is fine-tuned based on feature space alignment and knowledge distillation. The specific steps are as follows:

[0055] 1) Model pruning is performed on the second network model to generate a third network model.

[0056] 2) Set the second network model as the teacher network and the third network model as the student network. The input of the second network model is an image with height H and width W, and the input of the third network model is an image with height H / N and width W / N.

[0057] 3) To increase the generalization and robustness of the third network model, more image data outside the training data set is added in the knowledge distillation, i.e. in each round of training, the training set images with true value and images without true value annotation are mixed and used for knowledge distillation training of the third network model. Preferably, the specific operation is as follows:

[0058] ① The input of the second network model is selected between the original image and the processed image. The processed image is an image with height H and width W generated by N times down-sampling of the original image and then padding operation.

[0059] ② The input of the third network model is an image with N times down-sampling of the original image, i.e. an image with resolution (1 / N)H x (1 / N)W.

[0060] 4) Select corresponding feature maps in the second network model and the third network model, respectively, and use pixel recombination mechanism to up-sample the third network model feature map and down-sample the second network model feature map to obtain two sets of size-aligned feature maps. The specific operation steps are as follows:

[0061] ① Select K feature maps T = {T1, T2,..., T i ,...,T K} in the backbone network part of the second network model and the corresponding K feature maps S = {S1, S2,..., S i ,...,S K} in the backbone network part of the third network model. The feature maps selected by the second network model are N times larger in height and width than the corresponding feature maps selected by the third network model, and the number of channels is consistent.

[0062] ② Use pixel recombination mechanism to up-sample each feature map S i selected from the third network model to expand its height and width by N times: first, use a convolution with size M1 x M1, channel number C1 x N 2 , and step size 1 to expand the feature map Si Perform a convolution operation to generate a product of size [size missing]. The number of channels is C1×N 2 Feature map S′ i Subsequently, the feature map S′ i Pixel rearrangement generates a size of Feature map S″ with C1 channels i Finally, a convolutional sub-network is used to process the feature map S″. i Perform a convolution operation to generate a final size of The number of channels is C i Feature map

[0063] ③ Utilize a pixel recombination mechanism to process the feature maps T selected from the second network model. i Downsampling is performed, reducing both its height and width by a factor of N: First, using a size of M2×M2 and a channel number of C2×(1 / N) 2 A convolution pair with a stride of 1 is used to construct the feature map T. i Perform a convolution operation to generate a size of The number of channels is C2×(1 / N) 2 Feature map T′ i Subsequently, the feature map T′ i Pixel rearrangement generates a size of Feature map T″ with C2 channels i Finally, stacked convolutional subnetworks are used to process the feature map T″. i Perform a convolution operation to generate a final size of The number of channels is C i Feature map

[0064] ④ Apply convolution to feature map T respectively i and feature map S i Perform a convolution operation with a stride of 1 to obtain the feature map. and feature map

[0065] ⑤ Based on the above steps, according to the feature map Feature map Feature map and feature map Two sets of size-aligned feature maps {T^,S} are obtained. *} and {S^,T *},in,

[0066] 5) Based on each feature map in the two feature map sets obtained in 3), respectively calculate feature maps containing spatial and channel context-dependent information. The specific operation is as follows:

[0067] ①Reshape feature maps , and , and , and to feature maps with dimensions of C i ×W i , respectively. , and , and , and wherein,

[0068] ②Reshape feature maps , and , and , and to feature maps with dimensions of W i ×C i , respectively. , and , and , and

[0069] ③Based on feature maps , and , and , and , respectively calculate normalized feature maps , and , and , and containing feature spatial context-dependent information. , and , and , and containing feature channel context-dependent information. , and , and , and The specific formula of H(x) is as follows:

[0070] H(x) = softmax(x T x)

[0071] 6) Calculate the target loss function E * , drive the third network model to perform back propagation, and adjust the weights of each layer of the network. Compared with the loss function in step 2, the target loss function of this part further increases the constraint E feature of the teacher network on the student network in the feature space, i.e. E* = E' + E feature .

[0072] wherein,

[0073] E' hard is the output y' of the second student network s target loss function composed of the true value y corresponding to the training set image; E' soft is the output y' of the second student network s target loss function composed of the output of the second teacher network; wherein y'1 represents the output of the second teacher network when the input is the training set image with the true value, and y'2 represents the output of the second teacher network when the input is the image without true value annotation, μ1 and μ2 are both weighting coefficients.

[0074] E feature The specific mathematical expression formula of E is as follows:

[0075]

[0076] wherein, δ1, δ2, δ3, δ4 are all weight coefficients, and L(·) is L1 norm or L2 norm.

[0077] It is not difficult to find that the present application can simultaneously compress the structure, output and input image resolution of the model by using the progressive model pruning and knowledge distillation technology, can obtain a network model with low resolution input, simplified output and light structure, and can guarantee the detection accuracy and robustness of the network.

Claims

1. A lightweight compression method for deep convolutional networks, characterized in that, Includes the following steps: (1) The output of the first network model is simplified and the backbone network of the first network model is pruned to obtain the second network model; the first network model is a trained deep convolutional network model. (2) Fine-tuning the second network model based on knowledge distillation and multi-resolution images, specifically including: (21) Set the first network model as the first teacher network and the second network model as the first student network; (22) The first teacher network is used to train the first student network, and in each training round, training set images with ground truth and images without ground truth labels are mixed; wherein, when the first teacher network is used to train the first student network, the input of the first teacher network is an original image with height H and width W; the input of the first student network is an original image and a processed image that are distributed according to a preset ratio, which are the same as the input of the first teacher network; the processed image is an image with height H and width W generated by downsampling the image input to the first teacher network by N times and then filling the image. (23) Calculate the first objective loss function, drive the first student network to backpropagate, and adjust the weights of each layer of the first student network; (3) Pruning the second network model to generate a third network model, fine-tuning the third network model based on feature space alignment and knowledge distillation, and using the trained third network model as the final model, specifically including: (31) Prune the second network model to generate a third network model; (32) The second network model is used as the second teacher network, and the third network model is used as the second student network; (33) The second teacher network is used to train the second student network, and in each round of training, training set images with ground truth and images without ground truth labels are mixed together; (34) Select corresponding feature maps in the second teacher network and the second student network, and use the pixel recombination mechanism to upsample the feature map of the second student network and downsample the feature map of the second teacher network to obtain two sets of feature map sets with corresponding sizes. (35) For each feature map in the two sets of feature maps obtained, calculate the feature map containing spatial and channel up-down dependency information respectively; (36) Calculate the second objective loss function, drive the second student network to backpropagate, and adjust the weights of each layer of the second student network.

2. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, Step (1) specifically includes: (11) Retain the backbone network part responsible for image feature extraction and the necessary detection output branch part in the first network model; (12) Prune the backbone network and remove redundant network model parameters to obtain the second network model.

3. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, The mathematical expression for the first objective loss function E in step (23) is: Among them, E hard For the output y of the first student network s The target loss function is composed of the ground truth y corresponding to the training set images; E soft For the output y of the first student network s The target loss function is formed by the input of the first teacher network and the output of the first teacher network; where y1 represents the output of the first teacher network when the input is a training set image with ground truth labels, and y2 represents the output of the first teacher network when the input is an image without ground truth labels. μ1 and μ2 are both weighting coefficients.

4. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, In step (33), when the second teacher network is used to train the second student network, the input of the second teacher network is the original image and the processed image distributed according to a preset ratio; the processed image is an image with a height of H and a width of W generated by downsampling the original image by N times and then filling it; the input of the second student network is an image of the original image downsampled by N times.

5. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, Step (34) specifically includes: K feature maps T = {T1, T2, ..., T} are selected from the backbone network of the second teacher network. i ,...,T K K feature maps S = {S1, S2, ..., S} are selected from the backbone network portion of the second student network. i ,…,S K }; where the feature map selected from the second teacher network is N times larger in both height and width than the feature map selected from the second student network, but the number of channels is the same; Using a size of M1×M1 and a number of channels of C1×N 2 A convolution with stride of 1 is applied to the feature map S. i Perform a convolution operation to generate a product of size [size missing]. The number of channels is C1×N 2 Feature map S i ′;For feature map S i Perform pixel rearrangement to generate a size of Feature map S with C1 channels i "; Using convolutional subnetworks to process feature map S i "Perform convolution operation," The generated size is The number of channels is C i Feature map Using a size of M2×M2 and a number of channels of C2×(1 / N) 2 A convolution pair with a stride of 1 is used to convolve the feature map T. i Perform a convolution operation to generate a product of size [size missing]. The number of channels is C2×(1 / N) 2 Feature map T i ′; for feature map T i Perform pixel rearrangement to generate a size of Feature map T with C2 channels i "; Using convolutional subnetworks to process feature map T" i Perform a convolution operation to generate a product of size [size missing]. The number of channels is C i Feature map T i * ; Convolution is used on feature map T respectively i and feature map S i Perform a convolution operation with a stride of 1 to obtain the feature map T. i ^ and feature map Based on feature map T i ^, Feature Map Feature map T i * and feature map Two sets of size-aligned feature maps {T^,S} are obtained. * } and {S^,T * },in, 6. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, Step (35) specifically includes: Feature map T i ^, Feature Map Feature map and feature map T i * Reshape them into dimension C respectively i ×W i Feature map Feature map Feature map and feature map Feature map T i ^, Feature Map Feature map and feature map T i *Reshape each to dimension W i ×C i Feature map Feature map Feature map and feature map Based on feature maps Feature map Feature map and feature map Calculate the normalized feature maps that contain contextual dependency information of the feature space respectively. Feature map Feature map and feature map Based on feature maps Feature map Feature map and feature map Calculate the normalized feature map containing feature channel context dependency information respectively. Feature map Feature map and feature map Where H(x) = softmax(x) T x).

7. The lightweight compression method for deep convolutional networks according to claim 1, characterized in that, The second objective loss function E in step (36) * For E * =E′+E feature ,in, E′ hard The output y′ of the second student network s The target loss function is formed by the ground truth y corresponding to the training set images; E′ soft The output y′ of the second student network s The target loss function is formed by the input of the second teacher network and the output of the second teacher network; where y′1 represents the output of the second teacher network when the input is a training set image with ground truth labels, and y′2 represents the output of the second teacher network when the input is an image without ground truth labels. μ1 and μ2 are both weighting coefficients; δ1, δ2, δ3, and δ4 are all weighting coefficients, and L(·) is the L1 norm or L2 norm.

Images