A deep neural network model compression method and system based on combined pruning

Abstract

The application discloses a kind of deep neural network model compression method and system based on combination type pruning, based on the deep neural network model of pre-trained, for each layer network in network model, obtain the numerical matrix of all convolution kernel combinations in this layer;With the convolution kernel combination as the smallest unit, the distribution diversity of combination is calculated, and the numerical vector group representing the corresponding convolution layer is obtained;According to the distribution diversity result of combination result, select part of combination to activate, and through nonlinear mapping to convolution kernel, obtain combination type parameter model;Based on combination type parameter model, select single use or jointly use with other pruning methods based on single convolution kernel, structured model pruning and retraining are carried out.The method is based on the structured model pruning method of multiple convolution kernel combinations, overcomes the information loss inside actual convolution kernel and between convolution kernel, and significantly expands our search space without paying too much computing cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning and model compression, and particularly relates to a deep neural network model compression method and system based on combined pruning. BACKGROUND

[0002] Deep convolutional neural networks have achieved extremely superior performance in the academic field, but one of the difficulties in applying deep convolutional neural networks to the industrial field is that it is difficult to deploy deep models on resource-constrained hardware, such as mobile phones and wearable devices, and the like. Model compression methods are proposed to solve this problem, and network pruning is the most widely used method in model compression. Deep convolutional neural network pruning can effectively simplify the model and reduce the limitations from model size and operation cost under the premise of ensuring performance, and has strong practical application significance.

[0003] Deep convolutional neural network pruning can be divided into unstructured pruning and structured pruning. Among them, structured pruning obtains a compact network with lower physical and computational cost by pruning redundant convolution kernels, and does not require a specific format to store the network and a specific sparse matrix multiplication, which can accelerate and improve the actual inference time in hardware without specific library or hardware support, and shows broad prospects in various emerging applications. Since the pruned network can be well applied to mainstream hardware to obtain considerable speedup, structured pruning has been widely used in industrial products.

[0004] However, existing methods always prune individual features of convolution kernels independently, without considering the role of convolution kernels in multiple convolution kernel combinations. These single convolution kernel-based pruning algorithms will lose valuable information carried by convolution kernels inside and between convolution kernels. The core of structured pruning is the selection of convolution kernels, which should produce the highest compression ratio and the lowest accuracy loss. The waste of information of convolution kernels will cause a suboptimal solution of pruning.

[0005] Therefore, how to better utilize the effective information inside and between convolution kernels for the selection of convolution kernels to solve the efficiency problem of convolutional neural network pruning can enable convolutional neural networks to be more widely applied to embedded devices, which has important practical significance and great challenge. SUMMARY

[0006] In order to solve the problems in the prior art, the application provides a deep neural network model compression method based on combined pruning, which better utilizes the effective information inside and between convolution kernels for the selection of convolution kernels, solves the efficiency problem of convolutional neural network pruning, and helps to enable convolutional neural networks to be more widely applied to embedded devices.

[0007] In order to achieve the above object, the technical scheme adopted by the present application is: a deep neural network model compression method based on combined pruning, comprising the following steps:

[0008] S1, based on a pre-trained deep neural network model, for each layer of the network model, obtaining all possible combinations of convolution kernels, the number of convolution kernels included in all combinations in a single calculation is the same and not less than two, obtaining the numerical matrix of all convolution kernel combinations in the layer;

[0009] S2, taking the convolution kernel combination as the smallest unit, performing distribution diversity calculation of the combination, and obtaining a numerical vector group representing the corresponding convolution layer;

[0010] S3, based on the distribution diversity result of the combination result, selecting part of the combination for activation, and mapping to the convolution kernel through nonlinear mapping to obtain a combined parameter model;

[0011] S4, based on the combined parameter model, selecting single use or combined use with other single convolution kernel based pruning methods, and performing structured deep neural network model pruning and retraining based on the convolution kernel.

[0012] The pre-trained deep neural network model in S1 is an uncompressed network model or a compressed model after pruning.

[0013] The number of convolution kernels in each combination in S1 is the same in a single calculation, and is not less than two.

[0014] The distribution diversity of the combination in S2 is calculated by calculating the information richness of the combination distribution to measure the complexity and diversity of the combination, and a group of numerical vectors is generated for each layer of the network, the number of elements of the numerical vector is consistent with the number of combinations of the layer.

[0015] The specific calculation method of the distribution diversity in S2 is: the parameter value distribution of the combination is counted, the continuous distribution of the combination is uniformly divided into K value segments, and the distribution diversity of the combination is calculated using the information entropy formula according to the proportion of the number of parameters in each value segment to the total parameters.

[0016] In S3, the numerical vector groups corresponding to all convolution layers are combined to obtain a distribution diversity numerical matrix of the entire network, the activation operation is performed on the convolution kernel combinations of a set proportion, the threshold value corresponding to the distribution diversity of the combination result is found according to the activation proportion, the distribution diversity result of the combination greater than the threshold value is retained, the part of the numerical value without activation is set to zero, and the activated combination distribution diversity result is obtained;

[0017] According to the histogram statistical method based on the activated result, the combination result is mapped to the convolution kernel to realize smooth structured pruning and obtain a combined parameter model.

[0018] S4 is specifically implemented as follows:

[0019] According to the combined parameter model, structured deep neural network pruning is performed to obtain a deep neural network model with smaller parameter quantity and operation quantity;

[0020] The deep neural network model after pruning is retrained to restore the model accuracy, and an applicable deep neural network model is obtained.

[0021] On the other hand, the application provides a deep neural network model compression system based on combined pruning, which comprises a model acquisition module, a numerical vector group calculation module, an activation and mapping module, and a training module,

[0022] The model acquisition module, based on a pre-trained deep neural network model, acquires all possible convolution kernel combinations for each layer of the network model, and the number of convolution kernels included in all combinations in a single calculation is the same and not less than two, and acquires the numerical matrix of all convolution kernel combinations in the layer

[0023] The numerical vector group calculation module calculates the distribution diversity of the combinations with the convolution kernel combinations as the minimum unit, and acquires the numerical vector group representing the corresponding convolution layer;

[0024] The activation and mapping module selects part of the combinations for activation based on the distribution diversity of the combination results, and maps the combinations to the convolution kernel through nonlinear mapping to obtain a combined parameter model;

[0025] The training module, based on the combined parameter model, selects a single use or a combination with other pruning methods based on a single convolution kernel, and performs structured deep neural network model pruning and retraining in units of convolution kernels.

[0026] The application also provides a computer device comprising a processor and a memory, the memory being used to store a computer executable program, the processor reading part or all of the computer executable program from the memory and executing, and the processor executing part or all of the computer executable program to realize the deep neural network model compression method based on combined pruning.

[0027] Meanwhile, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the deep neural network model compression method based on combined pruning.

[0028] Compared with the prior art, the application has at least the following beneficial effects:

[0029] The structured model pruning method based on multiple convolution kernel combinations provided by the application overcomes information loss in actual convolution kernels and between convolution kernels, and significantly expands the search space without paying too much computational cost.

[0030] The application constructs a combined parameter model based on the information richness of the combination distribution, and can evaluate convolution kernels in a combined manner and mine implicit information.

[0031] The application is a non-exclusive general model pruning method, which can improve any other pruning method based on a single convolution kernel, improve model compression efficiency, and obtain better performance.

[0032] The application can improve model compression efficiency, better apply deep neural networks to mainstream embedded devices, and bring economic benefits in software development and mobile and wearable device fields. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows.

[0034] Figure 1 is a flowchart of a deep neural network model compression method based on combined pruning provided by an embodiment of the application;

[0035] Figure 2 is a flowchart of a combined parameter model provided by an embodiment of the application;

[0036] Figure 3 is a comparison diagram before and after pruning provided by an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to make the innovative technical solutions and advantages of the present application relatively clearer, the technical solutions involved in the present application will be described more clearly and completely in combination with the specific embodiments of the application and the corresponding drawings. Meanwhile, the embodiments described in the present text are only some of the embodiments of the present application, not all the embodiments.

[0038] Reference Figure 1 and Figure 2 , Figure 2 is a detailed flowchart of a combined parameter model provided by an embodiment of the application, specifically including the following steps:

[0039] Step 1: Train an image classification model. In the present application example, the image classification model is a deep neural network model to be compressed, and the pruning effect is illustrated by comparing the test accuracy, model parameter pruning rate and model floating point operation pruning rate after pruning.

[0040] Step 2: Extract the combined numerical matrix according to the trained classification model. For the lth convolutional layer in the trained classification model, obtain the weight value matrix W l , which consists of c convolutional kernels, that is, According to all possible convolutional kernel combinations, construct all combined numerical matrices in this layer, and convert the data to Numpy format as the input for the next step. In a single calculation, the number of convolutional kernels in each combination is the same and not less than two.

[0041] Step 3: After constructing all combined numerical matrices in each layer in step 2, use the server to calculate the corresponding distribution diversity. The specific calculation method is: in the lth convolutional layer in the model, a convolutional kernel combination consisting of p, q, and multiple convolutional kernels not explicitly expressed is denoted as The parameter value distribution of combination A p…q is uniformly divided into K numerical segments, and K is an integer not less than 10, which is a hyperparameter related to the network size. According to the proportion of the number of parameters in each numerical segment to the total parameters, the distribution is denoted as {P(0; p…q), …, P(k; p…q), …, P(K; p…q)}. The distribution diversity of the combination is calculated using the information entropy formula, as shown in formula (1).

[0042]

[0043] The distribution diversity of the combination is calculated by calculating the information richness of the combination distribution to measure the complexity and diversity of the combination. Each layer of the network generates a set of numerical vectors, and the number of elements in the numerical vector is consistent with the number of combinations in the layer network.

[0044] Step 4: Perform concatenation processing on the results of step 3, that is, combine the numerical vector corresponding to all convolutional layers to obtain the distribution diversity numerical matrix of the entire network. Perform activation operation on a set proportion of convolutional kernel combinations, find the threshold value thre corresponding to the distribution diversity of the combination result according to the activation proportion, retain the distribution diversity results of the combinations greater than the threshold value, and set the values of the unactivated parts to zero to obtain the activated combination distribution diversity result. The activation function is shown in formula (2).

[0045]

[0046] Step 5: The results of step 4 are mapped into the convolution kernel through a nonlinear mapping to obtain a combined parameter model. According to the results after activation, a histogram statistical method is used to compare the occurrence frequency of the efficient convolution kernel, and the convolution kernel corresponding to the occurrence frequency exceeding the average value is determined. The combined results of this part are mapped to the convolution kernel, so as to ensure that the search space is within a reasonable range, reduce the amount of calculation in the subsequent calculation process, and remove the redundant information in the combination, obtain the combined parameter model, and this is a nonlinear operation, which helps to realize smooth structured pruning and achieve better results, refer to Figure 3 .

[0047] Optionally, the present application also provides a device for deep neural network model compression based on combined pruning, which comprises a processor and a memory. The memory is used to store computer executable programs. The processor reads part or all of the computer executable programs from the memory and executes them. When the processor executes part or all of the computer executable programs, the steps of the method for deep neural network model compression based on combined pruning according to the present application can be realized.

[0048] A computer readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of the method for deep neural network model compression based on combined pruning according to the present application can be realized.

[0049] A computer program can be written in a computer programming language to execute the method according to the present application. The computer program can be in the form of source code, object code, executable file or some intermediate form. The computer programming language can be C++, Java, Fortran, C# or Python.

[0050] The device for deep neural network model compression based on combined pruning can be a notebook computer, a tablet computer, a desktop computer, a mobile phone or a workstation.

[0051] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a ready programmable gate array (FPGA).

[0052] The memory according to the present application can be an internal storage unit of a notebook computer, a tablet computer, a desktop computer, a mobile phone or a workstation, such as a memory or a hard disk. An external storage unit such as a mobile hard disk or a flash card can also be used.

[0053] The computer-readable storage medium can include computer storage media and communication media. The computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The computer-readable storage medium can include read-only memory (ROM), random access memory (RAM), solid state disk (SSD), optical disk, etc. Among them, the random access memory can include resistance random access memory (ReRAM).

Claims

1. A method for compressing a deep neural network model based on combined pruning, characterized in that, The method comprises the following steps: S1, based on a pre-trained deep neural network model, for each layer of the network model, obtain all possible combinations of convolution kernels, the number of convolution kernels included in all combinations in a single calculation is the same and not less than two, and obtain the numerical matrix of all convolution kernel combinations in the layer; the deep neural network model is an image classification model; S2, taking the convolution kernel combination as the smallest unit, the distribution diversity of the combination is calculated to obtain a numerical vector group representing the corresponding convolution layer; S3, based on the distribution diversity of the combination result, select part of the combination for activation, and map it to the convolution kernel through nonlinear mapping to obtain a combined parameter model; S4, based on the combined parameter model, select single use or joint use with other pruning methods based on single convolution kernel, and perform structured deep neural network model pruning and retraining based on convolution kernel. 2.The method of claim 1, wherein, The pre-trained deep neural network model in S1 is an uncompressed network model or a compressed model after pruning. 3.The method of claim 1, wherein, In S1, the number of convolution kernels in each combination in a single calculation is the same and not less than two.

4. The combined pruned deep neural network model compression method of claim 1, wherein, In S2, the distribution diversity of the combination is calculated by calculating the information richness of the combination distribution to measure the complexity and diversity of the combination. Each layer of the network generates a numerical vector group, and the number of elements of the numerical vector is consistent with the number of combinations of the layer.

5. The combined pruning-based deep neural network model compression method of claim 1, wherein, In S2, the specific calculation method of the distribution diversity is: the parameter value distribution of the combination is counted, the continuous distribution of the combination is uniformly divided into K numerical segments, and the proportion of the number of parameters in each numerical segment to the total parameters is used to calculate the distribution diversity of the combination using the information entropy formula.

6. The combined pruned deep neural network model compression method of claim 1, wherein, In S3, the numerical vector groups corresponding to all convolution layers are combined to obtain a distribution diversity numerical matrix of the entire network. The distribution diversity of the combination result is found according to the activation proportion, the distribution diversity results of the combinations greater than the threshold value are retained, the parts of the numerical value without activation are set to zero, and the activated combination distribution diversity result is obtained; According to the histogram statistical method based on the result after activation, the combination result is mapped to the convolution kernel to realize smooth structured pruning and obtain a combined parameter model.

7. The combined pruned deep neural network model compression method of claim 1, wherein, The specific implementation method of S4 is as follows: According to the combined parameter model, perform structured deep neural network pruning to obtain a deep neural network model with smaller parameter quantity and operation quantity; Re-train the pruned deep neural network model to restore the model accuracy and obtain an applicable deep neural network model.

8. A deep neural network model compression system based on combined pruning, characterized in that, The method comprises a model acquisition module, a numerical vector group calculation module, an activation and mapping module, and a training module, The model acquisition module, based on a pre-trained deep neural network model, for each layer of the network model, obtains all possible combinations of convolution kernels, the number of convolution kernels included in all combinations in a single calculation is the same and not less than two, and obtains the numerical matrix of all convolution kernel combinations in the layer The numerical vector group calculation module calculates the distribution diversity of the combination to obtain a numerical vector group representing the corresponding convolution layer. The activation and mapping module selects some combinations for activation based on the distribution diversity of the combination results, and obtains a combined parameter model by nonlinear mapping to the convolution kernel. The training module selects a single convolution kernel or combines it with other pruning methods based on a single convolution kernel, and performs structured pruning and retraining of the deep neural network model based on the combined parameter model.

9. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the deep neural network model compression method based on the combined pruning.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the deep neural network model compression method based on the combined pruning.

Images