Method and apparatus for pruning a neural network

Abstract

A block-wise importance estimation method is used to accurately estimate the contribution of a block to the target. The layers of the neural network are first grouped into virtual blocks, the block size is determined by the computational requirements. The last layer of each block is connected to a second group of layers including a scaling layer. The scaling layer is a trainable scalar that is multiplied with the previous feature map. The scaled outputs of all blocks are summed and forwarded or predicted. During training, blocks that contribute higher accuracy with respect to the ground truth are assigned higher scaling factors. A layer-wise decoupling method is used to reduce the dependencies between neurons in each layer. Based on the determined importance measure of the virtual blocks and the diversity of neurons in the layer, neurons are pruned from the layer.

Description

TECHNICAL FIELD

[0001] Aspects of the present disclosure generally relate to neural networks, and more particularly, to pruning convolutional neural networks. BACKGROUND

[0002] Deep neural networks (DNNs) are powerful models that have been successfully applied to various artificial intelligence (AI) tasks such as image recognition, speech recognition, and machine translation. However, deep neural networks are very demanding in terms of computational resources, requiring a large number of floating point operations per second (FLOP). Model compression and acceleration are crucial in various DNN-based AI applications.

[0003] Pruning is a method to reduce the size of a neural network by compression. After pre-training a network, the network is fine-tuned to determine the importance of connections. Previous structural pruning methods usually compute a global scaling factor for each neuron as a measure of importance and prune neurons with lower importance to maintain accuracy. However, there are two limitations that can lead to improper pruning and compromise accuracy. Given the large number of layers in modern deep neural networks, it is difficult to find a globally fair scaling factor. In addition, the scaling factor cannot reflect the correlation between filters, so pruning filters with lower importance can still affect related filters and compromise accuracy.

[0004] Therefore, it is desirable to be able to provide a pruning architecture and method that can address at least some of the above problems. SUMMARY

[0005] Aspects of the disclosed embodiments relate to a method for structured pruning of deep neural networks. This object is achieved by the subject matter of the independent claims. Further advantageous modifications are provided in the dependent claims.

[0006] According to the first aspect, the above and other objects and advantages are achieved by means of an apparatus. In one embodiment, the apparatus includes a processor configured to: group layers of a convolutional neural network into one or more virtual blocks, wherein each of the one or more virtual blocks includes at least one layer; determine an importance metric for the virtual blocks in the one or more virtual blocks; apply diversity regularization to neurons in the layers of the virtual blocks to reduce dependencies between neurons in the layers; and prune neurons from the layers based on the determined importance metric of the virtual blocks and the diversity of neurons in the layers. The disclosed aspects provide a block-by-block importance metric and a neuron decoupling process based on the block's contribution to target accuracy to reduce dependencies between neurons from the same layer, regardless of their importance.

[0007] According to the first aspect, in a first possible implementation of the device, the last convolutional layer of each virtual block is connected to a corresponding layer in a second layer group, wherein the second layer group includes an average pooling layer, a fully connected layer, and a scaling layer. The scaling factor is a trainable scalar multiplied by the previous feature map and supports determining a block-by-block importance metric based on the block's contribution to the target accuracy.

[0008] In one possible implementation of the device, the processor is further configured to determine the importance metric of the one or more virtual blocks by: summing the outputs of the scaling layer associated with each of the one or more virtual blocks; forwarding the summed outputs to the prediction layer for accuracy prediction during training; assigning higher scaling factors to virtual blocks that contribute higher accuracy during training, and assigning lower scaling factors to virtual blocks that contribute lower accuracy during training. The importance of neurons is measured at the block level, rather than requiring a global importance metric, which enables unbiased estimation.

[0009] In one possible implementation of the device, the diversity regularization is defined as:

[0010]

[0011] Where L is the number of layers, W is the weight matrix for each layer L, and I is the recognition matrix, ||·|| F It is the Frobenius norm. During training, the orthogonality of the column and row vectors in matrix W is enforced, which is intended to cover both overcomplete and undercomplete cases of W. Dependencies between neurons in the same layer are reduced, and the importance of neurons is measured at the block level without having to find a global importance metric, thus achieving unbiased estimation.

[0012] In a possible implementation form of the apparatus, the processor is configured to prune neurons from the layer based on the determined importance measure of the virtual block and the diversity of neurons in the layer by: selecting the virtual block from the one or more virtual blocks; selecting the layer from the selected virtual block; calculating a correlation value D between weight vectors of two neurons of the selected layer; sorting the correlation values D in ascending order; pruning neurons from the selected layer based on the correlation values D. Pruning neurons is performed randomly based on block-wise importance and correlation between neurons. The dependency between neurons in the same layer is reduced, and the importance of neurons is measured at block level without having to find a global importance measure, thereby enabling unbiased estimation.

[0013] In a possible implementation form of the apparatus, the virtual block is randomly selected from the one or more virtual blocks. Pruning neurons is performed randomly based on block-wise importance and correlation between neurons. By pruning neurons in a random manner rather than a deterministic manner based on the diversity of neurons, false pruning of neurons is reduced.

[0014] According to a second aspect, the above mentioned and other objectives and advantages are achieved by a method. In one embodiment, the method comprises: grouping layers of a convolutional neural network into one or more virtual blocks, wherein each virtual block of the one or more virtual blocks comprises at least one layer; determining an importance measure of a virtual block of the one or more virtual blocks; applying diversity regularization to neurons in a layer of the virtual block to reduce dependency between neurons in the layer; pruning neurons from the layer based on the determined importance measure of the virtual block and the diversity of neurons in the layer.

[0015] In a possible implementation form of the method, determining the importance measure of the one or more virtual blocks comprises: summing outputs of scaled layers associated with each virtual block of the one or more virtual blocks; forwarding the summed outputs to a prediction layer for accuracy prediction during training; assigning higher scaling factors to virtual blocks that contribute higher accuracy during training and lower scaling factors to virtual blocks that contribute lower accuracy during training. This enables determining a block-wise importance measure based on the contribution of a block to a target accuracy.

[0016] In a possible implementation form of the method, the diversity regularization is defined as:

[0017]

[0018] where L is the number of layers, W is the weight matrix of each layer L, I is the identity matrix, and ||·|| is the L2 norm. Fis the Frobenius norm. The orthogonality of column vectors and row vectors in the matrix W is enforced during training, which aims to cover the over-complete and under-complete W cases. The orthogonality of column vectors and row vectors in the matrix W is enforced during training, which aims to cover the over-complete and under-complete W cases. The dependency between neurons in the same layer is reduced, and the importance of neurons is measured at the block level, without having to find a global importance measure, thus achieving an unbiased estimation.

[0019] In a possible implementation form of the method, pruning neurons from the layer based on the determined importance measure of the virtual block and the diversity of neurons in the layer further comprises: selecting the virtual block from the one or more virtual blocks; selecting the layer from the selected virtual block; calculating a correlation value D between weight vectors of two neurons of the selected layer; sorting the correlation values D in ascending order; pruning neurons from the selected layer based on the correlation values D. The dependency between neurons in the same layer is reduced, and the importance of neurons is measured at the block level, without having to find a global importance measure, thus achieving an unbiased estimation.

[0020] In a possible implementation form of the method, the method comprises randomly selecting the virtual block from the one or more virtual blocks. Pruning neurons is performed randomly based on the block-wise importance and the correlation between neurons. By pruning neurons based on the diversity of neurons in a random manner rather than a deterministic manner, the false pruning of neurons is reduced.

[0021] These and other aspects, implementation forms and advantages of the exemplary embodiments will become apparent from the embodiments described herein considered in conjunction with the accompanying drawings. It is to be understood, however, that the description and drawings are for illustrative purposes only and should not be construed as limiting the application; any limitation with respect to the application should only be determined with reference to the appended claims. Additional aspects and advantages of the application will be set forth in the description to follow, and in part will become apparent to those having ordinary skill in the art upon examination of the following or can be learned from the practice of the application. The aspects and advantages of the application will be realized and attained by means of the elements and combinations BRIEF DESCRIPTION OF DRAWINGS

[0022] In the following detailed portion of the detailed description of the application, the application will be explained in detail with reference to the exemplary embodiments shown in the drawings,

[0023] wherein:

[0024] Figure 1 a schematic block diagram of an exemplary apparatus provided by aspects of the disclosed embodiments is shown;

[0025] Figure 2Exemplary block-wise estimation operations provided by aspects of the disclosed embodiments are shown.

[0026] Figure 3 A flowchart of an exemplary process provided by aspects of the disclosed embodiments is shown. DETAILED DESCRIPTION

[0027] Figure 1 A schematic block diagram of an exemplary apparatus 100 provided by aspects of the disclosed embodiments is shown. The apparatus 100 is used to provide a structural pruning method for deep neural networks. Aspects of the disclosed embodiments involve block-wise importance estimation in order to accurately estimate the contribution of a block to a target. A neuron decoupling scheme reduces the dependency between neurons regardless of their importance in maintaining accuracy after pruning.

[0028] As Figure 1 shown, the apparatus 100 includes a processor 102 and a memory 104 for storing one or more programs that are executable by the processor 102 to perform the methods described herein. The apparatus 100 also includes an input device 106 for receiving input data and an output device 108 for outputting data. In one embodiment, the apparatus 100 is a computing apparatus that includes one or more processors.

[0029] In one embodiment, and with reference also to Figure 2 , the processor 102 is configured to group layers 200 of a convolution neural network (CNN) into blocks, such as virtual blocks 202-208. The layers 200 of the convolution neural network can be divided into any suitable number of virtual blocks.

[0030] For the purposes of the description herein, reference is made to the virtual block 202. Similar descriptions can apply to the other virtual blocks 204-208. A virtual block, such as the virtual block 202, will include one or more layers, also referred to herein as convolution layers. For the purposes of the description herein, in Figure 2 the example, the first virtual block (#1) is referred to as the virtual block 202, and the last virtual block (#N) is referred to as the virtual block 208. N represents any suitable number, which depends in part on the size of the neural network and the size of the virtual blocks.

[0031] The size of a virtual block (referring to the number of layers in a virtual block) will be greater than or equal to 1. In Figure 2 the example, the virtual block 202 includes three layers. In alternative embodiments, the virtual block can include any number of layers, in addition to including three layers.

[0032] The size of a virtual block (e.g., virtual block 202) can be determined based on computational complexity and budget. For example, introducing a virtual block into a convolutional neural network will introduce a certain amount of computational complexity, primarily attributable to the prediction layer. If the computational budget is C, and each block uses c, then the maximum number of virtual blocks is C / c. In this way, there is a trade-off between the number of virtual blocks and the allowed additional computational complexity.

[0033] like Figure 2 As shown, virtual block 202 and the last layer in virtual blocks 204 through 208 are connected to another layer group. The original last layer of the neural network is also connected to a similar layer group 240. Figure 2 In the example, the last convolutional layer 212 in virtual block 202 is connected to layer group 220. For the purposes of this description, reference will be made only to layer group 220. The descriptions of the other layer groups connected to the last layer of a corresponding virtual block in virtual blocks 204 through 208 are substantially the same.

[0034] like Figure 2 As shown, layer group 220 includes a pooling layer 221, a fully connected (FC) layer 222, and a scaling layer 223. In one embodiment, pooling layer 221 is an average pooling layer. Scaling layer 223 is associated with a scaling factor λ1. The scaling factors associated with the respective layer group are shown as λ0 to λ1. N Layer group 220 is added to the output of the last layer 212 of block 202 to achieve the same prediction task as the original neural network. Therefore, layer group 220 is consistent with the final prediction layer of the original neural network.

[0035] Scaling layer 223 is included in layer group 220 to determine an importance factor, also known as a scaling factor. Scaling layer 223 is a trainable scalar multiplied by the previous feature map.

[0036] like Figure 2 As shown, the output of the scaling layer, or the scaled output, of each virtual block 202 to 208 is summed 230. During training, the output of summation 230 is forwarded to prediction layer 232 (also known as the softmax layer) for prediction. This is to measure the contribution of each virtual block 202 to 208 to the final prediction relative to the ground truth during training. As will be understood, each neural network performs a specific prediction task, such as image classification or face detection. During training, virtual blocks that contribute higher accuracy relative to the ground truth are assigned a higher scaling factor, i.e., higher importance. Virtual blocks that contribute lower accuracy are assigned a lower scaling factor.

[0037] This aspect of the disclosed embodiments provides a layer-wise neuron decoupling method. For each neuron in a layer (e.g., layer 212 in virtual block 202), decoupling regularization is applied to reduce dependencies between neurons of the same layer (e.g., layer 212). For example, decoupling regularization for L layers is defined as:

[0038]

[0039] where W is the weight matrix of each layer, I is the identity matrix, and ||·||F is the Frobenius norm. This formulation enforces the orthogonality of column vectors and row vectors in the matrix W during training, which is designed to cover both overcomplete and undercomplete W cases. Undercomplete means #rows >= #columns, and overcomplete means #rows < #columns. F

[0040] Training:

[0041] The neural network with virtual blocks 202-208 and additional layers (e.g., layer 220) is trained in a manner similar to the original task by optimizing the following loss function:

[0042]

[0043] where, is the task-based loss (e.g., cross-entropy loss for image classification), is a regularization term that promotes sparsity of the scaling layer, defined as the L-1 norm of the scaling factor:

[0044]

[0045] and a2 are parameters of the regularization term.

[0046] Pruning phase:

[0047] The block-wise importance estimation and neural decoupling phases are substantially simultaneous during the training phase. The pruning phase is a post-training process. Redundancy of neurons is determined during training, and pruning is used to remove less important neurons without compromising accuracy after training.

[0048] Scaling factor l i First, normalize with respect to all scaling factors:

[0049]

[0050] Then, the pruning process is executed as follows:

[0051]

[0052] ​D, as defined in Algorithm 1, computes the correlation between the weight vectors of two neurons. In this example, the correlation values D are sorted from large to small. Small correlation values D are pruned. The pruned network can be fine-tuned to meet certain target accuracy requirements.

[0053] Figure 3 is a flowchart of a method provided by aspects of the disclosed embodiments. For a given convolutional neural network or deep learning network, the layers of the neural network are grouped into virtual blocks (302). An importance measure for each block is determined (304). Layer-wise neuron decoupling is applied (306). Training is performed as with the original task by optimizing a loss function (308). Then, neurons of the neural network are randomly pruned based on the block-wise importance measure applied to the virtual blocks and the diversity of neurons in the layers of the virtual blocks (310).

[0054] Aspects of the disclosed embodiments measure the importance of neurons at the block level or block-wise based on the contribution of the blocks to a target accuracy. By applying diversity regularization to each neuron in a layer, the dependency between neurons in the same layer is reduced. Then, neurons are spliced from the neural network based on the block-wise importance measure and the diversified neurons. By pruning neurons based on their diversity in a random manner rather than a deterministic manner, the false pruning of neurons is reduced.

[0055] While there have been shown and described and pointed out fundamental novel features as applied to example embodiments of the application, it will be understood that various omissions and substitutions and changes in the form and details of the devices illustrated, and in their operation, can be made by those skilled in the art without departing from the spirit and scope of the application. Further, it is expressly intended that all combinations of these elements which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the application. Moreover, it should be recognized that structures and / or elements from one disclosed or described form or embodiment of the application can be incorporated into any other disclosed or described or suggested form or embodiment as equipment design choices. Further, as will be obvious from context, they can be used alone or in combination with other features and / or elements. It is therefore clear that the application is intended to be as broad as possible and is limited only by the following claims, which fully and completely recite the full scope of the application.

Claims

1. An apparatus (100) for optimizing a convolutional neural network for image classification, characterized in that, comprising a processor (102) configured to: group layers of the convolutional neural network into one or more virtual blocks, wherein each virtual block of the one or more virtual blocks comprises at least one layer; determine a measure of importance of a virtual block of the one or more virtual blocks; apply diversity regularization to neurons in a layer of the virtual block to reduce dependencies between neurons in the layer; prune neurons from the layer based on the determined measure of importance of the virtual block and diversity of neurons in the layer; input image data to the pruned convolutional neural network for image classification.

2. The apparatus (100) according to claim 1, characterized in that a last convolutional layer of each virtual block is connected to a corresponding layer in a second set of layers, wherein the second set of layers comprises an average pooling layer, a fully connected layer, and a scaling layer.

3. The apparatus (100) according to claim 1 or 2, characterized in that The processor (102) is further configured to determine the measure of importance of the one or more virtual blocks by: summing outputs of scaling layers associated with each virtual block of the one or more virtual blocks; forwarding the summed outputs to a prediction layer for accuracy prediction during training; assigning higher scaling factors to virtual blocks that contribute higher accuracy during training and lower scaling factors to virtual blocks that contribute lower accuracy during training.

4. The apparatus (100) according to any one of the preceding claims, characterized in that The diversity regularization is defined as: where L is the number of layers, W is the weight matrix of each layer L, I is the identity matrix, is the Frobenius norm.

5. The apparatus (100) according to any one of the preceding claims, characterized in that The processor (102) is configured to prune neurons from the layer based on the determined measure of importance of the virtual block and diversity of neurons in the layer by: selecting the virtual block from the one or more virtual blocks; selecting the layer from the selected virtual block; computing a correlation value D between weight vectors of two neurons of the selected layer; ordering the correlation values D in ascending order; pruning a neuron from the selected layer based on the correlation values D.

6. The apparatus (100) according to claim 5, characterized in that The virtual blocks are randomly selected from the one or more virtual blocks.

7. A method (300) of optimizing a convolutional neural network for image classification, characterized in that, comprising: grouping (302) layers of the convolutional neural network into one or more virtual blocks, wherein each virtual block of the one or more virtual blocks comprises at least one layer; determining (304) a measure of importance of a virtual block of the one or more virtual blocks; applying (306) diversity regularization to neurons in a layer of the virtual block to reduce dependencies between neurons in the layer; pruning (310) neurons from the layer based on the determined measure of importance of the virtual block and diversity of neurons in the layer; inputting image data to the pruned convolutional neural network for image classification.

8. The method (300) according to claim 7, characterized by Determining the measure of importance of the one or more virtual blocks comprises: summing outputs of scaling layers associated with each virtual block of the one or more virtual blocks; forwarding the summed outputs to a prediction layer for accuracy prediction during training; assigning higher scaling factors to virtual blocks that contribute higher accuracy during training and lower scaling factors to virtual blocks that contribute lower accuracy during training.

9. The method (300) according to claim 7 or 8, characterized by, The diversity regularization is defined as: where L is the number of layers, W is the weight matrix of each layer L, I is the identity matrix, is the Frobenius norm.

10. The method (300) according to any one of claims 7 to 9, characterized by, pruning neurons from the layer based on the determined importance measure of the virtual block and a diversity of neurons in the layer further comprises: selecting the virtual block from the one or more virtual blocks; selecting the layer from the selected virtual block; computing a correlation value D between weight vectors of two neurons of the selected layer; ordering the correlation values D in ascending order; pruning neurons from the selected layer based on the correlation values D.

11. The method (300) according to claim 10, characterized by further comprising randomly selecting the virtual block from the one or more virtual blocks.

12. A computer program product, characterised in that, comprising a non-transitory computer readable medium for executing machine readable instructions to perform the method according to claims 7 to 11.

Images