Method and apparatus for compressing neural networks

Abstract

The application discloses a method and device for compressing a neural network, wherein the method comprises: obtaining a neural network to be compressed comprising a plurality of parameter values to be compressed; dividing the plurality of parameter values to be compressed into J blocks to be compressed; compressing the Jth block to be compressed using K j different compression rates to generate K j operation branches; obtaining K j weight factors corresponding to the K j operation branches; replacing the block to be compressed with the K j operation branches to generate a replaced neural network; performing forward propagation on the replaced neural network, weighting and summing K j operation results of the K j operation branches and the K j weight factors as an output of the Jth block to be compressed; performing backward propagation on the replaced neural network, calculating update values of the K j weight factors based on a model loss of the replaced neural network relative to the neural network to be compressed; and determining, for the Jth block to be compressed, an operation branch corresponding to a largest one of the update values of the K j weight factors as a compressed Jth block.

Description

TECHNICAL FIELD

[0001] The present application relates to neural network technology, and more particularly, to a method and apparatus for compressing a neural network. BACKGROUND

[0002] In recent years, neural network technology has been applied to many technical fields, such as image recognition, speech recognition, autonomous driving, medical imaging, and the like. For example, a convolutional neural network (CNN) as a representative network structure and algorithm in neural network technology has achieved great success in image processing applications. However, due to the deep number of layers, the large number of parameters, and the large amount of computation of the neural network, its application range is limited. SUMMARY

[0003] It is an object of the present application to provide a method for compressing a neural network, which can trade off a lower accuracy loss for a higher compression effect.

[0004] According to some aspects of the present application, a method for compressing a neural network is provided. The method comprises: obtaining a neural network to be compressed, the neural network to be compressed comprising one or more operation layers to be compressed, each operation layer to be compressed comprising one or more operation channels to be compressed, each operation channel to be compressed comprising one or more operation cores to be compressed, each operation core to be compressed comprising one or more parameter values to be compressed; dividing the plurality of parameter values to be compressed into J blocks to be compressed, where J is an integer greater than 1; compressing the J blocks to be compressed using K j different compression rates to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1; for the jth block to be compressed, obtaining a set of K j weight factors corresponding to the K j operation branches respectively, the K j weight factors having initial values respectively; for the jth block to be compressed, replacing the block to be compressed with the K j operation branches weighted by the set of K j weight factors, to generate a replaced neural network; performing forward propagation on the replaced neural network on a preset data set, wherein for the jth block to be compressed, combining K j operation results of the K j operation branches to obtain an output of the jth block to be compressed; and determining a weight factor in the set of K jsumming the K j weight factors as the output of the jth to-be-compressed block; performing back propagation on the replaced neural network on a preset data set, wherein for the jth to-be-compressed block, updating values of the K j weight factors are calculated based on a model loss of the replaced neural network relative to the to-be-compressed neural network; and for the jth to-be-compressed block, determining an operation branch corresponding to the maximum one of the updating values of the K

[0005] According to some aspects of the present disclosure, there are provided apparatuses for compressing a neural network. The apparatuses include: an obtaining unit configured to obtain a to-be-compressed neural network, the to-be-compressed neural network including one or more to-be-compressed operation layers, each to-be-compressed operation layer including one or more to-be-compressed operation channels, each to-be-compressed operation channel including one or more to-be-compressed operation cores, and each to-be-compressed operation core including one or more to-be-compressed parameter values; a block dividing unit configured to divide the plurality of to-be-compressed parameter values into J to-be-compressed blocks, where J is an integer greater than 1; a compressing unit configured to compress, for a jth to-be-compressed block of the J to-be-compressed blocks, using K j different compression rates to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1; a weight factor obtaining unit configured to, for the jth to-be-compressed block, obtain a set of K j weight factors corresponding to the K j operation branches respectively, the K j weight factors having initial values respectively; a replacing unit configured to, for the jth to-be-compressed block, replace the to-be-compressed block with the K j operation branches weighted by the set of K j weight factors to generate a replaced neural network; a forward propagating unit configured to perform forward propagation on the replaced neural network on a preset data set, wherein for the jth to-be-compressed block, K j operation results of the K j operation branches are summed with the K j weight factors as the output of the jth to-be-compressed block; a back propagating unit configured to perform back propagation on the replaced neural network on the preset data set, wherein for the jth to-be-compressed block, updating values of the K j weight factors are calculated based on a model loss of the replaced neural network relative to the to-be-compressed neural network; and a determining unit configured to, for the jth to-be-compressed block, determine an operation branch corresponding to the maximum one of the updating values of the K jThe operation branch corresponding to the maximum one of the updated values of the weight factors is determined as the jth compressed block.

[0006] According to still another aspect of the present application, an electronic device is provided, comprising: a processor; and a storage device for storing a computer program capable of running on the processor; wherein when the computer program is executed by the processor, the processor performs the method for compressing a neural network described above.

[0007] According to still another aspect of the present application, a non-volatile computer readable storage medium is provided, having a computer program stored thereon, which, when executed by a processor, implements the method for compressing a neural network described above.

[0008] The technical solution of the present application, when compressing a neural network, introduces weight factors, re-trains the to-be-compressed blocks with multiple different compression rates on a preset data set, updates the weight factors based on model loss, and selects the operation branch corresponding to the maximum weight factor in each to-be-compressed block as the compressed block, thereby being able to select a suitable compression rate for multiple to-be-compressed blocks while taking into account the accuracy rate, and greatly reducing the amount of computation.

[0009] In addition, the technical solution of the present application, when compressing a neural network, is able to update different preset compression rates, thereby expanding the range of selectable compression rates, and being able to select a more optimal compression rate for the to-be-compressed block.

[0010] The above is a summary of the present application, which may be simplified, generalized and omitted details, so those skilled in the art should recognize that this part is only illustrative and is not intended to limit the scope of the present application in any way. This summary part is neither intended to determine the key features or essential features of the claimed subject matter, nor intended to serve as an auxiliary means to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other features of the present application will become more fully understood from the following detailed description and the accompanying drawings, wherein: It can be understood that these drawings only depict several embodiments of the present application and therefore should not be considered as limiting the scope of the present application. The present application will be more clearly understood through the use of the accompanying drawings, in which:

[0012] Figure 1 A flowchart of a method for compressing a neural network according to an embodiment of the present application is shown;

[0013] Figure 2 An exemplary neural network structure diagram is shown;

[0014] Figure 3 An exemplary convolutional layer structure schematic diagram is shown.

[0015] Figure 4 A schematic diagram of pruning the convolution kernel with different compression rates is shown.

[0016] Figure 5 (a) and Figure 5 (b) shows a logic block diagram for forward propagation of a neural network according to an embodiment of the present application.

[0017] Figure 6 A flowchart of a method for updating the compression rate according to an embodiment of the present application is shown.

[0018] Figure 7 A performance comparison diagram of the neural network compression method according to the present application and the conventional sensitivity analysis-based compression method is shown.

[0019] Figure 8 (a), 8(b), 8(c), 8(d) and 8(e) respectively show the distribution of the actual pruning rate of each operation layer under different pruning rates, Figure 8 (f) shows the proportion of the parameter value of each operation layer of the neural network model before compression to the total parameter value of the model.

[0020] Figure 9 A distribution diagram of the equal error rate of the neural network compression method of the present application under different pruning rates is shown; and

[0021] Figure 10 A block diagram of an apparatus for compressing a neural network according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and form part of this disclosure.

[0023] In order to facilitate the deployment of neural networks on hardware, the neural networks are usually compressed in the prior art, for example, pruning and quantization. Among them, pruning refers to performing a pruning operation (for example, setting to zero) on a unit in the neural network that has little or no contribution to the output accuracy, so as to achieve the purpose of compressing the neural network. As a special form of pruning, structured pruning can further improve the running rate of the pruned neural network on hardware, because the pruned units have certain regularity in the operation space. Taking a convolution layer as an example, a common operation in structured pruning is to prune the entire convolution kernel level or the entire filter. Before the pruning operation, sensitivity analysis is usually required to determine the compression rate of each convolution layer. A higher compression rate is set for a convolution layer that has a greater impact on the output accuracy of the neural network, and fewer weight values are pruned. A lower compression rate is set for a convolution layer that has a smaller impact on the output accuracy of the neural network, and more weight values are pruned. In this application, the compression rate refers to the ratio of the data size after compression to the data size before compression. Then, based on the compression rate of each convolution layer, a pruning operation is performed on a plurality of convolution kernels or a plurality of filters in the convolution layer.

[0024] However, the inventors of the present application found that the structured pruning method of the prior art has many defects. First, sensitivity analysis needs to traverse each operation layer of the neural network, and each operation layer needs to traverse different compression rates. The output accuracy of the neural network needs to be evaluated on a test data set at each compression rate of each operation layer, resulting in excessive computational load. Although the number of compression rates that need to be traversed at each operation layer can be reduced in order to reduce the computational load, the reduction in the number of compression rates will reduce the accuracy of the sensitivity analysis of the operation layer. Second, the sensitivity analysis of the prior art is performed independently between each operation layer, but the operation layers of the neural network usually have a coupling effect, and the prior art does not consider the influence of the coupling effect on the sensitivity of each operation layer. Third, the compression rate of the operation layer usually has a positive correlation with the sensitivity, but it is very difficult to establish a quantitative relationship between the compression rate and the sensitivity. In actual operation, it needs to be set based on the experience of technical personnel, which has great limitations.

[0025] Based on the above research, the application provides a method for compressing a neural network. In the method, after obtaining a neural network to be compressed, a plurality of parameter values in the neural network to be compressed are divided into a plurality of compressed blocks to be compressed, each compressed block to be compressed is compressed using one or more different compression rates to generate one or more operation branches corresponding to each compressed block to be compressed; and for each compressed block to be compressed, one or more weight factors corresponding to the one or more operation branches are generated; then, the neural network is retrained on a preset data set, the retraining including forward propagation and backward propagation, wherein for each compressed block to be compressed, the one or more operation branches are used to replace the compressed block to be compressed in the forward propagation process, and one or more operation results of the one or more operation branches are weighted and summed with the one or more weight factors as the output of the compressed block to be compressed; in the backward propagation process, an update value of the one or more weight factors is calculated based on a model loss related to the performance of the neural network; then for each compressed block to be compressed, the operation branch corresponding to the largest one of the update values of the one or more weight factors is determined as the compressed block, thereby forming a compressed neural network. The method can retrain the neural network on a small sample of a preset data set by introducing a weight factor, update the weight factor based on the model loss, and select the operation branch corresponding to the largest weight factor in each block as the compressed block, thereby selecting a suitable compression rate for multiple blocks that can take into account the accuracy rate. Compared with the method of determining the sensitivity of each operation layer by traversing the compression rate in the prior art, the method of the application greatly reduces the amount of calculation, which is mainly because the weight factors of multiple compressed blocks to be compressed can be updated at the same time, and the coupling effect between the multiple compressed blocks to be compressed is considered; further, in the method of the application, the weight factor is updated based on the model loss, rather than the experience of the technician, and does not require human intervention, so its applicability is wider.

[0026] The method for compressing a neural network of the application will be described in detail below with reference to the accompanying drawings. Figure 1 A flowchart of the method 100 for compressing a neural network according to some embodiments of the application is shown, which specifically includes the following steps 110-190.

[0027] Step 110, obtaining a neural network to be compressed, the neural network to be compressed including a plurality of compressed parameter values to be compressed.

[0028] In some embodiments, the neural network to be compressed can be a deep neural network (DNN) obtained after training using a training sample data set, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), a restricted Boltzmann machine (RBM), a fully connected network (FCN), a deep convolutional network (DCN), a long short-term memory (LSTM) network or a gated recurrent unit (GRU), etc.

[0029] Figure 2 An exemplary to-be-compressed neural network structure diagram is shown, which includes multiple operation layers 220-1, 220-2, …, 220-L capable of sequentially processing input data 210 received by the neural network to output output data 230 after processing. In some embodiments, the to-be-compressed neural network can be a convolutional neural network, such as LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, or other types of convolutional neural networks, and the multiple operation layers can include convolutional layers (Conv), batch normalization layers (BN: Batch Normalization), activation layers (e.g., ReLU layers), pooling layers (Pooling), fully connected layers, and / or other types of operation layers. Each operation layer can include one or more operation channels, each of which can include one or more operation cores, each of which can include or be configured with one or more parameter values; various operation layers can include different types or numbers of parameter values for participating in the operation of the to-be-compressed neural network. Taking a convolutional layer as an example, a convolutional layer can include multiple filters (i.e., operation channels), each of which can include multiple convolutional kernels (i.e., operation cores), each of which can include multiple weight values (i.e., parameter values) arranged in a matrix, such as 3x3, 5x5, or 7x7 matrix, which are used as parameter values of the convolutional layer to participate in the operation of the to-be-compressed neural network. For image processing type convolutional neural networks, the input data 210 can be an image, which can be represented in the form of a pixel matrix and can include multiple different color channels (e.g., R, G, B color channels). The operation layer 220-1 can be a convolutional layer including one or more filters to perform convolution operations on the input data 210 to generate multiple feature Figure 1 maps as output of the operation layer 220-1; then, the feature Figure 1 maps are provided to the next operation layer 220-2 (e.g., a batch normalization layer, an activation layer, a pooling layer, or a fully connected layer) and continue to be processed as its input data to generate multiple feature Figure 2This process continues until output data 230 is generated after processing by operational layers 220-L, where L can be an integer greater than 1. The output data 230 will vary depending on the application of the convolutional neural network. For example, for an image classification network, output data 230 may be the classification of targets in the input image into various different categories; for an image detection network, output data 230 may be the detection results of targets in the input image. In other embodiments, input data 210 and output data 230 may also be audio data, text data, etc., depending on the specific application scenario of the convolutional neural network. It should be noted that although the above embodiments are illustrated using convolutional neural networks as an example, those skilled in the art will understand that the method for compressing neural networks in this application is applicable to any neural network containing operational layers to be compressed.

[0030] In some embodiments, the neural network to be compressed includes one or more computational layers to be compressed, each computational layer to be compressed includes one or more computational channels to be compressed, each computational channel to be compressed includes one or more computational kernels to be compressed, and each computational kernel to be compressed includes one or more parameter values ​​to be compressed. In some embodiments, the parameter values ​​in all computational layers of the neural network to be compressed are parameter values ​​to be compressed. In some embodiments, only the parameter values ​​in some computational layers, some computational channels, or some computational kernels are used as parameter values ​​to be compressed. Whether to compress all parameter values ​​in the neural network to be compressed, or which computational layers, computational channels, or computational kernels are selected for compression, can vary based on the characteristics of the neural network to be compressed or the specific application requirements.

[0031] Step 120: Divide the multiple parameter values ​​to be compressed into J blocks to be compressed, where J is an integer greater than 1.

[0032] In some embodiments, dividing multiple parameter values ​​to be compressed into J blocks means dividing the parameter values ​​to be compressed in the same computational layer into a single block. For example, if the neural network to be compressed includes L layers to be compressed, then the J blocks to be compressed are J computational channels to be compressed.

[0033] by Figure 2 For example, if all parameter values ​​in operation layers 220-1, 220-2, ..., 220-L are parameter values ​​to be compressed, then the parameter values ​​to be compressed in operation layer 220-1 are treated as a single block to be compressed, the parameter values ​​to be compressed in operation layer 220-2 are treated as a single block to be compressed, ..., and so on, until L blocks to be compressed are generated, at which point J = L.

[0034] In other embodiments, dividing multiple parameter values ​​to be compressed into J blocks means dividing the parameter values ​​to be compressed in the same computation channel into a single block. For example, if the neural network to be compressed includes L layers to be compressed, and each layer includes C computation channels to be compressed, then the J blocks to be compressed are L×C computation channels to be compressed.

[0035] refer to Figure 3 Taking convolutional layer 320 as an example, for Figure 2 The computational layers shown are illustrated. Convolutional layer 320 can be... Figure 2 The shown is a certain computational layer.

[0036] like Figure 3 As shown, the convolutional layer 320 is located between the input feature map 310 and the output feature map 330. It is used to perform convolution operations on the data input to the input feature map 310 to generate an operational structure, which is then output by the output feature map 330. Figure 3 In the example shown, convolutional layer 320 may include five operational channels, namely filters 320-1, 320-2, 320-3, 320-4, and 320-5, which perform convolution operations with the data of input feature map 310, and the results are represented as the five output channels 330-1, 330-2, 330-3, 330-4, and 330-5 of output feature map 330. Each of filters 320-1, 320-2, 320-3, 320-4, and 320-5 may include six convolutional kernels, which are used to perform convolution operations with the six input channels 310-1, 310-2, 310-3, 310-4, 310-5, and 310-6 of input feature map 310, respectively. Each convolutional kernel includes 3 rows and 3 columns of weight values.

[0037] by Figure 3 For example, if the weight values ​​in filters 320-1, 320-2, 320-3, 320-4, and 320-5 of convolutional layer 320 are all parameters to be compressed, then the weight values ​​in filter 320-1 are treated as a single block to be compressed, the weight values ​​in filter 320-2 are treated as a single block to be compressed, and so on, until five blocks to be compressed are generated in convolutional layer 320. The blocks to be compressed can be similarly divided for other operational layers outside of convolutional layer 320.

[0038] In other embodiments, dividing multiple parameter values ​​to be compressed into J blocks means dividing the parameter values ​​to be compressed within the same computational core into a single block. For example, if the neural network to be compressed includes L layers to be compressed, each layer includes C computational channels to be compressed, and each computational channel includes Q computational cores to be compressed, then the J blocks to be compressed are L×C×Q computational cores to be compressed.

[0039] Continuing with the example, Figure 3 if the weight values in the filters 320-1, 320-2, 320-3, 320-4, and 320-5 of the convolution layer 320 are all to-be-compressed parameter values, and each filter includes 6 convolution kernels, then dividing the weight values in each convolution kernel into a separate to-be-compressed block will result in 5 x 6 = 30 to-be-compressed blocks corresponding to the convolution layer 320. For other operation layers other than the convolution layer 320, to-be-compressed blocks can be similarly divided.

[0040] In some embodiments, dividing the plurality of to-be-compressed parameter values into J to-be-compressed blocks includes dividing to-be-compressed parameter values in two or more adjacent operation layers, operation channels, or operation kernels into a separate to-be-compressed block. In other embodiments, dividing the plurality of to-be-compressed parameter values into J to-be-compressed blocks includes dividing to-be-compressed parameter values in two or more operation layers, operation channels, or operation kernels that are not adjacent but are of the same type into a separate to-be-compressed block, such as dividing to-be-compressed parameter values in two or more convolution layers into a separate to-be-compressed block. By processing to-be-compressed parameter values in two or more operation layers, operation channels, or operation kernels in a separate to-be-compressed block, unnecessary intermediate outputs can be reduced, and operation efficiency can be improved.

[0041] In the different embodiments described above, to-be-compressed parameter values are divided into different numbers of to-be-compressed blocks based on different “granularities” (i.e., dimensions of operation layers, operation channels, and operation kernels). A finer granularity helps to obtain a lower compression ratio, but increases the amount of computation; therefore, in actual applications, the granularity of division of to-be-compressed blocks can be adjusted based on the characteristics of the to-be-compressed neural network or specific application requirements. In addition, in the different embodiments described above, Figure 2 and Figure 3 The method of dividing a plurality of to-be-compressed parameter values into a plurality of to-be-compressed blocks is described in the present application, but, Figure 2 , Figure 3 and other related figures, the specific numbers of input feature maps, output feature maps, convolution layers, filters, convolution kernels, or feature values are only examples, and those skilled in the art can understand that in other embodiments they can also have other different numbers.

[0042] At step 130, the jth to-be-compressed block of the J to-be-compressed blocks is compressed using K j different compression ratios to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1.

[0043] The jth to-be-compressed block is compressed using K jSetting different compression rates in advance requires setting the number and values of different compression rates. The number of compression rates (i.e., K j ) can be 2, 3, 4, 5, 6, 7 or other suitable values. The selection of the number of compression rates will affect the number of operation branches subsequently formed, and too many will not only increase the operation burden but also increase the memory burden; while too few may not be able to filter out the best compression rate suitable for the block to be compressed. The value of the compression rate can be 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95 or other suitable values. If the value of the compression rate is set too large, the desired compression effect may not be achieved, and if the value of the compression rate is set too small, it will affect the output accuracy of the entire neural network. Therefore, in different embodiments, the number and value of the compression rate need to be selected according to the characteristics of different blocks to be compressed or specific application requirements. The number and / or value of the compression rate of different blocks to be compressed can be equal or unequal.

[0044] In some embodiments, the K j compression rates corresponding to the jth block to be compressed have respective initial values R j , R j + Δ, R j + 2Δ, …, R j + (K j - 1)Δ, where 0 < R j < 1, 0 < Δ < 1, and 0 < (R j + (K j - 1)Δ) < 1. That is, the multiple different compression rates of the same block to be compressed form an arithmetic sequence. In one example, K j = 3, R j = 0.2, and Δ = 0.15, then the corresponding three different compression rates are 0.2, 0.35 and 0.5.

[0045] If the K j compression rates are set in the form of an arithmetic sequence, the values of each compression rate can be controlled or adjusted by setting the values of R j and Δ without processing them one by one, which is particularly convenient when the number of compression rates is large. It should be noted that Δ is the difference between the compression rates of two adjacent operation branches in the block to be compressed, and can be regarded as the scanning step in the process of finding a suitable compression rate for the block to be compressed. Although a smaller scanning step is more conducive to finding a more suitable compression rate, it will correspondingly increase the amount of calculation; therefore, in different embodiments, the value of Δ can also be determined according to the characteristics of different blocks to be compressed or specific application requirements.

[0046] It should be noted that the multiple compression ratios in the form of an arithmetic sequence are only examples. In other embodiments, different methods can be used to select K from 0 to 1. j Different compression ratios are used for the j-th block to be compressed, and this application does not restrict the specific selection method.

[0047] After determining the j-th block to be compressed, K... j After using K at different compression ratios, j Compression ratios are applied to the j-th block to generate K. j Each operation branch. Corresponding to step 120, the parameter values ​​to be compressed are divided into J blocks to be compressed (e.g., L operation layers to be compressed, L×C operation channels to be compressed, or L×C×Q operation cores to be compressed) based on different "granularity". The compression methods for these J=L operation layers to be compressed, J=L×C operation channels to be compressed, or J=L×C×Q operation cores to be compressed are also different.

[0048] In the embodiment of dividing the blocks to be compressed based on the computation layer, firstly, the L1 norm of all computation channels of the j-th computation layer to be compressed is calculated; then, based on the K of the j-th computation layer to be compressed... j Each compression ratio determines the P1, P2, ..., P1 with the smallest L1 norm. Kj Each computing channel, specifically, can be based on K j A compression ratio determines K j The number of processing channels that need to be retained is then subtracted from the total number of processing channels by K. j The number of processing channels that need to be retained, thus generating K that needs to be compressed. j The number of processing channels P1, P2, ... and P1 are: Kj Next, the P1, P2, ..., P in the j-th layer to be compressed are... Kj After setting all weight values ​​in each calculation channel to zero, K is generated respectively. j There are 12 computational branches. Specifically, the j-th computational layer to be compressed is taken as... Figure 3 Taking the convolutional layer 320 shown as an example, the L1 norm of all filters 320-1, 320-2, 320-3, 320-4, and 320-5 is calculated. For example, the L1 norm of filter 320-1 is obtained by summing the absolute values ​​of all weights in the six convolutional kernels of filter 320-1; and so on, the L1 norms of filters 320-2, 320-3, 320-4, and 320-5 are obtained. Assume K... j= 3, corresponding to 3 different compression rates 0.2, 0.35 and 0.5 of the convolution layer 320; the convolution layer 320 is compressed using the 3 different compression rates 0.2, 0.35 and 0.5, and the number of filters to be reserved for the 3 operation branches generated subsequently is determined based on the 3 different compression rates 0.2, 0.35 and 0.5, respectively, as (5 x 0.2), (5 x 0.35) and (5 x 0.5), which are rounded to 1, 2 and 3, respectively; then it can be determined that the number of filters to be pruned for the 3 operation branches is 4, 3 and 2, respectively. Next, the 4, 3 and 2 filters with the smallest L1 norm among the 5 filters 320-1, 320-2, 320-3, 320-4 and 320-5 are determined, i.e., P1=4, P2=3 and P3=2. Assuming that the L1 norms of the 5 filters are in ascending order as the filter 320-1, the filter 320-2, the filter 320-3, the filter 320-4 and the filter 320-5, then all the weight values of the 4 filters (320-1, 320-2, 320-3 and 320-4) with the smallest L1 norm are set to zero to form the first operation branch; all the weight values of the 3 filters (320-1, 320-2 and 320-3) with the smallest L1 norm are set to zero to form the second operation branch; and all the weight values of the 2 filters (320-1 and 320-2) with the smallest L1 norm are set to zero to form the third operation branch.

[0049] In the embodiment of dividing the to-be-compressed block based on the operation channel, first, the L1 norm of all operation cores of the jth to-be-compressed operation channel is calculated; then, the P1, P2, …, P j operation cores with the smallest L1 norm are determined based on the K Kj compression rates of the jth to-be-compressed operation channel, specifically, the number of K j operation cores to be reserved can be determined based on the K j compression rates, and then the number of K j operation cores to be reserved is subtracted from the total number of operation cores, thereby generating the number P1, P2, …, and P j of operation cores to be compressed. Kj ; then, all the weight values of the P1, P2, …, P Kj operation cores in the jth to-be-compressed operation channel are set to zero, respectively, to generate K j operation branches. Specifically, the jth to-be-compressed operation channel is taken as an example. Figure 3 The L1 norm of each of the 6 convolution kernels (i.e., the convolution kernel 320-1-1 to the convolution kernel 320-1-6) is calculated. For example, the absolute values of the 9 weight values of each convolution kernel are added to obtain the L1 norm of the convolution kernel. Assuming that the K j=3, corresponding to three different compression ratios of 0.2, 0.35, and 0.5 for filter 320-1; using these three different compression ratios of 0.2, 0.35, and 0.5 to compress filter 320-1, and based on these three different compression ratios of 0.2, 0.35, and 0.5, the number of convolution kernels that need to be retained in the three subsequent operational branches are determined to be (6×0.2), (6×0.35), and (6×0.5), which, after rounding, are 1, 2, and 3 respectively; then the number of convolution kernels that need to be pruned in the three operational branches can be determined to be 5, 4, and 3 respectively. Next, the convolution kernels with the smallest L1 norm among the six convolution kernels are determined to be 5, 4, and 3, i.e., P1=5, P2=4, and P3=3. The first operational branch is formed by setting all weights in the five convolutional kernels with the smallest L1 norm to zero; the second operational branch is formed by setting all weights in the four convolutional kernels with the smallest L1 norm to zero; and the third operational branch is formed by setting all weights in the three convolutional kernels with the smallest L1 norm to zero.

[0050] In the embodiment of dividing the blocks to be compressed based on the computational core, firstly, the absolute values ​​of all the parameters to be compressed in the j-th computational core are calculated; then, based on the K of the j-th computational core... j Each compression ratio is used to determine the smallest absolute value of P1, P2, ..., P. Kj The parameter values ​​to be compressed, specifically, can be based on K. j A compression ratio determines K j The number of parameter values ​​that need to be retained is then subtracted from the total number of parameter values ​​for each K value. j The number of parameter values ​​that need to be retained results in K that needs to be compressed. j The number of parameter values ​​P1, P2, ... and P Kj Next, the P1, P2, ..., P in the j-th kernel to be compressed are... Kj After setting each parameter value to be compressed to zero, K is generated respectively. j Each computational branch. Specifically, refer to... Figure 3 and Figure 4 Taking the j-th core to be compressed as Figure 3 Taking the first convolution kernel 320-1-1 in the filter 320-1 shown as an example, calculate the absolute values ​​of all nine weight values ​​in the convolution kernel 320-1-1. Assume K... j= 3, corresponding to 3 different compression rates 0.2, 0.35 and 0.5 of the convolution kernel 320-1-1; the convolution kernel 320-1-1 is compressed using the 3 different compression rates 0.2, 0.35 and 0.5, and the number of weight values to be reserved for the 3 operation branches generated subsequently is determined based on the 3 different compression rates 0.2, 0.35 and 0.5, respectively, as (9x0.2), (9x0.35) and (9x0.5), and after rounding, as 2, 3 and 5, respectively; it can be determined that the number of weight values to be pruned for the 3 operation branches is 7, 6 and 4, respectively. Then, the 7, 6 and 4 weight values with the smallest absolute values among the 9 weight values are determined, i.e., P1=7, P2=6 and P3=4. Assuming that the 7 weight values with the smallest absolute values among the 9 weight values in the convolution kernel 320-1-1 are sequentially located at positions (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0) and (2, 2) from small to large; the first operation branch 320-1-1a is formed by setting the 7 weight values with the smallest absolute values to zero; the second operation branch 320-1-1b is formed by setting the 6 weight values with the smallest absolute values (located at (0, 1), (0, 2), (1, 0), (1, 1), (1, 2) and (2, 0)) to zero; and the third operation branch 320-1-1c is formed by setting the 4 weight values with the smallest absolute values (located at (0, 1), (0, 2), (1, 0) and (1, 1)) to zero.

[0051] It should be noted that in the above embodiments, the K j Kj When the P1, P2, …, P Kj operation channels, operation kernels or parameter values with the smallest L1-norm are determined based on the K Kj compression rates of the jth operation layer to be compressed, operation channel to be compressed or operation kernel to be compressed, respectively, the rounding method is used for rounding. In some other embodiments, the rounding-up or rounding-down approximation method can also be used.

[0052] In step 140, for the jth block to be compressed, a group of K j weight factors corresponding to the K j operation branches are obtained, where K j The initial values of the K

[0053] In some embodiments, the initial values of the weight factors can be automatically generated by software. In some other embodiments, the initial values of the weight factors can be defined and provided by a user. Since the sensitivity of each operation layer, operation channel or operation kernel to the compression rate is unknown before the neural network is retrained, in preferred embodiments, the initial values of the K j weight factors corresponding to the K j operation branches representing different compression rates are usually set to be equal.

[0054] Continue with K j = 3, and using 3 different compression rates 0.2, 0.35 and 0.5, 3 operation branches are compressed respectively, 3 weight factors can be set for the 3 operation branches, denoted as β1, β2, and β3. In an embodiment, the initial values of the 3 weight factors can be set as 0. It can be understood that in other examples, the initial values of the weight factors can also be set as other equal or unequal values.

[0055] At step 150, for the jth compressed block, the K j operation branches weighted by the K j weight factors are used to replace the compressed block, to generate a replaced neural network.

[0056] Referring to Figure 5 (a) and 5(b), taking the jth compressed block of the compressed neural network as the convolution channel conv j , for example, assuming that the jth compressed block is compressed using 3 different compression rates (for example, R j , R j + Δ and R j + 2Δ), to generate 3 operation branches conv j1 , conv j2 and conv j3 , then the 3 operation branches conv j1 , conv j2 and conv j3 weighted by the 3 weight factors β1, β2 and β3 are used to replace the compressed convolution channel conv j . After the above replacement operation is completed for all J compressed blocks, a replaced neural network is generated.

[0057] Although the above is described taking the compressed block as the convolution channel, it can be understood that similar replacement operations can also be implemented for the convolution layer or the convolution kernel, and for the corresponding operation layer, operation channel or operation kernel of the batch normalization layer, activation layer or fully connected layer, etc.

[0058] Still referring to Figure 1 , at step 160, the replaced neural network is forward propagated on the preset data set, wherein for the jth compressed block, the K j operation results of the K j operation branches are weighted and summed with the K j weight factors as the output of the jth compressed block.

[0059] The preset dataset can be a dataset different from the training sample dataset used to train the deep neural network. The preset dataset can include a plurality of training examples, each example consisting of an input object and a desired output value. Forward propagation of the neural network on the preset dataset can generate an output value for the input object of each example, and by comparing the output value with the desired output value, the accuracy of the neural network can be evaluated. For example, for an image classification network, the input object is an image, and the output of the forward propagation is to classify the target in the input image into a plurality of different classification types, and by comparing the classification to which the output belongs with the expected classification result, the parameter value of the image classification network can be adjusted or its performance can be evaluated. Similarly, if the neural network is used in a speech recognition system, the input object is a speech sample, and the expected output is the corresponding text. In some examples, the aforementioned preset dataset can be an ImageNet dataset, or a CIFAR10 dataset, etc. It can be understood that those skilled in the art will be able to select a suitable dataset for retraining of the neural network according to the specific application, and examples are not listed here.

[0060] With reference to Figure 5 (a) and Figure 5 (b), the to-be-compressed block conv j is replaced with three operation branches conv j1 , conv j2 and conv j3 weighted using three weight factors β1, β2 and β3. Figure 5 (a) shows the normal forward propagation process of the jth to-be-compressed block conv j , Figure 5 (b) shows the forward propagation process after the jth to-be-compressed block conv j is replaced with three operation branches conv j1 , conv j2 and conv j3 weighted using three weight factors β1, β2 and β3 when K j = 3.

[0061] In Figure 5 (a) and Figure 5 (b), input j represents the input data of the jth to-be-compressed block. First, the K j replaced operation branches are operated with the input data input j , to generate operation results corresponding to the K j operation branches. These K j operation results are then combined with the corresponding K jA weighted sum is generated by weighting each weight factor, and this weighted sum can be used as the output of the block to be compressed. The j-th block to be compressed is used as the convolution channel (conv). j For example, the above calculation and weighting steps can be represented by formula (1):

[0062]

[0063] in, This represents the k-th computational branch conv in the j-th computational layer. jk with input data j Perform convolution operation, α jk Weighting factor β jk The normalized representation of represents the k-th normalized weight factor in the j-th block to be compressed. In some examples, the weight factor β is... jk Normalization can be performed using the softmax function below, i.e., formula (2):

[0064]

[0065] It can be understood that the weighting factor β jk Normalization can also be performed using other functions depending on the specific application.

[0066] Continue with Figure 5 (b) For example, when K j When =3, the above formula (1) can be expressed as the following formula (3):

[0067]

[0068] The specific calculation process will not be elaborated here.

[0069] It should be noted that, in Figure 5 In the examples of (a) and 5(b), the block to be compressed is used as the convolution channel. However, it is understood that a similar approach can be used for forward propagation of convolutional layers or kernels, as well as the corresponding computational layers, computational channels, or computational kernels of activation layers or fully connected layers.

[0070] Step 170: Perform backpropagation on the replaced neural network on the preset dataset. For the j-th block to be compressed, calculate K based on the model loss of the replaced neural network relative to the original neural network. j Updated values ​​of each weight factor.

[0071] After pruning the parameter values of the to-be-compressed neural network, the accuracy of the to-be-compressed neural network is lost while saving computing and storage resources. Therefore, the replaced neural network can be back propagated to adjust the parameters of the replaced neural network, thereby reducing the accuracy loss. It should be noted that when the replaced neural network is back propagated, only the weight factors of each to-be-compressed block are updated, and other parameters (for example, convolution layer weight values) of the neural network are not adjusted.

[0072] In some embodiments, the back propagation of the replaced neural network updates the K j weight factors of the jth to-be-compressed block can use the following formula (4):

[0073]

[0074] wherein β jk represents the initial value of the kth weight factor of the jth to-be-compressed block, β′ jk represents the updated value of the kth weight factor of the jth to-be-compressed block, Loss represents a function of the model loss of the replaced neural network relative to the original to-be-compressed neural network, and λ represents a learning rate parameter and can be defined by the user.

[0075] In some embodiments, the model loss function Loss includes a loss function term and a performance indicator term. The loss function term is related to the application type of the neural network, such as classification, positioning, detection, or segmentation. The performance indicator term is related to the hardware indicators of the hardware platform to which the neural network is to be deployed, such as storage space, floating point operation times, delay, or energy consumption, etc.

[0076] The model loss function Loss is described below using the storage space of a compressed classification neural network on a hardware platform as an example. Specifically, the model loss function Loss can be represented by the following formula (5):

[0077] Loss = loss 1 + loss 2 (5)

[0078] wherein loss1 represents the loss function term, and loss2 represents the performance indicator term.

[0079] For a classification neural network, the loss function term loss1 can be further represented by the cross-entropy function shown in formula (6):

[0080]

[0081] wherein label is a classification label, usually represented by one-hot code; I represents the total dimension of the classification label, for example, for the ImageNet dataset, the total dimension I of the classification label is 1000; label(i) represents the i-th classification among all I classification labels, if the i-th classification is true, label(i) is 1, otherwise label(i) is 0; correspondingly, pred(i) represents the probability of identifying the i-th classification.

[0082] The performance index item loss2 can be further represented by a model size function shown in formula (7):

[0083] loss2 = (model_size / model_ref) γ (7)

[0084] wherein model_size represents the model size of the neural network in training, in unit of "bit"; model_ref represents the model size of the expected compressed neural network; γ is a hyperparameter, used to balance the accuracy loss of the neural network model and the model size. When a neural network model with higher accuracy is expected, a smaller γ value can be set, and when a smaller neural network model is expected, a larger γ value can be set. Therefore, the value of γ can be selected according to specific requirements. In some examples, the value of γ is 0.05-5. More specifically, in some examples, the value of γ is 2-3, which can better balance the accuracy loss of the neural network model and the model size.

[0085] In some embodiments, the model size model_size of the neural network in training in the above formula (7) can be represented by the following formula (8):

[0086]

[0087] wherein J represents the total number of compressed blocks in the neural network, j represents the number of each compressed block, then 1≤j≤J; K j represents the total number of operation branches in the j-th compressed block, k represents the number of operation branches in the compressed block, then 1≤k≤K j ; conv jk represents the convolution layer of the k-th operation branch in the j-th compressed block; size(conv jk ) represents the bit number of the convolution layer conv jk , for example, size(conv Figure 5 ) represents the bit number of the first convolution layer in the first operation branch of the first compressed block, size(conv j1 ) represents the bit number of the second convolution layer in the second operation branch of the first compressed block, and size(conv j2 ) represents the bit number of the third convolution layer in the third operation branch of the first compressed block. j3The values of the three terms on the right side of the equation respectively equal the number of non-zero weight values in the three operation branches multiplied by the bit width (e.g., 32 bits, 16 bits, 8 bits, or 4 bits, etc.) corresponding to the non-zero weight values. In addition, a jk The normalization weight factor corresponding to the kth operation branch in the jth operation layer can be obtained by the aforementioned equation (2).

[0088] The above example illustrates the model loss function Loss in terms of the storage space of the compressed classification neural network on a hardware platform. Those skilled in the art can understand that the model loss function Loss can be modified accordingly according to the specific application scenario. In some embodiments, the loss function term loss1 in the model loss function Loss can be adjusted according to the application type of the neural network (e.g., classification, positioning, detection, or segmentation, etc.). Taking image processing as an example, a classification neural network can be used to determine the category to which a given image belongs; a positioning neural network can be used to determine the specific position of a target in a given image; a detection neural network can be used to identify and locate a target in a given image when the category and number of targets are unknown; and a segmentation neural network can segment a given image into multiple components to locate the boundary of a target in the image. Similar to the loss function term loss1 of the classification neural network described above, which is related to the classification label and the probability of each classification identified, when adjusting the loss function term loss1, the positioning neural network needs to consider the position parameters of the target in the image; the detection neural network needs to consider not only the category and number of targets, but also the positions of the targets in the image; and the segmentation neural network needs to consider the position parameters and shape parameters of the target in the image. In some embodiments, according to the specific application type of the neural network, the loss function term loss1 can be selected from one or more of the cross-entropy function, the absolute value loss function, the logarithmic loss function, the square loss function, the exponential loss function, the hinge loss function, the perceptron function, or other suitable loss functions, which are not listed one by one here. Accordingly, the performance indicator term loss2 in the model loss function Loss can be adjusted according to the hardware indicators (e.g., storage space, number of floating point operations, delay, or energy consumption, etc.) of the hardware platform to be deployed. For example, if the indicators related to the number of floating point operations, the delay, or the energy consumption of the compressed neural network are set for the hardware platform to be deployed, the model_size term and the model_ref term in equation (7) related to the storage space can be replaced by parameters related to the number of floating point operations, the delay, or the energy consumption, thereby forming the adjusted performance indicator term loss2.

[0089] In some embodiments, after step 170 ends, the obtained K jThe operation branch corresponding to the maximum of the updated values of the K weight factors is determined as the compressed jth block. The j compressed blocks are used to replace the j blocks before compression, thereby obtaining a compressed neural network. This can reduce the amount of calculation required for compressing the neural network.

[0090] In the embodiment shown in FIG. 17, after step 170, the replaced neural network can be further retrained on the preset data set one or more times to obtain a compressed neural network with better performance. Figure 1

[0091] Step 180: The forward propagation and the backward propagation of the replaced neural network are iteratively executed multiple times on the preset data set. During the iterative execution, the K j weight factors are updated based on the operation branch corresponding to the maximum of the updated values of the K weight factors, and the size of the replaced neural network model is calculated. It is determined whether the difference between the sizes of the replaced neural network model calculated in the two iterations is within a preset numerical range. If yes, step 190 is executed; if no, step 160 is returned, and the forward propagation and the backward propagation of the replaced neural network are iteratively executed.

[0092] The retraining of the replaced neural network on the preset data set includes the forward propagation and the backward propagation, and the purpose is to update the multiple weight factors allocated to the multiple operation branches to balance the accuracy loss and the model size of the replaced neural network, and obtain an optimized mixed compression scheme. During the iterative execution, for the jth block to be compressed, the updated values of the K j weight factors generated by the previous backward propagation are assigned to the K j weight factors used in the subsequent forward propagation. After multiple iterations of the forward propagation and the backward propagation of the replaced neural network, if the difference between the size of the replaced neural network model in the subsequent iteration and the size of the replaced neural network model in the previous iteration is within a preset numerical range, it is indicated that the training converges, and the training can be ended; if not, the training of the replaced neural network needs to be continued, and the forward propagation and the backward propagation steps are continued to update the weight factors.

[0093] It should be noted that for the first time of executing the forward propagation and the backward propagation, since there is no previous iteration process, the initial size of the neural network model to be compressed can be used as the size of the replaced neural network model in the previous iteration process, which is used to calculate the difference between the sizes of the replaced neural network model calculated in the two iterations.

[0094] ​It should be further noted that in some cases, there can be a situation where the calculation does not converge, that is, the aforementioned difference value cannot converge to less than a preset numerical range after multiple iterations, therefore, the maximum number of iterations can be set in advance, and when the number of iterations reaches the maximum number of iterations, the iteration loop is forced to exit, and the retraining of the replaced neural network is ended.

[0095] In some embodiments, the model size of the replaced neural network can be calculated using formula (9):

[0096]

[0097] The same symbols in formula (9) as in formula (8) represent the same meaning, model_size' represents the model size of the replaced neural network, and α jk is determined by the following formula (10).

[0098]

[0099] Wherein, the operator arg max represents taking the maximum value of K j weight factors β jk . That is, when calculating the model size of the replaced neural network, only the size of the operation branch corresponding to the maximum weight factor in each to-be-compressed block is counted into the model size of the replaced neural network, and the other items are discarded. Through the above formulas (9) and (10), the model size of the replaced neural network can be calculated in each iteration process. The difference between the model sizes of the replaced neural network in the previous and subsequent iterations is compared with the preset numerical range set by the user, if the difference is within the preset numerical range, it means that the model size of the replaced neural network has no change or small change before and after iteration, and the training can be ended; if not, the training of the replaced neural network on the preset data set is continued.

[0100] In step 190, for the jth to-be-compressed block, the operation branch corresponding to the maximum of the updated values of K j weight factors is determined as the jth compressed block.

[0101] As described above, when the replaced neural network is retrained on the preset data set and reaches the convergence condition, it means that the accuracy loss and the model size of the replaced neural network have been relatively balanced. For the jth to-be-compressed block, the operation branch corresponding to the maximum of the updated values of K j weight factors can be directly determined as the jth compressed block, and the compressed neural network is formed by using all the J compressed blocks to replace the compressed blocks.

[0102] In combination Figure 1In the described embodiments, although the plurality of different compression rates can be used to compress each to-be-compressed block and the subsequent neural network can be used to retrain the optimal compression rate from the plurality of different compression rates to compress the to-be-compressed block, since the plurality of different compression rates are pre-set, there can be a more optimal compression rate outside the plurality of different compression rates. Therefore, it is necessary to update the plurality of different compression rates in time to seek a more optimal compression rate applicable to the to-be-compressed block.

[0103] Based on the above considerations, in some embodiments of the present application, the plurality of different compression rates are updated by referring to the following steps. Figure 6 Steps 172 to 178 are further performed between steps 170 and 180 to update the plurality of different compression rates for each to-be-compressed block.

[0104] For brevity, Figure 6 Steps 110 to 150 are not shown in FIG. 1, and steps 160, 170, 180 and 190 are similar to the corresponding steps of FIG. 2, which will not be described here. Figure 1 Steps 110 to 150 are not shown in FIG. 1, and steps 160, 170, 180 and 190 are similar to the corresponding steps of FIG. 2, which will not be described here.

[0105] Step 172, the forward propagation and the backward propagation of the replaced neural network are iteratively performed on the preset data set. During the iterative performance, the updated value of the K j weight factors generated by the previous backward propagation is assigned to the K j weight factors used in the subsequent forward propagation; and after each iteration is completed, it is determined whether the number of iterations reaches a preset number M. If yes, one of steps 174, 176 or 178 is selected; if not, the forward propagation and the backward propagation of the replaced neural network are iteratively performed again from step 160. The preset number M is an integer greater than 1, for example, it can be between 3 and 20. More specifically, in some examples, M is 3, 5 or 10.

[0106] After the forward propagation and the backward propagation of the replaced neural network are iteratively performed M times, the updated values of the weight factors corresponding to each operation branch in the replaced neural network tend to be stable, and then the updated values of the weight factors can be used to determine whether the corresponding compression rate needs to be updated and how to update it. Specifically, in some embodiments, for the jth to-be-compressed block, whether the K j weight factors need to be updated or how to update them can be determined based on the comparison between the updated values of the K j weight factors and a preset weight factor threshold. The preset weight factor threshold can be set based on the number K j of weight factors of the to-be-compressed block, which is greater than 1 / K j and less than 1. In the following, K j compression rates have initial values Rj , R j + Δ, R j + 2 Δ, …, R j + (K j - 1) Δ as an example to illustrate steps 174 to 178. It can be understood that when the K j compression rates are other values of non-arithmetic progression, the K j compression rates can also be sorted from small to large and then updated using the method of steps 174 to 178.

[0107] Step 174, when the update value of the weight factor corresponding to the first operation branch in the K j operation branches is greater than or equal to the preset weight factor threshold, the update values of the K j compression rates are respectively set to be less than the initial values. If the update value of the weight factor corresponding to the first operation branch (i.e., the operation branch corresponding to the minimum compression rate R j ) in the K j operation branches is greater than or equal to the preset weight factor threshold, it indicates that the jth to-be-compressed block tends to use the minimum value in all K j compression rates for compression, and therefore, it is necessary to provide a smaller compression rate for the jth to-be-compressed block to determine whether there is a more optimal compression rate smaller than R j for the jth to-be-compressed block. Specifically, the initial values of all K j compression rates can be subtracted by a predetermined amount (for example, Δ), and if the difference between the minimum compression rate R j and the predetermined amount is greater than 0, the difference between the K j initial values and the predetermined amount is taken as the update values of the K j compression rates. In other cases, if the difference between the minimum compression rate R j and the predetermined amount is less than or equal to 0, it indicates that the current K j compression rates have taken the minimum compression rate value within the allowed range, and the update values of the K j compression rates can be respectively set to be equal to the initial values, or in other words, the initial values remain unchanged. After step 174, return to step 130 to use the update values of the K j compression rates to compress the J to-be-compressed blocks to generate updated K j operation branches, and in step 140, the K j weight factors are reset to the initial values and are updated in the subsequent retraining process, which will not be described here.

[0108] Step 176, when the update value of the weight factor corresponding to the K j th operation branch in the K j operation branches is greater than or equal to the preset weight factor threshold, the update values of the Kj The updated values ​​for each compression ratio are greater than the initial values. If K j The Kth branch of the arithmetic operations j Each computational branch (i.e., maximum compression ratio R) j +(K j If the updated value of the weight factor of the operation branch corresponding to -1)Δ is greater than or equal to the preset weight factor threshold, it indicates that the j-th block to be compressed tends to use all K j The maximum compression ratio among the compression ratios is used for compression. Therefore, it is necessary to provide a larger compression ratio for the j-th block to determine whether there is a better compression ratio greater than R for the j-th block. j +(K j -1)Δ compression ratio. Specifically, this can be applied to all K. j An initial value for the compression ratio is added to a predetermined amount (e.g., Δ), if the maximum compression ratio R... j +(K j -1) If the sum of Δ and the predetermined amount is less than 1, then K will be... j The sum of the initial value and the predetermined amount is K. j The updated value for the compression ratio. In other cases, if the maximum compression ratio R... j +(K j If the sum of -1)Δ and the predetermined amount is greater than or equal to 1, then it indicates that the current K j The compression ratio is already the maximum allowed value; K can be set. j The updated compression ratio values ​​are equal to the initial values, or in other words, the initial values ​​remain unchanged. After step 176, return to step 130 and execute the subsequent steps.

[0109] Step 178, when K j The first operational branch in the Kth operational branch and the Kth operational branch j When the updated values ​​of the weight factors corresponding to each computational branch are all less than the preset weight factor threshold, K is set. j The updated values ​​for each compression ratio are equal to the initial values. If K j The first operational branch in the Kth operational branch and the Kth operational branch j If the updated values ​​of the weight factors of all computational branches are not greater than or equal to the preset weight factor threshold, then it means that the j-th block to be compressed has a positive impact on all K-level weight factors. j There is no clear bias between the minimum and maximum values ​​of the compression ratios. In this case, the compression ratio suitable for the j-th block to be compressed is between the minimum and maximum values, and K can be set accordingly. j The updated compression ratio values ​​are equal to the initial values, or in other words, the initial values ​​remain unchanged. After step 178, continue with steps 180 and 190, which can be found in the following reference. Figure 1 The details of the process will not be repeated here.

[0110] In one specific embodiment, with Figure 5 (b) For example, K j =3, the initial values ​​of the compression ratios of the three computation branches are R. j R j +Δ and R j +2Δ. Assuming a preset weight factor threshold of 0.5, after M iterations of forward and backward propagation on the replaced neural network, if the updated value of the weight factor of the first operational branch is greater than or equal to 0.5, it indicates that the weight factor of the first operational branch is significantly greater than that of the second and third operational branches. A predetermined amount Δ can be subtracted from each of the three initial compression ratio values ​​to generate a new value R. j -Δ、R j and R j +Δ; if R j If -Δ is greater than 0, then these three values ​​R j -Δ、R j and R j +Δ is set as the updated value of the compression ratio. If the updated value of the weight factor of the third computational branch is greater than or equal to 0.5, it indicates that the weight factor of the third computational branch is significantly greater than that of the first and second computational branches. A predetermined amount Δ can be added to each of the three initial compression ratio values ​​to generate a new value R. j +Δ、R j +2Δ and R j +3Δ; if R j If +3Δ is less than 1, then these three values ​​R j +Δ、R j +2Δ and R j +3Δ is set as the update value for the compression ratio. If the update value of the weight factor of the second computation branch is greater than or equal to 0.5, or if the update values ​​of the weight factors of all three computation branches are less than 0.5, then the update values ​​of the three compression ratios are set to be equal to the initial values, i.e., the initial values ​​are kept unchanged.

[0111] refer to Figure 7 The figure shows a comparison of the results of compressing a ResNet34-based voiceprint recognition neural network using the method for compressing neural networks described in this application and a conventional compression method based on sensitivity analysis. Figure 7 The horizontal axis represents the pruning rate (pruning rate = 1 - compression rate), and the vertical axis represents the equal error rate (EER), which is the most common performance metric in voiceprint recognition. A lower EER indicates better neural network performance. Figure 7 As can be seen, under different pruning rates, the performance of the compressed neural network method in this application is better than that of the traditional compression method based on sensitivity analysis, and the performance gap is greater as the pruning rate is higher.

[0112] To study the compression effect of the neural network model under different pruning rates, the present application also simulates the different pruning rates and the corresponding compression rate scanning step Δ listed in Table 1. Referring to Figure 8 (a) to 8(e), wherein Figure 8 (a), 8(b), 8(c), 8(d) and 8(e) respectively show the distribution of the pruning rate of each operation layer after the neural network model is compressed using the method of the present application when the pruning rate is 30%, 52%, 70%, 82% and 85%, respectively; Figure 8 (f) shows the proportion of the parameter value of each operation layer of the neural network model before compression to the total parameter value of the model. It can be seen that as the pruning rate increases, the pruning rate of each layer of the compressed neural network model also generally increases accordingly.

[0113] Table 1

[0114]

[0115] Referring to Table 2, which shows the equal error rate values under different pruning rates after the method of the present application is used to compress the voiceprint recognition model; Figure 9 The equal error rate values of Table 2 are graphically represented, where the horizontal axis is the size of the compressed model and the vertical axis is the equal error rate value. Specifically, the voxceleb dataset, voxceleb2 dataset or open source speech samples intercepted from the Internet are selected, which are preprocessed to 5s, 8s or 10s in length, or remain the original length; the original model and the compressed model with pruning rates of 30%, 52.5%, 70.16%, 81.84% and 84.98% are tested on the above speech samples, and the corresponding equal error rate values can be obtained. From Table 2 and Figure 9 It can be seen that even for the compressed model with a pruning rate of 84.98%, the performance loss (i.e., the difference in equal error rate) compared with the original model is within 0.45%. It can be seen that the method for compressing the neural network of the present application can have less impact on the performance of the compressed neural network model under a higher pruning rate.

[0116] Table 2

[0117]

[0118] The present application also provides a device 1000 for compressing a neural network. As Figure 10As shown, the apparatus 1000 for compressing neural network comprises an obtaining unit 1010, a dividing unit 1020, a compressing unit 1030, a weight factor obtaining unit 1040, a replacing unit 1050, a forward propagating unit 1060, a backward propagating unit 1070 and a determining unit 1080. The obtaining unit 1010 is configured to obtain a neural network to be compressed, the neural network to be compressed comprising one or more operation layers to be compressed, each operation layer to be compressed comprising one or more operation channels to be compressed, each operation channel to be compressed comprising one or more operation cores to be compressed, each operation core to be compressed comprising one or more parameter values to be compressed; the dividing unit 1020 is configured to divide the plurality of parameter values to be compressed into J blocks to be compressed, where J is an integer greater than 1; the compressing unit 1030 is configured to compress the jth block to be compressed among the J blocks to be compressed using K j different compression rates to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1; the weight factor obtaining unit 1040 is configured to obtain, for the jth block to be compressed, a set of K j weight factors corresponding to the K j operation branches respectively, the K j weight factors having initial values respectively; the replacing unit 1050 is configured to replace, for the jth block to be compressed, the block to be compressed with the K j operation branches weighted by the set of K j weight factors, to generate a replaced neural network; the forward propagating unit 1060 is configured to perform forward propagation on the replaced neural network on a preset data set, wherein for the jth block to be compressed, the K j operation results of the K j operation branches are weighted and summed as an output of the jth block to be compressed with the K j weight factors; the backward propagating unit 1070 is configured to perform backward propagation on the replaced neural network on the preset data set, wherein for the jth block to be compressed, update values of the K j weight factors are calculated based on a model loss of the replaced neural network relative to the neural network to be compressed; and the determining unit 1080 is configured to determine, for the jth block to be compressed, an operation branch corresponding to a maximum one of the update values of the K j weight factors as a compressed jth block. For details of the apparatus 1000, reference can be made to the foregoing description of the corresponding method, which will not be repeated here. Figures 1 to 9 For details of the apparatus 1000, reference can be made to the foregoing description of the corresponding method, which will not be repeated here.

[0119] In some embodiments, the apparatus for compressing neural network can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components. In addition, the above-described apparatus embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical or other forms. The units described as separate components can be or can not be physically separated, and the components shown as units can be or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0120] In some other embodiments, the apparatus for compressing neural network can also be implemented in the form of software functional units. If the functional units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium and executed by a computer apparatus. Based on this understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a mobile terminal, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.

[0121] The embodiments of the present application also provide an electronic device, which includes a processor and a storage device, the storage device being configured to store a computer program capable of running on the processor. When the computer program is executed by the processor, the processor executes the method for compressing neural network in the above embodiments. In some embodiments, the electronic device can be a mobile terminal, a personal computer, a tablet computer, a server, etc.

[0122] The embodiments of the present application further provide a non-volatile computer readable storage medium, which stores a computer program. The computer program is executed by a processor to perform the method for pruning the convolutional layer in the neural network in the above embodiments. In some embodiments, the non-volatile computer readable storage medium can be a flash memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a CD-ROM, or any other form of non-volatile computer readable storage medium known in the art.

[0123] Those skilled in the art can understand and implement other changes to the disclosed embodiments by studying the specification, the disclosed content, and the drawings and the attached claims. In the claims, the word "comprising" does not exclude other elements and steps, and the word "a" or "one" does not exclude a plurality. In the practical application of the present application, one part can perform the functions of multiple technical features referred to in the claims. Any reference signs in the claims should not be understood as limiting the scope.

Claims

1. A method for compressing neural networks, characterized in that, The method includes: Obtain a neural network to be compressed, wherein the neural network to be compressed includes one or more computational layers to be compressed, each computational layer to be compressed includes one or more computational channels to be compressed, each computational channel to be compressed includes one or more computational kernels to be compressed, and each computational kernel to be compressed includes one or more parameter values ​​to be compressed. The compressed neural network's parameter values ​​are divided into units based on computational layers, computational channels, or computational kernels. J There are blocks to be compressed, among which J It is an integer greater than 1; For the above J The first block to be compressed j Each block to be compressed is used K j Compression is performed at different compression ratios to generate K j There are 1 computational branch, among which j and K j All are integers, and 1 ≤ j ≤ J , K j ≥1; For the first j Each block to be compressed is obtained from... K j A set of operation branches corresponding to each set K j Each weighting factor, the K j Each weighting factor has its own initial value; For the first j One block to be compressed, using the group K j The weighted sum of each weight factor K j Each computational branch replaces the block to be compressed, generating a replaced neural network; The replaced neural network is forward-propagated on a preset dataset, wherein, for the first... j One block to be compressed, the... K j Each computational branch K j The calculation result and the above K j The weighted sum of the weighted factors is taken as the first weighted factor. j The output of each block to be compressed; Backpropagation is performed on the replaced neural network on the preset dataset, wherein, for the first... j The method calculates the model loss of the replaced neural network relative to the original neural network based on the given blocks to be compressed. K j The updated values ​​of each weight factor, wherein the model loss includes a loss function term and a performance metric term, the loss function term being related to the application type of the neural network to be compressed, and the performance metric term being related to the hardware metrics of the hardware platform on which the neural network to be compressed is deployed; and For the first j One block to be compressed, the... K j The computational branch corresponding to the largest updated value of each weight factor is determined as the compressed i-th weight factor. j Each block.

2. The method according to claim 1, characterized in that, The compressed neural network's parameter values ​​are divided into units based on computational layers, computational channels, or computational kernels. J Each block to be compressed includes dividing the parameter values ​​to be compressed in the same computational layer into a single block.

3. The method according to claim 2, characterized in that, For the above J The first block to be compressed j One block to be compressed K j Compression at different compression ratios includes: Calculate the first j Each of the processing channels of the block to be compressed L 1-norm; Based on the first j One block to be compressed K j Each compression ratio is determined separately. L The smallest 1 norm P 1. P 2, ... P Kj One processing channel, of which P 1. P 2, ... P Kj For each of the following: greater than or equal to 1 and less than 1 J integers; and The first j The blocks to be compressed P 1. P 2, ... P Kj After setting all the parameters to be compressed in each processing channel to zero, the following are generated: K j Each operation branch.

4. The method according to claim 1, characterized in that, The compressed neural network's parameter values ​​are divided into units based on computational layers, computational channels, or computational kernels. J Each block to be compressed includes dividing the parameter values ​​to be compressed in the same processing channel into a single block.

5. The method according to claim 4, characterized in that, For the above J The first block to be compressed j One block to be compressed K j Compression at different compression ratios includes: Calculate the first j Each of the processing cores of each block to be compressed L 1-norm; Based on the first j One block to be compressed K j Each compression ratio is determined separately. L The smallest 1 norm P 1. P 2, ... P Kj One processing core, of which P 1. P 2, ... P Kj For each of the following: greater than or equal to 1 and less than 1 J integers; and The first j The blocks to be compressed P 1. P 2, ... P Kj After setting all the parameters to be compressed in each processing core to zero, the following are generated: K j Each operation branch.

6. The method according to claim 1, characterized in that, The compressed neural network's parameter values ​​are divided into units based on computational layers, computational channels, or computational kernels. J Each block to be compressed includes dividing the parameter values ​​to be compressed in the same processing core into a single block.

7. The method according to claim 6, characterized in that, For the above J The first block to be compressed j One block to be compressed K j Compression at different compression ratios includes: Calculate the first j The absolute values ​​of all parameters to be compressed in each block to be compressed; Based on the first j One block to be compressed K j Each compression ratio is determined by the smallest absolute value. P 1. P 2, ... P Kj There are several parameter values ​​to be compressed, among which P 1. P 2, ... P Kj For each of the following: greater than or equal to 1 and less than 1 J integers; and The first j The blocks to be compressed P 1. P 2, ... P Kj After setting each parameter value to be compressed to zero, the following are generated respectively: K j Each operation branch.

8. The method according to claim 1, characterized in that, The first j One block to be compressed K j Each compression ratio has a separate initial value. R j , R j + Δ , R j + 2Δ ... R j + (K j -1)Δ , where 0 < R j <1, 0< Δ <1, and 0<( R j + (K j -1)Δ )<1.

9. The method according to claim 8, characterized in that, Also includes: The forward and backward propagation of the replaced neural network are performed iteratively multiple times on the preset dataset. During the iterative execution, the results generated by the previous backward propagation are... K j The updated values ​​of each weight factor are assigned to the values ​​in the next forward propagation. K j Each weighting factor is used.

10. The method according to claim 9, characterized in that, Also includes: Iterative forward and backward propagation of the replaced neural network is performed on the preset dataset. M After that, for the first j A block to be compressed, based on the K j The updated values ​​of the weight factors are obtained. K j The updated value of the compression ratio, where M It is an integer greater than 1.

11. The method according to claim 10, characterized in that, Based on the above K j The updated values ​​of the weight factors are obtained. K j The updated values ​​for each compression ratio include: When the K j When the updated value of the weight factor corresponding to the first computational branch in a computational branch is greater than or equal to the preset weight factor threshold, the setting is... K j The updated values ​​for each compression ratio are less than the initial values; When the K j The first of the arithmetic branches K j When the updated value of the weight factor corresponding to each computational branch is greater than or equal to the preset weight factor threshold, the preset weight factor threshold is set. K j The updated values ​​for each compression ratio are greater than the initial values; and When the K j The first operational branch in the nth operational branch and the... K j When the updated values ​​of the weight factors corresponding to each computational branch are all less than the preset weight factor threshold, the preset weight factor threshold is set. K j The updated values ​​for each compression ratio are equal to the initial values.

12. The method according to claim 9, characterized in that, Also includes: During the iterative execution process, based on the K j The computational branch corresponding to the largest updated value of each weight factor calculates the model size of the replaced neural network; Compare the difference in model size of the replaced neural network obtained during the two iterations. The iterative execution process stops when the difference is within a preset numerical range.

13. The method according to claim 1, characterized in that, The application types of the neural network to be compressed include: classification, localization, detection, or segmentation.

14. The method according to claim 1, characterized in that, The hardware specifications of the hardware platform on which the compressed neural network is to be deployed include: storage space, number of floating-point operations, latency, or energy consumption.

15. The method according to claim 1, characterized in that, The computational layer to be compressed includes a convolutional layer, an activation layer, a batch normalization layer, a pooling layer, or a fully connected layer.

16. The method according to claim 1, characterized in that, Also includes: In the above K j Each computational branch K j The calculation result and the above K j Before performing a weighted summation on the weighted factors, the... K j The values ​​of each weight factor are normalized.

17. An apparatus for compressing neural networks, characterized in that, The device includes: An acquisition unit is used to acquire a neural network to be compressed, wherein the neural network to be compressed includes one or more computational layers to be compressed, each computational layer to be compressed includes one or more computational channels to be compressed, each computational channel to be compressed includes one or more computational kernels to be compressed, and each computational kernel to be compressed includes one or more parameter values ​​to be compressed. A partitioning unit is used to divide all the parameter values ​​to be compressed in the neural network into units of computational layers, computational channels, or computational kernels. J There are blocks to be compressed, among which J It is an integer greater than 1; Compression unit, used for the J The first block to be compressed j Each block to be compressed is used K j Compression is performed at different compression ratios to generate K j There are 1 computational branch, among which j and K j All are integers, and 1 ≤ j ≤ J , K j ≥1; The weight factor acquisition unit is used for the first weight factor acquisition unit. j Each block to be compressed is obtained from... K j A set of operation branches corresponding to each set K j Each weighting factor, the K j Each weighting factor has its own initial value; Replacement unit, used for the first j One block to be compressed, using the group K j The weighted sum of each weight factor K j Each computational branch replaces the block to be compressed, generating a replaced neural network; Forward propagation unit, used to perform forward propagation on the replaced neural network on a preset dataset, wherein, for the first... j One block to be compressed, the... K j Each computational branch K j The calculation result and the above K j The weighted sum of the weighted factors is taken as the first weighted factor. j The output of each block to be compressed; A backpropagation unit is configured to perform backpropagation on the replaced neural network on the preset dataset, wherein, for the first... j The method calculates the model loss of the replaced neural network relative to the original neural network based on the given blocks to be compressed. K j The updated values ​​of each weight factor, wherein the model loss includes a loss function term and a performance metric term, the loss function term being related to the application type of the neural network to be compressed, and the performance metric term being related to the hardware metrics of the hardware platform on which the neural network to be compressed is deployed; and Determining unit, used for determining the first j One block to be compressed, the... K j The computational branch corresponding to the largest updated value of each weight factor is determined as the compressed i-th weight factor. j Each block.

18. An electronic device, characterized in that, include: processor; and A storage device for storing computer programs capable of running on the processor; When the computer program is executed by the processor, the processor performs the method for compressing a neural network as described in any one of claims 1-16.

19. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for compressing a neural network as described in any one of claims 1-16.

Images