Method and apparatus for compressing neural networks

Abstract

The application discloses a method and device for compressing a neural network, wherein the method comprises: obtaining a neural network to be compressed, which comprises J operation layers to be compressed; compressing the J operation layers to be compressed using K j different compression rates respectively to generate K j operation branches; obtaining K j weight factors corresponding to the K j operation branches respectively; replacing the operation layers to be compressed with the K j operation branches to generate a replaced neural network; performing forward propagation on the replaced neural network, and performing weighted summation of K j operation results of the K j operation branches and the K j weight factors as an output of the jth operation layer to be compressed; performing backward propagation on the replaced neural network, calculating update values of the K j weight factors based on a model loss of the replaced neural network relative to the neural network to be compressed; and determining, for the jth operation layer to be compressed, an operation branch corresponding to a maximum one of the update values of the K j weight factors as a compressed jth operation layer.

Description

TECHNICAL FIELD

[0001] The present application relates to neural network technology, and more particularly, to a method and apparatus for compressing a neural network. BACKGROUND

[0002] In recent years, neural network technology has been applied to many technical fields, such as image recognition, speech recognition, autonomous driving, medical imaging, and the like. For example, a convolutional neural network (CNN) as a representative network structure and algorithm in neural network technology has achieved great success in image processing technology. However, due to the deep number of layers, the large number of parameters, and the large amount of computation of the neural network, the deployment of the neural network on a hardware platform with limited resources (for example, an embedded terminal) is limited. SUMMARY

[0003] It is an object of the present application to provide a method for compressing a neural network, which can trade off a lower accuracy loss for a higher compression rate.

[0004] According to some aspects of the present application, a method for compressing a neural network is provided. The method includes: obtaining a neural network to be compressed, the neural network to be compressed including J operation layers to be compressed, where J is an integer greater than 1; compressing the J operation layers to be compressed using K different compression rates for a jth operation layer to be compressed in the J operation layers to be compressed, to generate K operation branches, where j and K are integers, and 1≤j≤J, K≥1; for the jth operation layer to be compressed, obtaining a set of K weight factors corresponding to the K operation branches, respectively, the K weight factors having respective initial values; for the jth operation layer to be compressed, replacing the operation layer to be compressed with the K operation branches weighted by the set of K weight factors, to generate a replaced neural network; performing forward propagation on the replaced neural network on a preset data set, wherein for the jth operation layer to be compressed, K operation results of the K operation branches are combined to obtain an output of the jth operation layer to be compressed; and determining whether the output of the jth operation layer to be compressed meets a preset accuracy requirement. j j j j j j j j j j j j ​​​​​​​​​​​summing the Kjweight factors as the output of the jth to-be-compressed operation layer; performing back propagation on the replaced neural network on a preset data set, wherein for the jth to-be-compressed operation layer, updating values of the Kjweight factors are calculated based on a model loss of the replaced neural network relative to the to-be-compressed neural network; and determining, for the jth to-be-compressed operation layer, an operation branch corresponding to a maximum one of the updating values of the Kjweight factors as the jth compressed operation layer. j j

[0005] According to some aspects of the present disclosure, there are provided apparatuses for compressing a neural network. The apparatuses include an obtaining unit configured to obtain a to-be-compressed neural network, the to-be-compressed neural network including J to-be-compressed operation layers, where J is an integer greater than 1; a compressing unit configured to compress, for a jth to-be-compressed operation layer of the J to-be-compressed operation layers, using Kjdifferent compression rates, to generate Kjoperation branches, where j and Kjare integers, and 1≤j≤J, Kj≥1; a weight factor obtaining unit configured to, for the jth to-be-compressed operation layer, obtain a set of Kjweight factors corresponding to the Kjoperation branches respectively, the Kjweight factors having initial values respectively; a replacing unit configured to, for the jth to-be-compressed operation layer, replace the to-be-compressed operation layer with the Kjoperation branches weighted by the set of Kjweight factors, to generate a replaced neural network; a forward propagating unit configured to perform forward propagation on the replaced neural network on a preset data set, wherein for the jth to-be-compressed operation layer, Kjoperation results of the Kjoperation branches are summed with the Kjweight factors as the output of the jth to-be-compressed operation layer; a back propagating unit configured to perform back propagation on the replaced neural network on the preset data set, wherein for the jth to-be-compressed operation layer, updating values of the Kjweight factors are calculated based on a model loss of the replaced neural network relative to the to-be-compressed neural network; and a determining unit configured to, for the jth to-be-compressed operation layer, determine an operation branch corresponding to a maximum one of the updating values of the Kjweight factors as the jth compressed operation layer. j j j j j j j j j j j j j j

[0006] ​​​​​​​​​​​​​​​​According to still another aspect of the present application, there is provided an electronic device comprising: a processor; and a storage device for storing a computer program capable of running on the processor; wherein the computer program, when executed by the processor, causes the processor to perform the method for compressing a neural network described above.

[0007] According to still another aspect of the present application, there is provided a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method for compressing a neural network described above.

[0008] The above is a summary of the present application, which may have simplified, generalized or omitted details, and therefore those skilled in the art should recognize that this section is merely illustrative and is not intended to limit the scope of the present application in any way. Neither is this summary section intended to identify key or essential features of the claimed subject matter nor is it used to determine or delimit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features of the present application will become more fully understood from the following detailed description and the accompanying drawings, wherein:

[0010] Figure 1 (a) and Figure 1 (b) shows a schematic diagram of a unified quantization scheme and a hybrid quantization scheme;

[0011] Figure 2 A flow chart of a method for compressing a neural network according to an embodiment of the present application is shown;

[0012] Figure 3 An exemplary neural network structure diagram is shown;

[0013] Figure 4 (a) and Figure 4 (b) shows a logic block diagram of forward propagation of an operation layer of a neural network according to an embodiment of the present application;

[0014] Figure 5 (a), Figure 5 (b) and Figure 5 (c) shows a relationship diagram between a hyperparameter γ, a network model accuracy and a model size in a method for compressing a neural network according to an embodiment of the present application;

[0015] Figure 6A schematic diagram showing the compression results of a neural network model according to an embodiment of the present application is shown; and

[0016] Figure 7 A block diagram of an apparatus for compressing a neural network according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols generally indicate similar components, unless the context indicates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be employed, and other changes may be made, without departing from the spirit or scope of the subject matter of the present application. It is understood that various configurations, substitutions, combinations, and designs of the various aspects of the present application generally described herein and illustrated in the accompanying drawings may be made, all of which are expressly intended to form a part of the present application.

[0018] In order to facilitate the deployment of neural networks on hardware, existing technologies usually compress neural networks, such as pruning and quantization. Quantization refers to using low-bit-width binary numbers instead of high-bit-width binary numbers to represent the parameter values ​​in the neural network, thereby compressing the neural network. In a typical neural network quantization method, the parameter values ​​used in each operation layer in the neural network are quantized to a uniform bit width. For example, Figure 1 As shown in (a), the parameter values ​​of the convolutional layers (conv1, conv2, etc.) and the fully connected layers (fc) in the neural network 100 before quantization are all represented by 32-bit floating point numbers (fp 32); after quantization, these parameter values ​​can be uniformly represented by 8-bit fixed-point numbers. In other examples, more or fewer bits (or higher or lower precision) can also be used to quantize the parameter values ​​used in each operation layer of the neural network. It should be noted that the accuracy loss caused by quantizing the parameter values ​​of the neural network is usually related to the quantization accuracy. For example, if the parameter values ​​used by the neural network are uniformly quantized from 32-bit floating point numbers to 8-bit fixed-point numbers or 4-bit fixed-point numbers, the accuracy loss of quantization to 4 bits will be significantly higher than the accuracy loss of quantization to 8 bits.

[0019] The inventors of this application have discovered that the existing technology usually performs quantization processing with the same precision on each operation layer in the neural network, but the sensitivity of different operation layers to quantization precision is different. For example, if some operation layers are quantized to 8 bits, it will cause a significant loss in the accuracy of the neural network, while the accuracy loss caused by other operation layers even if they are quantized to 4 bits is within an acceptable range. Based on this, this application proposes a hybrid quantization scheme that uses different quantization precisions for different operation layers. For example, Figure 1(b) As shown, the convolutional layers and fully connected layers in the neural network 100 before quantization all use 32-bit floating-point numbers to represent parameter values, after quantization, the second convolutional layer conv2 which is sensitive to quantization uses 8-bit fixed-point numbers with higher precision to represent parameter values; while the fully connected layer fc which is not sensitive to quantization uses 4-bit fixed-point numbers with lower precision to represent parameter values; the first convolutional layer conv1 which is moderately sensitive to quantization uses 6-bit fixed-point numbers to represent parameter values. This way of adaptively adjusting quantization precision based on quantization sensitivity can achieve higher compression rate at the cost of smaller accuracy loss. Further, under the premise of considering the compression rate and accuracy of the neural network, the inventors creatively solve the problem of how to reasonably allocate different quantization precisions among multiple operation layers. For example, for a neural network with 50 operation layers, if there are 4 different quantization precisions for each layer, there are 4 50 different quantization schemes for the neural network, and it is obviously not feasible to find the best quantization scheme through brute-force search due to excessive computation.

[0020] Based on the above research and ideas, the present application provides a method for compressing a neural network. In this method, after obtaining a neural network to be compressed, multiple different compression rates are used to compress multiple operation layers to be compressed in the neural network, to generate one or more operation branches corresponding to each operation layer to be compressed; and for each operation layer to be compressed, one or more weight factors corresponding to one or more operation branches are generated; then the neural network is retrained on a preset data set, the retraining including forward propagation and backward propagation, wherein for each operation layer to be compressed, the one or more operation branches are used to replace the operation layer to be compressed in the forward propagation process, and the weighted sum of one or more operation results of the one or more operation branches and one or more weight factors is taken as the output of the operation layer to be compressed; in the backward propagation process, the update values of the one or more weight factors are calculated based on the model loss of the neural network; then for each operation layer to be compressed, the operation branch corresponding to the largest weight factor in the update values of the one or more weight factors is determined as the compressed operation layer, thereby forming a compressed neural network. This method introduces weight factors, which can be used to retrain the neural network on a small sample of a preset data set, and update the weight factors based on the model loss, and select the operation branch corresponding to the largest weight factor in each operation layer as the compressed operation layer, thereby selecting an appropriate compression rate (or quantization precision) for multiple operation layers that can take into account the accuracy; compared with the aforementioned brute-force search method, the computational complexity is greatly reduced.

[0021] The method for compressing a neural network of the present application will be described in detail below with reference to the accompanying drawings. Figure 2A flowchart of a method 200 for compressing a neural network is shown, according to some embodiments of the present application, including steps 210-280 as follows.

[0022] At step 210, a neural network to be compressed is obtained, which includes J operation layers to be compressed, where J is an integer greater than 1.

[0023] The neural network to be compressed can be a deep neural network (DNN) obtained after training using a training sample dataset, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), a restricted Boltzmann machine (RBM), a fully connected network (FCN), a deep convolutional network (DCN), a long short-term memory (LSTM) network, or a gated recurrent unit (GRU), etc.

[0024] Figure 3 An exemplary neural network structure diagram to be compressed is shown, which includes a plurality of operation layers 320-1, 320-2, …, 320-L capable of sequentially processing input data 310 received by the neural network to output output data 330. In some embodiments, the neural network to be compressed can be a convolutional neural network, such as LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, or other types of convolutional neural networks, and the plurality of operation layers can include convolutional layers (Conv), batch normalization layers (BN: Batch Normalization), activation layers (e.g., ReLU layers), pooling layers (Pooling), or fully connected layers, etc. Different operation layers can include different types or numbers of parameter values for participating in the operation of the neural network to be compressed, for example, a convolutional layer can include a plurality of filters, each filter can include a plurality of convolutional kernels, and each convolutional kernel can include a plurality of weight values arranged in a row and column (e.g., a 3x3, 5x5, or 7x7 size weight value matrix), which participate in the operation of the neural network to be compressed as parameter values of the convolutional layer. For image processing type convolutional neural networks, the input data 310 can be an image, which can be represented in the form of a pixel matrix and can include a plurality of different color channels (e.g., R, G, B color channels). The operation layer 320-1 can be a convolutional layer including one or more filters to perform convolution operations on the input data 310 to generate a plurality of feature maps Figure 1 ; then, the feature maps Figure 1 The input data continues to be processed as the input data of the next operation layer 320-2 (e.g., a batch normalization layer, an activation layer, a pooling layer, or a fully connected layer) to generate a plurality of feature Figure 2, and so on, until the output data 330 is generated after processing by the operation layer 320-L, where L can be an integer greater than 1. Depending on the application type of the convolutional neural network, the output data 330 will also be different. For example, for an image classification network, the output data 330 can be the classification of targets in the input image into multiple different classification types; for an image detection network, the output data 330 can be the detection result of the target in the input image. In other embodiments, the input data 310 and the output data 330 can also be sound data, text data, etc., depending on the specific application scenario of the convolutional neural network. It should be noted that although the above embodiment is described using a convolutional neural network as an example, those skilled in the art can understand that the method of compressing a neural network in the present application is applicable to any neural network that includes an operation layer to be compressed.

[0025] In some embodiments, all computation layers in the neural network to be compressed are computation layers to be compressed, that is, the number of computation layers to be compressed J is equal to the total number of computation layers (for example, Figure 3 , J=L). In some embodiments, the neural network to be compressed may also include operation layers that do not need to be compressed. In other words, the number of operation layers to be compressed J may be less than the total number of operation layers (for example, Figure 3 , J<L). Whether to compress all computing layers in the neural network to be compressed or which computing layers to be compressed can be changed based on the characteristics of the neural network to be compressed or specific application requirements.

[0026] Step 220: For the jth layer to be compressed among the J layers to be compressed, use K j Different compression rates are used to compress to generate K j operation branches, where j and K j are all integers, and 1≤j≤J, K j ≥1.

[0027] In order to improve the accuracy of the calculation, the parameter values ​​of the to-be-compressed operation layer of the to-be-compressed neural network obtained in step 210 are usually represented by higher-precision numerical values, such as high-bit-width floating-point numbers. In some examples, 32-bit floating-point numbers can be used to represent the parameter values ​​of the to-be-compressed operation layer. Taking the to-be-compressed convolution layer with a 5×5 convolution kernel as an example, the 25 weight values ​​(i.e., parameter values) in each convolution kernel are all represented by 32-bit floating-point numbers. Therefore, an effective method to compress the to-be-compressed neural network is to replace the parameter values ​​in each to-be-compressed operation layer in the neural network from high-bit-width numerical values ​​to low-bit-width numerical representations.

[0028] In some embodiments, the same number of compression rates are used to compress each of the J to-be-compressed operation layers in the to-be-compressed neural network, that is, the number of compression rates K1, K2, …, KJ used by the J to-be-compressed operation layers are equal. J In some embodiments, the number of compression rates K1, K2, …, KJ used by the J to-be-compressed operation layers can be mutually unequal or partially equal. The number of compression rates used by each to-be-compressed operation layer can be set based on the sensitivity of the to-be-compressed operation layer to compression rates or the target compression rate of the neural network, etc. J In some embodiments, the number of compression rates K1, K2, …, KJ used by the J to-be-compressed operation layers can be mutually unequal or partially equal. The number of compression rates used by each to-be-compressed operation layer can be set based on the sensitivity of the to-be-compressed operation layer to compression rates or the target compression rate of the neural network, etc.

[0029] In some embodiments, the parameter values of the jth to-be-compressed operation layer of the to-be-compressed neural network are represented using N 0j bit binary numbers, and the to-be-compressed operation layer is compressed using K j different compression rates to generate K j operation branches, where the parameter values of the K j operation branches are represented using bit binary numbers, where K j is an integer greater than or equal to 1, is an integer greater than or equal to 1, and is less than or equal to N 0j . The different operation branches corresponding to each to-be-compressed operation layer can have the same size and structure as the to-be-compressed operation layer, but each parameter value has a different precision than the corresponding parameter value in the to-be-compressed operation layer. It should be noted that the number of operation branches K j may also be determined based on the hardware platform on which the to-be-compressed neural network is to be deployed; if the hardware platform supports operations of K j different bit binary numbers, at most K j operation branches are generated for each to-be-compressed operation layer.

[0030] In one example, the parameter values of the jth to-be-compressed operation layer of the to-be-compressed neural network are represented using 32-bit floating point numbers (fp 32), and 4 (i.e., K j= 4) four operation branches generated after compressing the to-be-compressed operation layer at different compression rates, parameter values of which are represented by half-precision 16-bit floating-point numbers (fp 16), 8-bit fixed-point numbers (int 8), 6-bit fixed-point numbers (int 6), and 4-bit fixed-point numbers (int 4), respectively. After compression, the data amount of the parameter values of each operation branch is reduced to 50%, 25%, 18.75%, and 12.5% of the original data amount, respectively. It should be noted that the bit width of the above parameter values and the number of operation branches are only examples, and in other examples, the parameter values of the to-be-compressed operation layer and the operation branches formed after compression can be represented by numbers with higher or lower bit width, such as eight-precision floating-point numbers (fp 256), four-precision floating-point numbers (fp 128), double-precision floating-point numbers (fp 64), 2-bit fixed-point numbers (int 2), 1-bit fixed-point numbers (int 1), and the number of operation branches can also be greater than or less than 4.

[0031] In some embodiments, the compression of each parameter value of the to-be-compressed operation layer is performed using the following formula (1):

[0032] r = S (q - Z) (1)

[0033] wherein r represents a parameter value of the to-be-compressed operation layer, q represents a compressed parameter value, constant S represents a compression scale, and constant Z represents a zero point. For example, if the minimum parameter value in all parameter values of a certain to-be-compressed operation layer is a, the maximum parameter value is b, and the compressed parameter value is represented by an 8-bit fixed-point number with a range of [-127, 128], then Z = (b - a) / 2, and S = (b - a) / 256. That is, for a specific to-be-compressed operation layer and a specific compression precision, constants S and Z are determined. For example, after the operation of formula (1), the parameter value r represented by a 32-bit floating-point number in the to-be-compressed operation layer can be mapped to the parameter value q represented by an 8-bit fixed-point number in the range of [-127, 128].

[0034] In step 230, for the jth to-be-compressed operation layer, a set of K j weight factors corresponding to K j operation branches are obtained, wherein the K j weight factors have respective initial values.

[0035] In some embodiments, the initial values of the weight factors can be automatically generated by software. In other embodiments, the initial values of the weight factors can be defined and provided by a user. Since the sensitivity of each operation layer to the compression rate is unknown before retraining of the neural network, in preferred embodiments, the K j weight factors corresponding to the K jThe initial values ​​of the weight factors are usually set equal.

[0036] For the aforementioned example of a 32-bit floating-point parameter value, 16-bit floating-point numbers, 8-bit fixed-point numbers, 6-bit fixed-point numbers, and 4-bit fixed-point numbers can be used to represent the parameter values ​​of the four compressed operation branches, respectively. The corresponding four weight factors can be expressed as β1, β2, β3, and β4, respectively. In one embodiment, the initial values ​​of these four weight factors can all be set to 0. Those skilled in the art will appreciate that in other examples, the initial values ​​of the weight factors can also be set to other equal or unequal values.

[0037] Step 240: for the jth layer to be compressed, use the K j K weighted by weight factors j The operation branch replaces the operation layer to be compressed to generate a replaced neural network.

[0038] refer to Figure 4 (a) and 4(b), the jth compression operation layer of the neural network to be compressed, that is, the convolutional layer conv j For example, assuming that 4 different compression rates are used to compress the jth operation layer to be compressed, 4 operation branches conv are generated. j1 、conv j2 、conv j3 and conv j4 , then use the four weight factors β1, β2, β3 and β4 to weight the four operation branches β1conv j1 , β2conv j2 , β3conv j3 and β4conv j4 Replace the convolutional layer to be compressed conv j After completing the above replacement operation on all J layers to be compressed, a replaced neural network is generated.

[0039] Although the above description is made by taking the convolutional layer as an example of the layer to be compressed, those skilled in the art will appreciate that similar replacement operations can also be implemented for batch normalization layers, activation layers, or fully connected layers.

[0040] Step 250: forward propagation is performed on the replaced neural network on the preset data set, wherein for the jth layer to be compressed, K j K of the operation branches j The operation results and K j The weighted sum of the weight factors is used as the output of the jth compression operation layer.

[0041] The preset dataset can be a dataset different from the training sample dataset used for training the deep neural network. The preset dataset can include a plurality of training examples, each example consisting of an input object and a desired output value. Forward propagation of the neural network on the preset dataset can generate an output value for the input object of each example, and by comparing the output value with the desired output value, the accuracy of the neural network can be evaluated. For example, for an image classification network, the input object is an image, and the output of the forward propagation is to classify the target in the input image into a plurality of different classification types, and by comparing the classification to which the output belongs with the expected classification result, the parameter values of the image classification network can be adjusted or its performance can be evaluated. Similarly, if the neural network is used in a speech recognition system, the input object is a speech sample, and the expected output is the corresponding text. In some examples, the aforementioned preset dataset can be an ImageNet dataset, or a CIFAR10 dataset, etc. It can be understood that those skilled in the art will be able to select a suitable dataset for retraining of the neural network according to the specific application, and examples are not listed here.

[0042] With reference to Figure 4 (a) and Figure 4 (b), the to-be-compressed operation layer conv j is replaced by 4 operation branches conv j1 , conv j2 , conv j3 and conv j4 . Figure 4 (a) shows the normal forward propagation process of the jth to-be-compressed operation layer conv j , Figure 4 (b) shows the forward propagation process after the jth to-be-compressed operation layer conv j is replaced by 4 operation branches conv j1 , conv j2 , conv j3 and conv j4 when K j = 4.

[0043] First, for the to-be-compressed operation layer conv j , before the forward propagation, the input data can be compressed to generate K j compressed input data corresponding to the K j operation branches respectively, and the precision of the K j compressed input data corresponds to the precision of the K j operation branch parameter values respectively. For example, the input data can be quantized to generate K jcompressed input data, and the K j The bit widths of the compressed input data are K j The bit widths of the parameter values ​​of the three operation branches are equal or similar.

[0044] In some embodiments, the compression of input data can be expressed using formula (2):

[0045] q_input jk =quantize k (input j ) (2)

[0046] Among them, input j Indicates the input data of the jth layer to be compressed, the operator quantize k Indicates that it corresponds to K j The quantizer of the kth operation branch in the operation branches (k=1,2,……,K j ), q_input jk Represents the quantized input data of the jth operation layer corresponding to the kth operation branch. j =4, such as Figure 4 As shown in (b), for the jth compression operation layer conv j Input data input j Quantize and generate 4 operation branches conv j1 、conv j2 、conv j3 and conv j4 The corresponding 4 quantized input data q_input j1 ,q_input j2 ,q_input j3 , and q_input j4 , the four quantized input data are represented by 16-bit floating point numbers (fp 16), 8-bit fixed point numbers (int 8), 8-bit fixed point numbers (int 8), and 8-bit fixed point numbers (int 8) respectively. By quantizing the input data, not only can the storage space of the neural network model be saved, but also the calculation process between the quantized input data and the operation branch can be accelerated. For example, the convolution operation between the quantized input data and the operation branch can be changed from the multiplication and addition of floating point numbers to the multiplication and addition of integers. It should be noted that the number of bits of the quantized input data is not limited to the above example, and can also be adjusted based on the calculation accuracy requirements in the specific application. Reference Figure 3 , it can be understood that when the jth compression operation layer conv j For the first layer to be compressed, input data j That isFigure 3 As shown in the input data 310, when the jth layer to be compressed is a layer other than the first layer to be compressed, the input data input j It is the output data of the j-1th operation layer to be compressed.

[0047] Then, the replaced K j Operation branches and K j The quantized input data is operated to generate the corresponding K j The result of the operation of the operation branch. j The result of the operation is then compared with the corresponding K j The weighted operation is performed on the weight factors to generate a weighted sum, which can be used as the output result of the operation layer. The jth operation layer to be compressed is the convolution layer conv j For example, the above calculation and weighting steps can be expressed by formula (3):

[0048]

[0049] Among them, conv jk ○q_input jk Represents the kth convolution branch conv in the jth operation layer jk and the kth quantized input data q_input jk Perform convolution operation, operator quantize k Indicates the quantization operation of the convolution operation result of the kth convolution branch, α jk is the weight factor β jk The normalized representation of represents the kth normalized weight factor in the jth operation layer. In some examples, the weight factor β jk Normalization can be performed using the following softmax function, namely formula (4):

[0050]

[0051] Those skilled in the art will appreciate that the weight factor β jk Normalization can also be performed using other functions according to specific applications. Figure 4 (b) as an example, that is, K j =4, the above formula (3) can be expressed as the following formula (5):

[0052] The specific operation process will not be described in detail.

[0053] It should be noted that in Figure 4In the example, the to-be-compressed operation layer is a convolution layer, but it can be understood that the forward propagation of an activation layer, a fully connected layer, and the like can be performed by using a similar method. In addition, in some embodiments, a plurality of to-be-compressed operation layers of the to-be-compressed neural network can be combined to form a network block, and then the compression and forward propagation can be performed in units of the network block. For example, the operation relationship and data of each to-be-compressed operation layer in the neural network can be referred to, the relationship between adjacent to-be-compressed operation layers is extracted, and then two or more to-be-compressed operation layers are fused to form a network block for processing, so as to reduce unnecessary intermediate outputs and improve operation efficiency. For example, a convolution layer and an activation layer are very common in a convolutional neural network and appear in pairs, and then the convolution layer and the activation layer can be fused (Conv+ReLU) to form a network block, and the network block is taken as a whole to-be-compressed operation layer for participating in operation. In some other examples, a convolution layer and a pooling layer can be fused (Conv+Pooling), a convolution layer and a batch normalization layer and an activation layer can be fused (Conv+BN+ReLU), and the like.

[0054] It should be further noted that, although the process of forward propagation is described above by taking compression of input data as an example, in some other embodiments, the input data can not be compressed, and the original input data can be directly used for calculation with the operation branch.

[0055] In step 260, the replaced neural network is subjected to back propagation on a preset data set, and for the jth to-be-compressed operation layer, K j weight factors are updated based on the model loss of the replaced neural network relative to the to-be-compressed neural network.

[0056] As described above, compression of the parameter values of the to-be-compressed neural network by replacing high-bit-width values with low-bit-width values can save computing resources but cause a loss of accuracy of the to-be-compressed neural network. Therefore, the replaced neural network can be subjected to back propagation to adjust the parameters of the replaced neural network, so as to reduce the loss of accuracy. It should be noted that, when the replaced neural network is subjected to back propagation, only the weight factors of each to-be-compressed operation layer are adjusted, and other parameters (for example, convolution layer weights) of the neural network are not adjusted.

[0057] In some embodiments, the replaced neural network is subjected to back propagation, and the process of updating the K j weight factors of the jth to-be-compressed operation layer can be performed by using the following formula (6):

[0058]

[0059] wherein β jkβ' represents an initial value of the kth weight factor of the jth to-be-compressed operation layer jk β' represents an initial value of the kth weight factor of the jth to-be-compressed operation layer

[0060] In some embodiments, the model loss function Loss includes a loss function term and a performance indicator term. The loss function term is related to the application type of the neural network, such as classification, positioning, detection, or segmentation. The performance indicator term is related to the hardware indicators of the hardware platform to which the neural network is to be deployed, such as storage space, floating point operation times, delay, or energy consumption, etc.

[0061] The model loss function Loss is described below taking the storage space of a compressed classification neural network on a hardware platform as an example. Specifically, the model loss function Loss can be represented by the following formula (7):

[0062] Loss = loss1 * loss2 (7)

[0063] Wherein, loss1 represents the loss function term, and loss2 represents the performance indicator term.

[0064] For a classification neural network, the loss function term loss1 can be further represented by the cross-entropy function shown in formula (8):

[0065]

[0066] Wherein, label is a classification label, usually represented by one-hot code; I represents the total dimension of the classification label, for example, for the ImageNet dataset, the total dimension I of the classification label is 1000; label(i) represents the i-th classification among all I classification labels, if the i-th classification is true, label(i) is 1, otherwise label(i) is 0; correspondingly, pred(i) represents the probability of identifying the i-th classification.

[0067] The performance indicator term loss2 can be further represented by the model size function shown in formula (9):

[0068] loss2 = (model_size / model_ref) γ (9)

[0069] wherein model_size represents the model size of the neural network in training, in units of "bits"; model_ref is a constant related to the neural network model, and is set as the number of bits of the model size of the neural network after quantization using a uniform quantization precision, for example, the model_ref is the total number of parameter values of the neural network to be compressed multiplied by 8 bits after the neural network to be compressed is uniformly quantized using 8-bit fixed-point numbers; and γ is a hyperparameter for balancing the accuracy loss of the neural network model and the model size. When a neural network model with higher accuracy is desired, a smaller value of γ can be set, and when a smaller neural network model is desired, a larger value of γ can be set. Therefore, the value of γ can be selected according to specific requirements. In some examples, the value of γ is 0.05-5. More specifically, in some examples, the value of γ is 0.25-0.5, and the accuracy loss of the neural network model and the model size can be well balanced.

[0070] In some embodiments, the model size of the neural network in training model_size in the above formula (9) can be represented by the following formula (10):

[0071]

[0072] wherein J represents the total number of operation layers to be compressed in the neural network, j represents the number of each operation layer to be compressed, and thus 1≤j≤J; K j represents the total number of operation branches in the jth operation layer to be compressed, k represents the number of the operation branch in the operation layer, and thus 1≤k≤K j ; conv jk represents the convolution layer of the kth operation branch in the jth operation layer to be compressed; size(conv jk ) represents the number of bits of the convolution layer conv jk , which corresponds to Figure 4 In the example of (b), corresponding to the 4 operation branches of the jth layer, the values of size(conv j1 ), size(conv j2 ), size(conv j3 ) and size(conv j4 ) are equal to the number of weight values of the 4 operation branches respectively multiplied by 16, 8, 6 and 4; in addition, α jk is the normalization weight factor corresponding to the kth operation branch in the jth operation layer, which can be obtained by the aforementioned formula (4).

[0073] The above example illustrates the model loss function Loss in terms of the storage space of the compressed classification neural network on a hardware platform. Those skilled in the art can understand that the model loss function Loss can be modified accordingly according to the specific application scenario. In some embodiments, the loss function term loss1 in the model loss function Loss can be adjusted according to the application type of the neural network (e.g., classification, positioning, detection, or segmentation, etc.). Taking image processing as an example, a classification neural network can be used to determine the category to which a given image belongs; a positioning neural network can be used to determine the specific position of a target in a given image; a detection neural network can be used to identify and locate a target in a given image when the category and number of targets are not determined; and a segmentation neural network can be used to segment a given image into multiple components to locate the boundary of a target in the image. Similar to the loss function term loss1 of the classification neural network described above, which is related to the classification label and the probability of each classification identified, when adjusting the loss function term loss1, the positioning neural network needs to consider the position parameters of the target in the image; the detection neural network needs to consider not only the category and number of targets, but also the positions of these targets in the image; and the segmentation neural network needs to consider the position parameters and shape parameters of the target in the image. In some embodiments, according to the specific application type of the neural network, the loss function term loss1 can be selected from one or more of the cross-entropy function, the absolute value loss function, the logarithmic loss function, the square loss function, the exponential loss function, the hinge loss function, the perceptron function, or other suitable loss functions, which are not listed one by one here. Accordingly, the performance indicator term loss2 in the model loss function Loss can be adjusted according to the hardware indicators (e.g., storage space, number of floating point operations, delay, or energy consumption, etc.) of the hardware platform to be deployed. For example, if the indicators related to the number of floating point operations, delay, or energy consumption of the compressed neural network are set for the hardware platform to be deployed, the model_size term and the model_ref term in formula (9) related to the storage space can be replaced by parameters related to the number of floating point operations, delay, or energy consumption, thereby forming the adjusted performance indicator term loss2.

[0074] In some embodiments, after step 260, the operation branch corresponding to the largest one of the obtained updated values of the K j weight factors can be determined as the jth operation layer of the compressed neural network, thereby obtaining the compressed neural network. This can reduce the amount of calculation required for compressing the neural network.

[0075] In some embodiments, the model loss function Loss can be adjusted according to the application type of the neural network (e.g., classification, positioning, detection, or segmentation, etc.). Taking image processing as an example, a classification neural network can be used to determine the category to which a given image belongs; a positioning neural network can be used to determine the specific position of a target in a given image; a detection neural network can be used to identify and locate a target in a given image when the category and number of targets are not determined; and a segmentation neural network can be used to segment a given image into multiple components to locate the boundary of a target in the image. Similar to the loss function term loss1 of the classification neural network described above, which is related to the classification label and the probability of each classification identified, when adjusting the loss function term loss1, the positioning neural network needs to consider the position parameters of the target in the image; the detection neural network needs to consider not only the category and number of targets, but also the positions of these targets in the image; and the segmentation neural network needs to consider the position parameters and shape parameters of the target in the image. In some embodiments, according to the specific application type of the neural network, the loss function term loss1 can be selected from one or more of the cross-entropy function, the absolute value loss function, the logarithmic loss function, the square loss function, the exponential loss function, the hinge loss function, the perceptron function, or other suitable loss functions, which are not listed one by one here. Accordingly, the performance indicator term loss2 in the model loss function Loss can be adjusted according to the hardware indicators (e.g., storage space, number of floating point operations, delay, or energy consumption, etc.) of the hardware platform to be deployed. For example, if the indicators related to the number of floating point operations, delay, or energy consumption of the compressed neural network are set for the hardware platform to be deployed, the model_size term and the model_ref term in formula (9) related to the storage space can be replaced by parameters related to the number of floating point operations, delay, or energy consumption, thereby forming the adjusted performance indicator term loss2. Figure 2In the illustrated embodiment, after step 260, the replaced neural network can also be retrained on the preset data set one or more times to obtain a compressed neural network with better performance.

[0076] Step 270, the forward propagation and the backward propagation of the replaced neural network are iteratively executed multiple times on the preset data set, and in the iterative execution process, the size of the replaced neural network model is calculated based on the operation branch corresponding to the maximum of the updated values of the K j weight factors, and it is determined whether the difference between the sizes of the replaced neural network model calculated in the two iteration execution processes is within a preset numerical range; if yes, step 280 is executed; if no, step 250 is returned to continue the iterative execution of the forward propagation and the backward propagation steps of the replaced neural network.

[0077] The retraining of the replaced neural network on the preset data set includes the forward propagation and the backward propagation, and the purpose is to update the K j weight factors allocated to each compressed operation layer to balance the accuracy loss and the model size of the replaced neural network, and obtain an optimized mixed quantization scheme. In the iterative execution process, for the jth compressed operation layer, the updated values of the K j weight factors generated in the previous backward propagation are assigned to the K j weight factors used in the subsequent forward propagation. After multiple iterations of the forward propagation and the backward propagation of the replaced neural network, if the difference between the size of the replaced neural network model in the subsequent iteration and the size of the replaced neural network model in the previous iteration is within a preset numerical range, it is considered that the training converges, and the training can be ended; if not, the training of the replaced neural network needs to be continued, and the forward propagation and the backward propagation steps are continued to update the weight factors.

[0078] It should be noted that for the first time of executing the forward propagation and the backward propagation, since there is no previous iteration process, the initial size of the compressed neural network model can be used as the size of the replaced neural network model in the previous iteration process to calculate the difference between the sizes of the replaced neural network model calculated in the two iteration execution processes.

[0079] It should also be noted that in some cases, there can be a situation of non-convergent calculation, i.e., the aforementioned difference cannot converge to be less than the preset numerical range after multiple iterations, and therefore, a maximum number of iterations can be set in advance, and when the number of iterations reaches the maximum number of iterations, the iteration loop is forcibly exited, and the retraining of the replaced neural network is ended.

[0080] In some embodiments, the size of the replaced neural network can be calculated using formula (11):

[0081]

[0082] The same symbols in formula (11) as in formula (10) represent the same meanings, model_size' represents the model size of the replaced neural network, and a jk is determined by the following formula (12).

[0083]

[0084] wherein the operator arg max represents taking the maximum value in K j weight factors β jk . That is, when calculating the model size of the replaced neural network, only the size of the convolution layer of the operation branch corresponding to the maximum weight factor in all operation branches in each to-be-compressed operation layer is counted into the model size of the replaced neural network, and other items are discarded. The model size of the replaced neural network can be calculated in each iteration process by the above formula (11) and (12). The difference between the model sizes of the replaced neural network in the previous and subsequent iterations is compared with the preset numerical range set by the user. If the difference is within the preset numerical range, it means that the model size of the replaced neural network has no change or small change before and after iteration, and the training can be ended; if not, the training of the replaced neural network on the preset data set is continued.

[0085] In step 280, for the jth to-be-compressed operation layer, the operation branch corresponding to the maximum of the updated values of K j weight factors is determined as the jth compressed operation layer.

[0086] As described above, when the replaced neural network is retrained on the preset data set and reaches the convergence condition, it means that the accuracy loss and the model size of the replaced neural network have been relatively balanced. For the jth to-be-compressed operation layer, the operation branch corresponding to the maximum of the updated values of K j weight factors can be directly determined as the jth compressed operation layer, so as to obtain the compressed neural network.

[0087] Since the parameters of the compressed neural network are represented by using values with lower bit width than the original neural network, the compressed neural network has lower requirements for storage space and operation amount, and can be more easily deployed to hardware platforms with limited resources.

[0088] Reference is made to Figure 5 (a), Figure 5 (b) and Figure 5 (c), which show different experimental results when the method for compressing a neural network of the present application is applied to the EfficientNet-b0 network model. For example, Figure 5(a) shows the change in the accuracy of the compressed neural network model when different hyperparameter γ values ​​are selected. Figure 5 (b) shows the change in the size of the compressed neural network model when different hyperparameter γ values ​​are selected. Figure 5 (c) shows the relationship between the model size and accuracy of the compressed neural network. The above experimental results can be used to guide the selection of the hyperparameter γ under different accuracy and model size requirements. It can be seen that as the value of the hyperparameter γ increases, the model becomes smaller and smaller, but the accuracy is also getting lower and lower. This is because the increase in the value of the hyperparameter γ means that the model size accounts for a larger proportion in the model loss function Loss. In the extreme case, the loss function term loss1 will be ignored, and the compressed model will converge to the configuration that saves the most hardware. However, in actual applications, it is hoped that the accuracy loss is small while compressing the neural network model, so the hyperparameter γ value should not be set too large. After many simulation studies by the inventors, it is appropriate to set the γ value between 0.25 and 0.5. Figure 6 The figure shows the number of bits used for the parameter values ​​of different operation layers after hybrid quantization using the compressed neural network method of this application when the hyperparameter γ value is set to 0.36. The columns shown with slashes in the figure correspond to the depthwise convolution (DWC) layer, while the other columns represent the operation layers other than DWC. Figure 6 It can be seen that the front-end operation layer in the neural network is mostly quantized with a higher number of bits, and the back-end operation layer is quantized with a lower number of bits. This is because the front-end operation layer has fewer parameters and has a greater impact on network performance. In addition, the DWC layer, which has higher computational complexity and a greater impact on network performance, also generally uses a higher number of bits for quantization than the adjacent operation layer. Table 1 below shows a comparison of the model size and accuracy of the original neural network (FP 32), the unified 8-bit quantization scheme, and the hybrid quantization scheme using the method of compressing the neural network of the present application. It can be seen from Table 1 that the hybrid quantization scheme using the method of the present application can achieve better classification performance (accuracy of 76.228%) when the model compression rate exceeds 86%.

[0089] Table 1

[0090] Model size (KB) Accuracy (%) FP32 20454 (average individual weight value 32bit) 76.840 Uniform 8bit quantization 5114 (average individual weight value 8bit) 69.33 Mixed quantization 2780 (average individual weight value 4.35bit) 76.228

[0091] The embodiment of the present application also provides a device 700 for compressing a neural network. Figure 7As shown, the apparatus 700 for compressing neural network comprises an obtaining unit 710, a compressing unit 720, a weight factor obtaining unit 730, a replacing unit 740, a forward propagating unit 750, a backward propagating unit 760 and a determining unit 770. The obtaining unit 710 is configured to obtain a neural network to be compressed, the neural network to be compressed comprising J operation layers to be compressed, where J is an integer greater than 1; the compressing unit 720 is configured to compress, using K j different compression rates, a jth operation layer to be compressed in the J operation layers to be compressed, to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1; the weight factor obtaining unit 730 is configured to, for the jth operation layer to be compressed, obtain K j weight factors corresponding to the K j operation branches respectively, the K j weight factors having initial values; the replacing unit 740 is configured to, for the jth operation layer to be compressed, replace the operation layer to be compressed with the K j operation branches, to generate a replaced neural network; the forward propagating unit 750 is configured to perform forward propagation on the replaced neural network on a preset data set, where, for the jth operation layer to be compressed, the K j operation results of the K j operation branches are weighted and summed with the K j weight factors as an output of the jth operation layer to be compressed; the backward propagating unit 760 is configured to perform backward propagation on the replaced neural network on the preset data set, where, for the jth operation layer to be compressed, an update value of the K j weight factors is calculated based on a model loss of the replaced neural network relative to the neural network to be compressed; and the determining unit 770 is configured to, for the jth operation layer to be compressed, determine, as a jth operation layer after compression, an operation branch corresponding to a maximum one of the update values of the K j weight factors. For details of the apparatus 700, reference can be made to the foregoing description of the corresponding method, which will not be repeated here. Figures 1 to 6

[0092] ​In some embodiments, the apparatus for compressing neural network can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components. In addition, the above-described apparatus embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical or other forms. The units described as separate components can be or can not be physically separated, and the components shown as units can be or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0093] In some other embodiments, the apparatus for compressing neural network can also be implemented in the form of software functional units. If the functional units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium and executed by a computer apparatus. Based on this understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a mobile terminal, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.

[0094] The embodiments of the present application also provide an electronic device, which includes a processor and a storage device, the storage device being configured to store a computer program capable of running on the processor. When the computer program is executed by the processor, the processor executes the method for compressing neural network in the above embodiments. In some embodiments, the electronic device can be a mobile terminal, a personal computer, a tablet computer, a server, etc.

[0095] The embodiments of the present application further provide a non-volatile computer readable storage medium, which stores a computer program. The computer program is executed by a processor to perform the method for pruning the convolutional layer in the neural network in the above embodiments. In some embodiments, the non-volatile computer readable storage medium can be a flash memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a CD-ROM, or any other form of non-volatile computer readable storage medium known in the art.

[0096] Those skilled in the art can understand and implement other changes to the disclosed embodiments by studying the specification, the disclosed content, and the drawings and the attached claims. In the claims, the word "comprising" does not exclude other elements and steps, and the word "a" or "one" does not exclude a plurality. In the practical application of the present application, one part can perform the functions of multiple technical features referred to in the claims. Any reference signs in the claims should not be understood as limiting the scope.​

Claims

1. A method for compressing a neural network, characterized in that, The method comprises: obtaining a neural network to be compressed, the neural network to be compressed comprising J operation layers to be compressed, wherein J is an integer greater than 1; For a jth to-be-compressed operation layer of the J to-be-compressed operation layers, K j different compression rates are used for compression to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1. For the jth compressed operation layer to be compressed, a set of K j weight factors corresponding to K j operation branches are obtained, the K j weight factors having respective initial values; For the jth compressed operation layer, the K j operational branches weighted by the set of K j weight factors are used to replace the compressed operation layer, generating a replaced neural network. performing forward propagation on the replaced neural network on a preset data set, wherein, for the jth operation layer to be compressed, performing weighted summation of K j operation results of the K j operation branches and the K j weight factors as an output of the jth operation layer to be compressed; performing back propagation on the replaced neural network on the preset data set, wherein, for the jth compressed operation layer, an update value of the K j weight factors is calculated based on a model loss of the replaced neural network relative to the neural network to be compressed, wherein the model loss comprises a loss function item and a performance index item, the loss function item is related to an application type of the neural network to be compressed, the performance index item is related to a hardware index of a hardware platform to which the neural network to be compressed is to be deployed, and the hardware index of the hardware platform to which the neural network to be compressed is to be deployed comprises storage space, floating point operation times, delay or energy consumption; and For the jth compressed operation layer, the operation branch corresponding to the largest one of the K j updated weight factors is determined as the jth compressed operation layer.

2. The method of claim 1, wherein, the application type of the neural network to be compressed comprises classification, positioning, detection or segmentation.

3. The method of claim 1, wherein, The forward propagation and the backward propagation of the replaced neural network are performed on the preset data set for multiple iterations, wherein during the iterative performance, the update value of the K j weight factors generated by the previous backward propagation is assigned to the K j weight factors in the subsequent forward propagation.

4. The method of claim 3, wherein, Further comprising: In the iterative execution process, a model size of the replaced neural network is calculated based on a maximum one of the updated values of the K j weight factors. comparing the difference of the model size of the replaced neural network obtained in the process of the iteration execution before and after two times; when the difference is within a preset numerical range, stopping the iteration execution process.

5. The method of claim 1, wherein, The operation layer to be compressed comprises a convolution layer to be compressed, an activation layer to be compressed, a batch normalization layer to be compressed, a pooling layer to be compressed or a fully connected layer to be compressed.

6. The method of claim 1, wherein, Further comprising: Before the K j operation results of the K j operation branches are weighted and summed with the K j weight factors, the values of the K j weight factors are normalized.

7. The method of claim 1, wherein, Further comprising: For the jth compressed operation layer, before the forward propagation, the input data is compressed to generate K j compressed input data corresponding to the K j operation branches respectively.

8. The method of claim 1, wherein, The parameter value of the jth layer to be compressed is N 0j Bit binary number representation, using the K j The K generated by compressing the jth to-be-compressed operation layer with different compression rates j The parameter values ​​of the operation branches are respectively used Bit binary number representation, where is an integer greater than or equal to 1, and Less than or equal to N 0j .

9. An apparatus for compressing a neural network, comprising: The device comprises: an obtaining unit, configured to obtain a neural network to be compressed, the neural network to be compressed comprising J operation layers to be compressed, wherein J is an integer greater than 1; a compression unit configured to compress a jth to-be-compressed operation layer of the J to-be-compressed operation layers using K j different compression rates to generate K j operation branches, where j and K j are integers, and 1≤j≤J, K j ≥1. a weight factor obtaining unit, configured to obtain, for the jth to-be-compressed operation layer, a group of K j weight factors corresponding to K j operation branches respectively, the K j weight factors having initial values respectively; a replacement unit configured to replace, for the jth compressed operation layer, the operation layer to be compressed with the K j operation branches weighted by the set of K j weight factors, to generate a replaced neural network. a forward propagation unit configured to perform forward propagation on the replaced neural network on a preset data set, wherein for the jth operation layer to be compressed, the K j operation results of the K j operation branches are weighted and summed with the K j weight factors as the output of the jth operation layer to be compressed; a back propagation unit, configured to perform back propagation on the replaced neural network on the preset data set, wherein, for the jth to-be-compressed operation layer, an update value of the K j weight factors is calculated based on a model loss of the replaced neural network relative to the to-be-compressed neural network, wherein the model loss comprises a loss function item and a performance index item, the loss function item is related to an application type of the to-be-compressed neural network, the performance index item is related to a hardware index of a hardware platform to which the to-be-compressed neural network is to be deployed, and the hardware index of the hardware platform to which the to-be-compressed neural network is to be deployed comprises storage space, floating point operation times, delay or energy consumption; and A determining unit is configured to determine, for the jth to-be-compressed operation layer, an operation branch corresponding to a largest one of the K j weight factors as the jth operation layer after compression.

10. An electronic device, comprising: comprising: a processor; and a storage device, configured to store a computer program capable of running on the processor; when the computer program is executed by the processor, the processor executes the method for compressing a neural network according to any one of claims 1-8.

11. A non-transitory computer readable storage medium, comprising: The non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the method for compressing a neural network according to any one of claims 1-8.

Images