Method and apparatus for pruning convolutional layers in a neural network

Abstract

This application discloses a method and apparatus for pruning convolutional layers in a neural network. The method for pruning convolutional layers in a neural network includes: obtaining a target neural network, the target neural network including convolutional layers to be pruned, the convolutional layers to be pruned including C filters, each filter including K convolutional kernels, each convolutional kernel including M rows and N columns of weight values, where C, K, M, and N are positive integers greater than or equal to 1; determining the number P weight values ​​to be pruned in each convolutional kernel based on the number M×N of weight values ​​in the convolutional kernels and the target compression ratio, where P is a positive integer less than M×N; and setting the P weight values ​​with the smallest absolute value in each convolutional kernel of the convolutional layer to be pruned to zero, forming a pruned convolutional layer.

Description

Technical Field

[0001] This application relates to neural network technology, and more specifically, to a method and apparatus for pruning convolutional layers in a neural network. Background Technology

[0002] In recent years, deep learning technology has been applied to many technological fields, such as image recognition, speech recognition, autonomous driving, and medical imaging. Convolutional Neural Networks (CNNs) are a representative network structure and algorithm in deep learning, achieving great success in image processing. However, existing CNN models have a large number of parameters, consuming significant storage space and computational resources, which limits their application scope. Summary of the Invention

[0003] One objective of this application is to provide a method for pruning convolutional layers in neural networks to improve pruning efficiency and accuracy.

[0004] According to some aspects of this application, a method for pruning convolutional layers in a neural network is provided. The method includes: obtaining a target neural network, the target neural network including convolutional layers to be pruned, the convolutional layers to be pruned including C filters, each filter including K convolutional kernels, each convolutional kernel including M rows and N columns of weight values, where C, K, M, and N are positive integers greater than or equal to 1; determining the number P of weight values ​​to be pruned in each convolutional kernel based on the number M×N of weight values ​​in the convolutional kernels and a target compression ratio, where P is a positive integer less than M×N; and setting the P weight values ​​with the smallest absolute values ​​in each convolutional kernel of the convolutional layer to be pruned to zero, forming a pruned convolutional layer.

[0005] In some embodiments, the method further includes: retraining the target neural network including the pruned convolutional layer to form an updated neural network, the updated neural network including an updated convolutional layer generated by retraining the pruned convolutional layer, wherein the weights in the updated convolutional layer are zero at positions corresponding to the positions where the weights in the pruned convolutional layer are set to zero.

[0006] In some embodiments, retraining a target neural network including the pruned convolutional layer to generate an updated neural network includes: generating a masking tensor, wherein each element of the masking tensor corresponds to a weight value in the pruned convolutional layer, and the elements in the masking tensor corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero are 0, and the elements at the remaining positions are 1; and using the masking tensor to set the gradient values ​​in the error gradient tensor corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero, such that the weight values ​​in the updated convolutional layer corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero are zero.

[0007] In some embodiments, using the masking tensor to set the gradient value in the error gradient tensor to zero at the position corresponding to the zeroing position of the weight value in the pruned convolutional layer includes: performing an Adama multiplication operation on the masking tensor and the error gradient tensor.

[0008] In some embodiments, the target compression ratio is set based on a target accuracy, the target compression ratio being such that the updated neural network performs neural network operations with an accuracy greater than or equal to the target accuracy.

[0009] In some embodiments, the method further includes: obtaining the accuracy of the updated neural network performing neural network operations; comparing the updated accuracy with the target accuracy; and if the updated accuracy is less than the target accuracy, increasing the target compression ratio and redetermining the number P of the weight values ​​to be pruned based on the increased target compression ratio.

[0010] In some embodiments, setting the P weight values ​​with the smallest absolute values ​​in each convolutional kernel to zero includes: expanding the convolutional layer to be pruned into a two-dimensional matrix of C×K rows and M×N columns according to the number of weight values; sorting the M×N weight values ​​in each row of the two-dimensional matrix according to their absolute values; setting the P weight values ​​with the smallest absolute values ​​in each row of the M×N weight values ​​to zero; and rearranging the two-dimensional matrix to obtain C filters corresponding to the convolutional layer to be pruned, each filter including K convolutional kernels, each convolutional kernel including M rows and N columns of weight values, thereby forming a pruned convolutional layer.

[0011] In some embodiments, the convolutional layer to be pruned or the updated convolutional layer is used to perform convolution operations with the K input channels of the input layer to generate a computation result that is output by the C output channels of the output layer.

[0012] In some embodiments, the neural network is a convolutional neural network.

[0013] According to other aspects of this application, an apparatus for pruning convolutional layers in a neural network is provided. The apparatus includes: an acquisition unit for acquiring a target neural network, the target neural network including convolutional layers to be pruned, the convolutional layers to be pruned including C filters, each filter including K convolutional kernels, each convolutional kernel including M rows and N columns of weight values, where C, K, M, and N are positive integers greater than or equal to 1; a pruning quantity determination unit for determining the number P weight values ​​to be pruned in each convolutional kernel based on the number M×N of weight values ​​in the convolutional kernels and a target compression ratio, where P is a positive integer less than M×N; and a pruning unit for setting the P weight values ​​with the smallest absolute values ​​in each convolutional kernel of the convolutional layer to be pruned to zero, forming a pruned convolutional layer.

[0014] According to other aspects of this application, an electronic device is provided, comprising: a processor; and a storage device for storing a computer program executable on the processor; wherein, when the computer program is executed by the processor, the processor causes the processor to perform the above-described method for pruning convolutional layers in a neural network.

[0015] According to other aspects of this application, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for pruning convolutional layers in a neural network.

[0016] The above is an overview of this application, and there may be simplifications, generalizations, and omissions of details. Therefore, those skilled in the art should recognize that this section is merely illustrative and not intended to limit the scope of this application in any way. This overview section is neither intended to identify the key or essential features of the claimed subject matter nor to serve as an aid in determining the scope of the claimed subject matter. Attached Figure Description

[0017] The above and other features of this application will become more fully clear through the following description and appended claims, in conjunction with the accompanying drawings. It is understood that these drawings depict only a few embodiments of the application and should not be construed as limiting the scope of the application. The application will be described more clearly and in more detail through the use of the drawings.

[0018] Figure 1 A flowchart is shown below illustrating a method for pruning convolutional layers in a neural network according to an embodiment of this application;

[0019] Figure 2 A schematic diagram of a neural network according to an embodiment of this application is shown;

[0020] Figure 3 It shows Figure 2 A schematic diagram of some exemplary convolutional kernels in the convolutional layer of the neural network shown;

[0021] Figure 4 A flowchart is shown illustrating a method for retraining a neural network including pruned convolutional layers according to an embodiment of this application.

[0022] Figure 5 A schematic diagram illustrating convolution operations using a retrained and updated convolution kernel according to an embodiment of this application is shown;

[0023] Figure 6 A comparison diagram showing the effect of a method for pruning convolutional layers in a neural network according to an embodiment of this application and existing pruning methods is presented; and

[0024] Figure 7 A block diagram of an apparatus for pruning convolutional layers in a neural network according to an embodiment of this application is shown. Detailed Implementation

[0025] In the following detailed description, reference is made to the accompanying drawings, which form a part thereof. In the drawings, similar symbols generally denote similar components unless the context otherwise requires. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments and variations may be employed without departing from the spirit or scope of the subject matter of this application. It will be understood that various different configurations, substitutions, combinations, and designs can be made to the various aspects of the general description and illustrated in the drawings of this application, all of which explicitly form part of the subject matter of this application.

[0026] Convolutional Neural Networks (CNNs), as one of the representative algorithms of deep learning, are a type of feedforward neural network with a deep structure. A CNN typically includes one or more convolutional layers and corresponding pooling layers. The convolutional layers are used to extract features from the input data; the more convolutional layers there are, the more features can be extracted, which is beneficial for obtaining more accurate output results. However, as the number of convolutional layers and the size of the convolutional kernels increase, not only does the computational pressure increase, but the bandwidth requirements for reading weight values ​​from external memory for computation in batch processing mode also become increasingly higher.

[0027] The inventors of this application have discovered that in convolutional neural networks, the computational load and complexity primarily depend on convolutional layers with large-sized kernels (e.g., 3×3, 5×5, or 7×7 kernels). However, these kernels may contain significant redundancy, meaning they may contain weights that contribute little or no benefit to the accuracy of the convolutional neural network's output. Therefore, if these redundant weights can be pruned (e.g., set to zero), the computational load of the neural network can be reduced, thereby reducing energy consumption.

[0028] Based on the above-mentioned inventive concept, this application provides a method for pruning convolutional layers in a neural network. Specifically, based on the number of weight values ​​of the convolutional kernels in the convolutional layer and the target compression ratio, the number P weight values ​​to be pruned in each convolutional kernel is determined. The P weight values ​​with the smallest absolute values ​​in each convolutional kernel of that convolutional layer are directly set to zero, forming a pruned convolutional layer. This method does not perform sensitivity analysis on the convolutional layer or kernel during kernel pruning; that is, it does not evaluate the impact of the pruned convolutional layer or kernel on the accuracy of the neural network output. Instead, it directly prunes the same number of weight values ​​with the smallest absolute values ​​from all convolutional kernels within a single convolutional layer. Therefore, the implementation process of this technical solution is simpler, and since the pruning rules for each convolutional kernel are the same, the corresponding hardware circuit implementation complexity is also lower.

[0029] The method for pruning convolutional layers in a neural network according to this application will be described in detail below with reference to the accompanying drawings. Figure 1 A flowchart of a method 100 for pruning convolutional layers in a neural network according to some embodiments of this application is shown, specifically including the following steps S120-S180.

[0030] Step S120: Obtain the target neural network, which includes convolutional layers to be pruned.

[0031] The target neural network can be a neural network obtained after training on a training sample dataset. For example, the target neural network can be a LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, or other types of convolutional neural networks trained on CIFAR10, ImageNet, or other types of datasets. In a specific example, the target neural network can be a ResNet56 convolutional neural network trained on the CIFAR10 dataset. It should be noted that although the following embodiments are illustrated using convolutional neural networks as examples, those skilled in the art will understand that the pruning method of this application is applicable to any neural network containing convolutional layers.

[0032] In some embodiments, the target neural network may include one or more convolutional layers, as well as pooling layers, fully connected layers, etc. Figure 1 The method 100 shown can prune one, several, or all convolutional layers of the target neural network according to actual needs. For ease of description, the following description uses pruning a single convolutional layer to be pruned as an example, assuming that the layer to be pruned includes C filters, each filter includes K convolutional kernels, and each convolutional kernel includes M rows and N columns (M×N) weight values, where C, K, M, and N are positive integers greater than or equal to 1. The convolutional layer to be pruned is used to perform convolution operations with the outputs of the K input channels of the input layer, and the result is provided to the C output channels of the output layer. Those skilled in the art will understand that the number of filters in the convolutional layer is the same as the number of output channels of the output layer (i.e., both are C), and the number of convolutional kernels in each filter is the same as the number of input channels of the input layer 100 (i.e., both are K). Each filter performs convolution operations (i.e., dot multiplication and summation operations) with each input channel in the input layer to obtain the corresponding output channel of the output layer.

[0033] refer to Figure 2 , Figure 2 It shows Figure 1 The illustrated method is a schematic diagram of the target neural network to which the method is applied, including an exemplary convolutional layer 200. The convolutional layer 200 is located between the input layer 300 and the output layer 400, and is used to perform convolution operations on the data input to the input layer 300 to generate an operational structure, which is then output by the output layer 400. Figure 2 In the example shown, convolutional layer 200 may include five filters 210, 220, 230, 240, and 250, which are convolved with the corresponding data output from input layer 300, and the results are output by the five output channels 410, 420, 430, 440, and 450 of output layer 400. Each of filters 210, 220, 230, 240, and 250 may include three convolution kernels, which are used to perform convolution operations with the three input channels 310, 320, and 330 of input layer 300, respectively. For example, filter 210 includes... Figure 3 The three convolutional kernels 211, 212, and 213 shown in (a)-(c) each include 3 rows and 3 columns of weight values. In some exemplary applications for image processing or image recognition, Figure 2 The input layer 300 shown can be RGB format image data, and the three input channels 310, 320, and 330 can be the R, G, and B color channels of the image data, respectively. After processing with the convolutional layer 200, the five output channels 410, 420, 430, 440, and 450 of the output layer 400 can obtain the feature information of the image data in five dimensions. In other embodiments, the input layer can also be audio data, text data, etc., depending on the specific application scenario of the convolutional neural network.

[0034] refer to Figure 2 and Figure 3 In the example shown, when this convolutional layer 200 is used as a pruning convolutional layer, the values ​​of C, K, M, and N can be 5, 3, 3, and 3, respectively. It should be understood that... Figure 2 and Figure 3 The convolutional layer shown is only an example to describe this application. In other embodiments, the parameters C, K, M, and N of the convolutional layer to be pruned may also take other different values.

[0035] Step S140: Determine the number of weight values ​​to be pruned in each convolutional kernel based on the number of weight values ​​in the convolutional kernel of the convolutional layer to be pruned and the target compression ratio.

[0036] The target compression ratio is the ratio of the number of non-zero weight values ​​in the convolutional layer after pruning to the number of weight values ​​in the convolutional layer before pruning, denoted by R.

[0037] In some embodiments, the target compression ratio R for each convolutional layer to be pruned can be preset based on specific application scenarios or computational conditions, such as according to the amount of computation or storage space that needs to be reduced in that specific application scenario or computational condition. For example, the target compression ratio R is a value greater than zero and less than 1, such as it can be set to 4 / 5, 3 / 4, 2 / 3, 1 / 2, etc.

[0038] Taking a convolutional layer with the parameters C, K, M, and N as an example, the number of weights in each convolutional kernel is M×N. Combined with the target compression ratio R, the number P of weights to be pruned in each convolutional kernel can be determined. That is, multiplying the number of weights M×N by (1-R) ​​and then rounding down yields the number P of weights to be pruned. In some embodiments, when rounding the product of M×N and (1-R), the non-integer part is rounded to the nearest integer. In some embodiments, to ensure that the target compression ratio is achieved after pruning, the product of M×N and (1-R) ​​is rounded up. It is understood that in some embodiments, rounding down or other rounding methods may also be used depending on the specific application scenario. Since the target compression ratio R is greater than zero and less than 1, the value of P is a positive integer less than M×N.

[0039] It is understandable that a neural network can have multiple convolutional layers, and the number of weight values ​​in the convolutional kernels of different convolutional layers may be the same or different. For example, different convolutional layers may include 3×3, 3×5, 5×5, 5×7 or 7×7 convolutional kernels, and the number of weight values ​​included in these convolutional kernels are 9, 15, 25, 35 and 49 respectively. Taking a target compression ratio of 2 / 3 as an example, for a 3×3 convolution kernel, the number of weight values ​​to be pruned is (3×3)×(1-2 / 3)=3, so the number of weight values ​​to be retained is 6; for a 5×5 convolution kernel, the number of weight values ​​to be pruned is (5×5)×(1-2 / 3) rounded up, which is 9, so the number of non-zero weight values ​​to be retained is 16; and for a 7×7 convolution kernel, the number of weight values ​​to be pruned is (7×7)×(1-2 / 3) rounded up, which is 17, so the number of non-zero weight values ​​to be retained is 32.

[0040] Step S160: Set the weights with the smallest absolute values ​​in each convolution kernel of the convolutional layer to be pruned, which are equal to the number of kernels to be pruned, to zero to form a pruned convolutional layer.

[0041] Continuing with an example of a convolutional layer with the parameters C, K, M, and N, we will explain the specific pruning operation.

[0042] In some embodiments, the convolutional layer to be pruned is first unfolded into a two-dimensional matrix of C×K rows and M×N columns based on the number of weights. Then, the M×N weights in each row of this two-dimensional matrix are sorted according to their absolute values. Next, the P weights with the smallest absolute values ​​among the M×N weights in each row are set to zero. Then, this two-dimensional matrix is ​​rearranged to obtain C filters corresponding to the convolutional layer to be pruned. Each filter includes K convolutional kernels, and each convolutional kernel includes M rows and N columns of weights, thus forming the pruned convolutional layer. It can be understood that the weights that are not set to zero occupy the same positions in the pruned convolutional layer as before pruning. In other embodiments, the matrix unfolding operation described above can be omitted. Instead, the C×K convolutional kernels in the convolutional layer to be pruned are processed sequentially, and the P weights with the smallest absolute values ​​in each convolutional kernel are set to zero, forming each convolutional kernel of the pruned convolutional layer.

[0043] It should be noted that in the pruning method of this application embodiment, the number of weight values ​​set to zero in each convolution kernel of the convolutional layer to be pruned is the same, that is, the number of weight values ​​to be pruned is P. Compared with the prior art solution where the number of weight values ​​set to zero in each convolution kernel may be different, this solution is more advantageous for hardware circuit implementation.

[0044] refer to Figure 3 The figure shows a schematic diagram of pruning the filter 210 in the pruned convolutional layer 200 with a compression ratio of 2 / 3. It can be seen that, when the filter 210 in the pruned convolutional layer 200 is pruned... Figure 3 In (a), the three weights with the smallest absolute values ​​at positions (0,1), (2,0), and (2,2) of the convolution kernel 211 are set to zero to form... Figure 3 (d) The pruned convolution kernel 211'; in the case of... Figure 3 (b) The three weights with the smallest absolute values ​​at positions (0,0), (1,2), and (2,1) of the convolution kernel 212 are set to zero to form... Figure 3 (e) the pruned convolution kernel 212'; and in the case of... Figure 3 (c) The three weights with the smallest absolute values ​​at positions (0,2), (1,1), and (2,0) of convolution kernel 213 in (c) are set to zero to form... Figure 3 (f) shows the pruned convolution kernel 213'.

[0045] In some embodiments, after step S160, a convolutional layer in the target neural network completes the pruning operation. The pruned convolutional layer has fewer weight values, so performing convolution operations based on it can reduce the amount of computation.

[0046] exist Figure 1 In the illustrated embodiment, after step S160, further processing can be performed to adjust the target neural network, particularly to improve accuracy.

[0047] Specifically, in step S180, the target neural network including the pruned convolutional layer is retrained to form an updated neural network. The updated neural network includes updated convolutional layers generated by retraining the pruned convolutional layers; and the weights in the updated convolutional layers at positions corresponding to the zeroed weights in the pruned convolutional layers are zero.

[0048] In some embodiments, retraining the target neural network, including the pruned convolutional layers, can use the same sample dataset as the target neural network used for training, such as CIFAR10, ImageNet, or other types of datasets. In other embodiments, retraining can also use a different sample dataset than the target neural network used for training. The reason for performing the retraining operation in step S180 is that although pruning the convolutional layers in the target neural network can effectively reduce the parameters and computational cost of the convolutional layers, the accuracy of the target neural network including the pruned convolutional layers usually suffers some loss because some weight values ​​in the original convolutional layers are removed. Therefore, retraining the target neural network including the pruned convolutional layers, fine-tuning and updating the non-zeroed weight values ​​in the pruned convolutional layers, can reduce the loss of accuracy.

[0049] However, it should be noted that in some embodiments, retraining the target neural network including pruned convolutional layers only requires updating the non-zero weight values ​​of the pruned convolutional layers, avoiding updating the weight values ​​set to zero in the pruning operation to non-zero values. In other embodiments, retraining the target neural network including pruned convolutional layers may also update a portion of the zeroed weight values ​​to non-zero values. Preferably, to reduce computational load, retraining does not update the weight values ​​set to zero in the pruning operation to non-zero values. Accordingly, in some embodiments of the present invention, a masking tensor can be generated, in which each element corresponds to each weight value in the pruned convolutional layer, and the elements at the positions of the masking tensor corresponding to the zeroed weight values ​​in the pruned convolutional layer are 0, while the elements at the remaining positions are 1. In the process of retraining a target neural network, including pruned convolutional layers, to generate an updated neural network, a masking tensor is used to set the gradient values ​​in the error gradient tensor of the retraining process to zero at the positions where the weight values ​​in the pruned convolutional layers are set to zero. This ensures that the weight values ​​in the updated convolutional layers at the positions where the weight values ​​in the pruned convolutional layers are set to zero are all zero.

[0050] refer to Figure 4 This illustrates a process in some embodiments for retraining a target neural network, including pruned convolutional layers, to produce an updated neural network. The process includes the following steps.

[0051] Step S182: Generate the masking tensor.

[0052] Specifically, a masking tensor is generated. The masking tensor is a tensor with the same size as the pruned convolutional layer, and each element in the masking tensor corresponds to a weight value in the pruned convolutional layer. For example, the masking tensor also has four dimensions: C, K, M, and N. Next, the masking tensor is initialized so that the elements in the masking tensor corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero are 0, and the non-zero elements at other positions are 1.

[0053] Step S184: Retrain the target neural network, including the pruned convolutional layer, to obtain the error gradient tensor corresponding to the pruned convolutional layer.

[0054] In some embodiments, retraining includes forward propagation of the target neural network, including pruned convolutional layers, onto the training dataset. Forward propagation refers to inputting the input data of the training dataset into the target neural network including pruned convolutional layers for convolution operations, and obtaining the output of the pruned convolutional layer for the input data. This output is then compared with the standard output obtained by performing convolution operations on the same input data by the original target neural network's pruned convolutional layer; the difference between these two values ​​serves as the gradient tensor of the pruned convolutional layer.

[0055] Step S186: Obtain the pruning error gradient tensor based on the error gradient tensor and the masking tensor.

[0056] In some embodiments, the error gradient tensor gradient and the mask tensor mask can be multiplied by Hadamard, that is, element-wise multiplication, to obtain the pruned error gradient tensor gradient grsdient′. Similar to the mask tensor mask, the elements of this pruned error gradient tensor gradient grsdient′ at the positions corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero are 0.

[0057] Step S188: Update the pruned convolutional layer using the pruning error gradient tensor to generate an updated convolutional layer.

[0058] In some embodiments, the backpropagation algorithm is used to obtain the change in the weight values ​​of the convolutional layer based on the pruning error gradient tensor grsdient′, and the weight values ​​of the pruned convolutional layer are updated according to this change, so that the difference between the output value of the updated convolutional layer and the standard output value is reduced. Specifically, the updated convolutional layer can be obtained by performing gradient update on the pruned convolutional layer according to the following formula (1):

[0059] w′=w+λ*(gradientomask) formula (1)

[0060] Where w′ represents the updated convolutional layer, w represents the pruned convolutional layer, λ represents the learning rate, gradient represents the error gradient tensor, mask represents the mask tensor, o is the Adama operator, which represents the element-wise multiplication of two tensors, and (gradientomask) represents the pruned error gradient tensor gradient′.

[0061] By retraining the target neural network, including the pruned convolutional layers, during each error backpropagation and gradient update, the obtained error gradient tensor is multiplied by the masking tensor using an Adama multiplication to obtain the pruned error gradient tensor. This pruned error gradient tensor is then used to update the pruned convolutional layers. Since the elements in the pruned error gradient tensor are 0 at the positions corresponding to the zeroed-out weight values ​​in the pruned convolutional layers, it is ensured that the pruned weight values ​​remain 0 throughout the entire update process.

[0062] In some embodiments, steps S182 to S188 can be executed iteratively multiple times until the error gradient tensor reaches a small value. For example, an error gradient threshold can be set. After obtaining the error gradient tensor in step S184, it can be compared with the error gradient threshold: if the error gradient tensor is greater than the error gradient threshold, then proceed to the subsequent step S186; if the error gradient tensor is less than the error gradient threshold, then end the retraining process. After the retraining process is completed, the last obtained convolutional layer is used as the updated convolutional layer.

[0063] It should be noted that step S140 described above is an embodiment where the target compression ratio is pre-set based on a specific application scenario. In other embodiments, the target compression ratio can also be set based on target accuracy. For example, the target compression ratio can be set such that the accuracy of the updated neural network performing neural network operations is greater than or equal to the target accuracy. Target accuracy refers to the acceptable accuracy threshold of the neural network after pruning the convolutional layers in the neural network, resulting in accuracy loss. Generally, the lower the target compression ratio, the more weight values ​​need to be pruned, and the greater the accuracy loss of the neural network. Therefore, a trade-off needs to be made between the target compression ratio and the target accuracy, pruning as many weight values ​​as possible while ensuring that the target accuracy requirements of the current application scenario are met. Accordingly, in some embodiments, the target compression ratio can be adjusted according to the target accuracy. Specifically, updated accuracy data for neural network operations can be obtained; then, the updated accuracy data is compared with the target accuracy data. If the updated accuracy data is less than the target accuracy data, the target compression ratio is increased, and the number of branches to be pruned is re-determined based on the increased target compression ratio. Steps S140 to S180 are iteratively executed until the updated accuracy data is greater than or equal to the target accuracy data. Alternatively, after comparing the updated accuracy data with the target accuracy data, if the updated accuracy data is greater than the target accuracy data, the target compression ratio can be decreased, and the number of branches to be pruned can be re-determined based on the decreased target compression ratio to remove as many weight values ​​as possible.

[0064] It should also be noted that although the above embodiments illustrate this application by pruning one convolutional layer to be pruned in the target neural network, this is only for illustrative purposes. It should be understood that the technical solution of this application can also be used to prune multiple or all convolutional layers in the target neural network. Furthermore, as the size and depth of a convolutional neural network increase, it often contains a large number of convolutional layers. The number of filters, the size of the convolutional kernel, and its position within the convolutional neural network may all differ for each convolutional layer. To minimize the compression ratio of the entire target neural network while ensuring high accuracy, different target compression ratios can be set for each convolutional layer in the target neural network. For example, in a convolutional neural network, the redundancy of the initial convolutional layers is usually small, while the redundancy of later convolutional layers is usually higher. Therefore, a lower target compression ratio can be set for later convolutional layers, while a higher target compression ratio can be set for earlier convolutional layers.

[0065] In some embodiments, after obtaining the updated convolutional layer, the neural network containing the updated convolutional layer needs to be stored for use in subsequent calculations. Since the updated convolutional layer has undergone pruning and includes a large number of sparse weight matrices, it can be compressed for storage to reduce storage space. When the neural network is needed for specific computations, in a static configuration, the stored updated convolutional layer can be directly read out and rearranged for use; in a dynamic configuration (e.g., deformable network), it needs to be transformed according to the data path (e.g., offset, rotation, etc.) before use. During use, since a large number of weights in the convolutional layer have been set to zero, the bandwidth requirement for reading weights from external memory is reduced, the number of weights involved in the calculation is reduced, and thus the operating efficiency is improved. It is understood that the storage and retrieval of convolutional layers can be implemented in various suitable ways.

[0066] For example, the convolution operation using the unpruned convolutional layer can be described using formula (2):

[0067] y[i,j,c]=∑ k ∑ (m,n)∈Ω(ω) w[m,n,k,c]×[i+m,j+n,k] formula (2);

[0068] In this context, the convolutional layer is represented by a four-dimensional tensor w[m,n,k,c], where c is the index of the filter in the convolutional layer, k is the index of the convolutional kernel in each filter, and m and n are the row and column indices in each convolutional kernel; y[i,j,c] represents the output layer element; and [i+m,j+n,k] represents the input layer element. When the convolutional kernel is a 3×3 matrix and all weight values ​​are non-zero, the non-zero elements in the set Ω{ω} are ω={(0,0),(0,1),(0,2),(1,0),(1,1),(1,2),(2,0),(2,1),(2,2)}.

[0069] Correspondingly, the convolution operation of the updated convolutional layer after pruning can be described by formula (3):

[0070] y[i,j,c]=∑ k ∑ (m,n)∈Ω(ω′) w[m,n,k,c]×[i+m,j+n,k] formula (3);

[0071] In Equation (3), the same symbols represent the same elements as in Equation (2). However, since the updated convolutional layer has set a large number of weight values ​​to zero based on the target compression ratio, the number of non-zero elements in the set Ω(ω') is greatly reduced, which greatly reduces the computational cost of convolution operations.

[0072] The following example illustrates the pruning of filter 210. Figure 3 In the diagram, (a), (b), and (c) represent the element patterns of convolution kernels 211, 212, and 213 in filter 210 before pruning, and (d), (e), and (f) represent the element patterns of the corresponding convolution kernels 211', 212', and 213' in filter 210 after pruning. Shaded boxes represent non-zero elements, while blank boxes represent zero elements. It can be seen that before pruning, each corresponding non-zero element ω in convolution kernels 211, 212, and 213 is {(0,0),(0,1),(0,2),(1,0),(1,1),(1,2),(2,0),(2,1),(2,2)}; while after pruning, the corresponding non-zero elements of convolution kernels 211', 212', and 213' are:

[0073] ω211'={(0,0),(0,2),(1,0),(1,1),(1,2),(2,1)};

[0074] ω212'={(0,1),(0,2),(1,0),(1,1),(2,0),(2,2)};

[0075] ω213'={(0,0),(0,1),(1,0),(1,2),(2,1),(2,2)};

[0076] As can be seen, after pruning, the number of non-zero elements in each convolution kernel is reduced from 9 to 6, significantly reducing the computational load of convolution operations. (Reference) Figure 5 (a) to (b), which illustrate the computation of elements at (0,0) and (0,1) of the first output channel 410 using pruned convolution kernels 211', 212', and 213'. Specifically, as Figure 5 As shown in (a), the three 3×3 convolution kernels 211', 212', and 213' in the pruned filter 210 are multiplied by the top-left 3×3 matrices of the three input channels 310, 320, and 330 of the input layer 300, and then added together to obtain the element at (0,0) of the first output channel 410 of the output layer; then, as... Figure 5 As shown in (b), the value box of input layer 300 is "slid" one position to the right, so that the 3×3 matrices starting from the second column in the three input channels 310, 320, and 330 of input layer 300 are multiplied by the three convolution kernels 211', 212', and 213' respectively, and then added together to obtain the element at (0,1) of the first output channel 410 of the output layer. Continuing to "slide" the value box of input layer 300 to the right and down, the data matrices of the three input channels of input layer 300 are processed by the three convolution kernels to obtain the elements at the remaining positions of the first output channel 410 of the output layer, which will not be elaborated further here.

[0077] refer to Figure 6 The paper compares the proposed method for pruning convolutional layers in a neural network with traditional Filter_wise and Kernel_wise pruning methods using a ResNet56 convolutional neural network trained on the CIFAR10 dataset. The Filter_wise or Kernel_wise methods perform sensitivity analysis on each convolutional layer before pruning, independently pruning each layer based on filters or kernels, and then evaluate the accuracy of the pruned network on the test dataset. Convolutional layers with a greater decrease in accuracy are more sensitive. The pruning ratio for each layer's filters or kernels is then set based on the sensitivity, and the entire network is retrained. In contrast, the proposed method for pruning convolutional layers in a neural network does not perform sensitivity analysis; it only needs to set the number of kernels to be pruned and directly prunes the weights of all kernels, simplifying the process. Furthermore, from... Figure 6As can be seen, under different sparsity conditions (sparseness = 1 - compression ratio, for example, 90% sparsity corresponds to 10% compression ratio), the pruning method of this application has higher accuracy than the Filter_wise and Kernel_wise pruning methods. In other words, under the same accuracy conditions, the pruning method of this application can prune more weight values, resulting in higher performance gains.

[0078] This application also provides an apparatus for pruning convolutional layers in a neural network. For example... Figure 7 As shown, the apparatus 700 for pruning convolutional layers in a neural network includes an acquisition unit 710, a pruning quantity determination unit 720, and a pruning unit 730. The acquisition unit 710 acquires the target neural network, which includes convolutional layers to be pruned. Each pruned convolutional layer includes C filters, each filter includes K convolutional kernels, and each convolutional kernel includes M rows and N columns of weight values, where C, K, M, and N are positive integers greater than or equal to 1. The pruning quantity determination unit 720 determines the pruning quantity P based on the number of weight values ​​M×N in the convolutional kernels and the target compression ratio, where P is a positive integer less than M×N. The pruning unit 730 sets the P weight values ​​with the smallest absolute value in each convolutional kernel of the pruned convolutional layer to zero, forming the pruned convolutional layer. For a detailed description of the apparatus 700, please refer to the above text in conjunction with the appendix. Figures 1 to 6 The description of the corresponding methods will not be repeated here.

[0079] In some embodiments, the apparatus 700 for pruning convolutional layers in a neural network can be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. Furthermore, the apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] In other embodiments, the apparatus 700 for pruning convolutional layers in a neural network can also be implemented as a software functional unit. If the functional unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium and executed by a computer device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention.

[0081] This application also provides an electronic device, which includes a processor and a storage device. The storage device stores a computer program that can run on the processor. When the computer program is executed by the processor, it causes the processor to perform the method for pruning convolutional layers in a neural network as described in the above embodiments. In some embodiments, the electronic device may be a mobile terminal, a personal computer, a tablet computer, a server, etc.

[0082] This application also provides a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program performs the method for pruning convolutional layers in a neural network as described in the above embodiments. In some embodiments, the non-volatile computer-readable storage medium may be flash memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of non-volatile computer-readable storage medium known in the art.

[0083] Those skilled in the art can understand and implement other modifications to the disclosed embodiments by studying the specification, the disclosure, the drawings, and the appended claims. In the claims, the word "comprising" does not exclude other elements and steps, and the words "a" or "an" do not exclude a plurality. In practical applications of this application, a single part may perform the function of multiple technical features referenced in the claims. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A method for pruning one or more convolutional layers in a target neural network to process input data, characterized in that The method comprises: Obtain a to-be-pruned convolutional layer from one or more convolutional layers of the target neural network, wherein the to-be-pruned convolutional layer includes C filters, each filter includes K convolutional kernels, each convolutional kernel includes M rows and N columns of weight values, wherein C, K, M, and N are positive integers greater than or equal to 1; Determining the number P of weight values ​​to be pruned in each convolution kernel of the convolution layer to be pruned based on the number M×N of weight values ​​in the convolution kernel and the target compression rate of the convolution layer to be pruned, where P is a positive integer less than M×N; The P weight values ​​with the smallest absolute value in each convolution kernel of the convolution layer to be pruned are set to zero to form a pruned convolution layer; Retraining the target neural network including the pruned convolutional layer to form an updated neural network including an updated convolutional layer; and Processing the input data using the updated neural network to generate output data, wherein the updated convolutional layer is used to perform a convolution operation with the K input channels of the input layer to generate a calculation result to be output by the C output channels of the output layer, The input data includes image data in RGB format, each filter of the updated convolution layer includes 3 convolution kernels, and the 3 convolution kernels of each filter of the updated convolution layer are used to perform convolution operations with the R, G, and B color channel data of the image data, respectively, to generate corresponding operation results.

2. The method according to claim 1, characterized in that The weight value of the position in the updated convolutional layer corresponding to the position where the weight value in the pruned convolutional layer is set to zero is zero.

3. The method according to claim 2, characterized in that Retraining the target neural network including the pruned convolutional layer to generate an updated neural network includes: Generate a masked tensor, wherein each element in the masked tensor corresponds to each weight value in the pruned convolutional layer, and the elements at positions corresponding to the positions where the weight values ​​in the pruned convolutional layer are set to zero in the masked tensor are 0, and the elements at other positions are 1; and The masking tensor is used to set the gradient value at a position in the error gradient tensor corresponding to the position where the weight value in the pruned convolutional layer is set to zero, so that the weight value at a position in the updated convolutional layer corresponding to the position where the weight value in the pruned convolutional layer is set to zero is zero.

4. The method according to claim 3, characterized in that Using the masked tensor to set the gradient value at a position in the error gradient tensor corresponding to the position where the weight value in the pruned convolutional layer is set to zero comprises: Perform a Hadamard multiplication on the masked tensor and the error gradient tensor.

5. The method according to claim 2, characterized in that: The target compression rate is set based on a target accuracy, and the target compression rate makes the accuracy of the neural network operation performed by the updated neural network greater than or equal to the target accuracy.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining an update accuracy of the updated neural network performing a neural network operation; comparing the updated accuracy to the target accuracy; and If the updated accuracy is less than the target accuracy, the target compression rate is increased and the number P of weight values ​​to be pruned is re-determined based on the increased target compression rate.

7. The method according to claim 1, characterized in that Setting the P weight values ​​with the smallest absolute value in each convolution kernel of the to-be-pruned convolutional layer to zero includes: Expand the convolutional layer to be pruned into a two-dimensional matrix of C×K rows and M×N columns according to the number of weight values; Sort the M×N weight values ​​of each row in the two-dimensional matrix according to their absolute values; Set the P weight values ​​with the smallest absolute values ​​in each row of M×N weight values ​​to zero; and The two-dimensional matrix is ​​rearranged to obtain C filters corresponding to the convolutional layer to be pruned, each filter includes K convolution kernels, and each convolution kernel includes M rows and N columns of weight values, thereby forming a pruned convolutional layer.

8. The method according to claim 1, characterized in that: The neural network is a convolutional neural network.

9. The method according to claim 1, characterized in that: When the method is used to prune more than one convolutional layer in the target neural network, the method further comprises: Obtain another convolutional layer to be pruned from one or more convolutional layers of the target neural network, wherein the another convolutional layer to be pruned includes C′ filters, each filter includes K′ convolutional kernels, each convolutional kernel includes M′ rows and N′ columns of weight values, wherein C′, K′, M′ and N′ are positive integers greater than or equal to 1; Determining the number P′ of weight values ​​to be pruned in each convolution kernel of the other convolution layer to be pruned based on the number M′×N′ of weight values ​​in the convolution kernel and another target compression rate of the other convolution layer to be pruned, where P′ is a positive integer less than M′×N′; and The P′ weight values ​​with the smallest absolute value in each convolution kernel of the other convolution layer to be pruned are set to zero to form another pruned convolution layer.

10. The method according to claim 9, characterized in that The other target compression ratio is different from the target compression ratio.

11. An electronic device, characterized in that: include: processor; and A storage device for storing a computer program capable of running on the processor; Wherein, when the computer program is executed by the processor, the processor executes the method for pruning a convolutional layer in a neural network to process input data as described in any one of claims 1-10.

12. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for pruning a convolutional layer in a neural network to process input data as described in any one of claims 1 to 10 is implemented.

Images