Neural network compression method, device and storage medium

By adding encoding layer and activation layer to the convolutional neural network, the pruning method is optimized, and the complexity problem of the neural network running on resource-constrained devices is solved, and the effective compression and performance optimization of the model are achieved.

CN114764614BActive Publication Date: 2024-11-12CHINA MOBILE COMM LTD RES INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110029346.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-11
Publication Date
2024-11-12
Estimated Expiration
2041-01-11

AI Technical Summary

Technical Problem

When existing neural network models run on IoT devices with limited resources, they are too complex to increase power consumption, longer response time and excessive storage usage. There are difficulties in judging the importance of weights and constructing penalty items in the pruning method. .

Method used

In a convolutional neural network, by adding encoding and activation layers, the features are transformed and normalized, and iteratively trained using the training image set and the verification image set until the sparsity of the feature vector meets the threshold, and the redundant channel is deleted. Optimize the pruning method to achieve model compression.

Benefits of technology

Effectively reduce the model size, reduce the computational complexity, maintain high accuracy, is suitable for resource-constrained devices, and optimizes the compression effect of neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114764614B_ABST
    Figure CN114764614B_ABST
Patent Text Reader

Abstract

The present invention provides a neural network compression method, device and storage medium, which belong to the field of computer vision. The neural network compression method performs the following operations layer by layer on each designated convolution layer of the trained neural network: respectively add a coding layer and an activation layer after the designated convolution layer to obtain a neural network to be compressed; use a training image set and a verification image set to train the neural network to be compressed for m iterations until the sparsity of the feature vector output by the activation layer meets a first threshold; according to the feature vector output by the activation layer and a second threshold, delete the redundant channels in the designated convolution layer to obtain the pruned designated convolution layer; finally, obtain a compressed neural network model after pruning all designated convolution layers; the above method is used to optimize the compression effect of the convolution neural network by optimizing the pruning method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer vision technology, and in particular to a neural network compression method, device and storage medium. Background Art

[0002] As deep learning models become more and more complex, overly complex models in practical applications, especially when running on resource-constrained IoT devices, will lead to increased power consumption, longer response time, and excessive storage resources, thus affecting the performance of IoT devices. For example, for applications with a small training data set, an overly complex model may lead to a decline in performance due to problems such as overfitting.

[0003] Therefore, model compression technology has been widely used, and pruning is one of the neural network compression methods. Pruning refers to the method of removing redundant parameters in the model, thereby reducing the model size and reducing the computational complexity. There are many ways of pruning in the prior art, the most common of which are importance-based pruning and regularization-based pruning. The difficulty of the former lies in how to judge the importance of weights, and the difficulty of the latter lies in constructing a suitable penalty term. Summary of the invention

[0004] The present invention proposes a neural network compression method, device and storage medium, which optimize the compression effect of the convolutional neural network by optimizing the pruning method.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a neural network compression method, the method comprising:

[0007] For each specified convolutional layer in the trained neural network, perform the following operations in sequence:

[0008] A coding layer and an activation layer are added after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer;

[0009] Using a training image set and a verification image set, training the neural network to be compressed for m iterations until the sparsity of the feature vector output by the activation layer meets a first threshold;

[0010] According to the feature vector output by the activation layer and the second threshold, the redundant channels in the specified convolution layer are deleted to obtain the pruned specified convolution layer;

[0011] Obtain a compressed neural network model after pruning all specified convolutional layers.

[0012] Optionally, the encoding layer includes a global average pooling layer and two fully connected layers, and the features of the convolution layer are transformed by the global average pooling layer, and the features of the transformed convolution layer are extracted by the fully connected layer.

[0013] Optionally, the activation layer adjusts the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0014]

[0015] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0016] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0017] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0018] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0019] λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0020] Optionally, the neural network compression method further includes:

[0021] The compressed neural network after pruning all designated convolutional layers is trained for z iterations using a training image set and a validation image set.

[0022] In a second aspect, an embodiment of the present invention provides a neural network compression device, the device comprising:

[0023] The first processing module is used to perform the following operations in sequence for each specified convolutional layer in the trained neural network:

[0024] Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer;

[0025] Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold;

[0026] Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer;

[0027] The second processing module is used to obtain a compressed neural network model after pruning all designated convolutional layers.

[0028] Optionally, the encoding layer in the first processing module includes:

[0029] A global average pooling layer and two fully connected layers, the features of the convolutional layer are transformed by the global average pooling layer, and the features of the transformed convolutional layer are extracted by the fully connected layers.

[0030] Optionally, the activation layer in the first processing module is used to adjust the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0031]

[0032] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0033] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0034] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0035] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0036] λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0037] Optionally, the neural network compression device further includes:

[0038] The third processing module is used to use the training image set and the verification image set to train the compressed neural network after pruning all the designated convolutional layers for z iterations.

[0039] In a third aspect, an embodiment of the present invention provides a network device, including a transceiver and a processor;

[0040] The processor is used to perform the following operations in sequence for each specified convolutional layer in the trained neural network:

[0041] Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer;

[0042] Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold;

[0043] Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer;

[0044] The processor is also used to obtain a compressed neural network model after pruning all designated convolutional layers.

[0045] Optionally, the encoding layer includes a global average pooling layer and two fully connected layers, and the features of the convolution layer are transformed by the global average pooling layer, and the features of the transformed convolution layer are extracted by the fully connected layer.

[0046] Optionally, the activation layer adjusts the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0047]

[0048] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0049] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0050] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0051] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0052] λ∑α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0053] Optionally, the processor is further configured to use a training image set and a verification image set to train the compressed neural network for z iterations after pruning all designated convolutional layers.

[0054] In a fourth aspect, an embodiment of the present invention provides a network device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the neural network compression method of the first aspect.

[0055] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the neural network compression method of the first aspect are implemented.

[0056] The beneficial effects brought by the technical solution provided by the embodiment of the present invention include at least: the neural network compression method provided by the present invention performs the following operations on each designated convolution layer of the trained neural network layer by layer: adding a coding layer and an activation layer respectively after the designated convolution layer to obtain the neural network to be compressed; the coding layer is used to transform and extract the features output by the designated convolution layer; the activation layer is used to normalize and map the features output by the coding layer; using a training image set and a verification image set, the neural network to be compressed is trained for m iterations until the sparsity of the feature vector output by the activation layer meets a first threshold; according to the feature vector output by the activation layer and the second threshold, the redundant channels in the designated convolution layer are deleted to obtain the pruned designated convolution layer; finally, a compressed neural network model is obtained after pruning all designated convolution layers; the above method is used to optimize the compression effect of the convolution neural network by optimizing the pruning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0058] Figure 1 A flow chart of a neural network compression method provided by an embodiment of the present invention;

[0059] Figure 2A schematic diagram of an activation function provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram showing the effect of a change in the first adjustment parameter α on an activation function provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of a function g(x) provided in an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of the CIFAR10 image database used in an embodiment of the present invention;

[0063] Figure 6 A comparison diagram of whether a first adjustment parameter α is constrained or not provided in an embodiment of the present invention;

[0064] Figure 7 A schematic diagram of the structure of a neural network compression device provided by an embodiment of the present invention;

[0065] Figure 8 A schematic diagram of the structure of a network device provided by an embodiment of the present invention;

[0066] Fig. 9 A schematic diagram of the structure of a network device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Please refer to Figure 1 , Figure 1 A flow chart of a neural network compression method provided by an embodiment of the present invention is shown, and the method comprises:

[0069] For each specified convolutional layer in the trained neural network, perform the following operations in sequence:

[0070] Step 11, adding a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer.

[0071] Specifically, based on the trained convolutional neural network, a coding layer and an activation layer (Sigmoid) are added after the specified convolution layer. The coding layer transforms and extracts the features of the output generated by the current layer, and the activation layer normalizes and maps the features output by the coding layer. The physical meaning of the vector output by the activation layer is the importance of the current convolution kernel.

[0072] Specifically, the more important the feature, the higher the response value. After the activation function performs nonlinear transformation on the input feature, the importance of each channel can be better distinguished, which is more conducive to channel pruning according to the vector output by the activation layer.

[0073] Step 12: Using a training image set and a verification image set, train the neural network to be compressed for m iterations until the sparsity of the feature vector output by the activation layer meets a first threshold.

[0074] Specifically, after m iterations, the output vector of the activation layer has been sparse and can be used for the next pruning operation, and the specific sparsity threshold is the first threshold.

[0075] Step 13: Delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer.

[0076] Specifically, when the sparsity of the feature vector output by the activation layer satisfies the first threshold, that is, the output vector of the activation layer has been sparse, the channels corresponding to the values ​​close to 0 (setting the second threshold) in the vector are pruned.

[0077] Step 14, obtaining a compressed neural network model after pruning all specified convolutional layers.

[0078] Specifically, for each specified convolutional layer in the neural network, the above operations are performed layer by layer until the last convolutional layer that needs to be pruned.

[0079] It is worth noting that the encoding layer and activation layer added in step 11 have been deleted from the compressed neural network model.

[0080] The above-mentioned neural network compression method can optimize the compression effect of the convolutional neural network by optimizing the pruning method, can prune the specified convolutional layer, and configure different pruning ratios according to different needs, so that the pruned neural network maintains better accuracy.

[0081] Optionally, the encoding layer includes a global average pooling layer and two fully connected layers, and the features of the convolution layer are transformed by the global average pooling layer, and the features of the transformed convolution layer are extracted by the fully connected layer.

[0082] Specifically, the vector output by the currently connected convolutional layer is reduced in dimension to a one-dimensional vector through a global average pooling layer (GAP), and features are extracted from the output of the GAP layer through two fully connected layers.

[0083] Optionally, the activation layer adjusts the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0084]

[0085] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0086] Specifically, the method uses a custom activation function PSigmoid function in the activation layer, and adjusts the sparsity of the output vector of the activation layer through the first adjustment parameter α.

[0087] Specifically, each neuron node in the neural network accepts the output of the previous layer of neurons as the input of this neuron, and passes the output value to the next layer. In a multi-layer neural network, there is a functional relationship between the output of the upper layer node and the input of the lower layer node. This function is called the activation function.

[0088] Among them, the Sigmoid function is the most widely used type of activation function, such as Figure 2 The activation function diagram shown in Figure 2 is The Sigmoid function is continuous, smooth, strictly monotonic, and symmetric around (0, 0.5). It is a very good threshold function. The range of the Sigmoid function is limited to (0, 1). Since [0, 1] corresponds to the range of probability values, the Sigmoid function is associated with probability distribution.

[0089] In order to increase the separability (sparseness) of the output vector of the activation layer, this method uses the first adjustment parameter α to customize the PSigmoid function and add steep processing to the Sigmoid function. Figure 3 The activation function diagram shown in Figure 2 is The larger the value of the first adjustment parameter α is, the steeper the PSigmoid function is, and the more separable (sparse) the output vector is.

[0090] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0091] Specifically, during the training process, the parameter α in the PSigmoid function is added to the loss constraint to ensure that the α value gradually increases and remains at an appropriate value as the number of iterations increases.

[0092] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0093] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0094] λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0095] Specifically, in order to adapt to various network structures, the first adjustment parameter α is added to the loss function for constraint, and automatic optimization of the first adjustment parameter α is achieved, thereby achieving parameter adaptation.

[0096] Specifically, because in the process of network training, we always hope to minimize the loss function L, and we hope that the value of α will become larger. Putting α in the second term of the loss function will make it smaller and smaller under the constraint of the loss function. In order to make the first adjustment parameter α larger and larger, the g(α) function is used for adjustment. The g(α) function is a monotonically decreasing function. For example, refer to Figure 4 , g(α)=Sigmoid(-α) is a monotonically decreasing function in the range of (0, 1).

[0097] refer to Figure 6 As shown in the figure, if the α value in PSigmoid is not constrained by the loss function, the α value will not change in the direction we want, and the purpose of increasing the steepness of the Sigmoid function will not be achieved. When the α value is added to the constraint of the loss function, the α value will increase with the increase of the number of iterations and stabilize at a suitable value, which proves the effectiveness of our proposed method.

[0098] Optionally, the neural network compression method further includes:

[0099] The compressed neural network after pruning all designated convolutional layers is trained for z iterations using a training image set and a validation image set.

[0100] After pruning is completed, the compressed neural network can be made more accurate by retraining.

[0101] In summary, the neural network compression method provided by the embodiment of the present invention, the combination of the coding layer and the activation layer, can adapt to various network structures, has good universality, and makes full use of the characteristics of the nonlinear transformation function. By adding the activation layer parameters to the constraints of the loss function, parameter adaptation is achieved, the importance of each channel is better distinguished, and the pruning effect is optimized. The network structure can be optimized, computing resources and storage resources can be reduced, and support is provided for running high-performance neural networks on resource-constrained devices.

[0102] In order to further illustrate the neural network compression method provided by the present invention, the image classification task is taken as an example and the CIFAR10 database is used to further illustrate the neural network compression method provided by the present invention.

[0103] CIFAR10 is relatively simple. It is an image database consisting of 60,000 32x32 pixel color images, divided into 10 categories, with 6,000 images in each category. Figure 5 As shown, the 10 categories of pictures are airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.

[0104] In the first stage, data preparation is performed to format each image and the corresponding label, so that each image corresponds to a label and a training set is generated.

[0105] In the second stage, define the network structure, such as the classic network structure VGG16. Use the dataset in the first stage to train the VGG16 network for n iterations, or perform finetuning on the basis of the VGG16 model pre-trained on ImageNet. The model with the best accuracy on the validation set is obtained. Figure 1 The trained model in step 11 of .

[0106] In the third stage, the specified convolutional layers of the trained VGG16 model are pruned layer by layer (taking all convolutional layers as an example). The following operations are performed layer by layer for each specified convolutional layer:

[0107] A coding layer and an activation layer are added after each specified convolutional layer to obtain the neural network to be compressed. The coding layer consists of a global average pooling layer (GAP) and two fully connected layers.

[0108] The neural network to be compressed is trained for m iterations using a training image set and a validation image set until the sparsity of a feature vector output by the activation layer meets a first threshold.

[0109] Then pruning is performed based on the output of the activation layer. For example, for the first convolution layer, the dimension of the feature generated by convolution is 112*112*64. After a GAP layer, a 1*64-dimensional vector is obtained, and then two fully connected layers FC(64, 16) and FC(16, 64) are passed through. The numbers in brackets represent the input dimension and output dimension respectively. The vector generated by the encoding layer passes through an activation layer. Here, the activation function of the activation layer is a custom PSigmoid function. After training for m iterations, the output of the activation layer is a 1*64-dimensional vector. The range of each value in the vector is [0, 1]. The smaller the value, the less important the corresponding convolution kernel channel is. Since we constrained the parameters of the PSigmoid function, the output of the activation layer is more sparse, and the values ​​in the vector corresponding to the less important convolution kernel channels are distributed around 0. According to the second threshold, we delete the redundant channels and complete the pruning of a convolution layer. The VGG16 network has a total of 13 convolution layers. We repeat the above operation 13 times.

[0110] In the fourth stage, the model after pruning all convolutional layers is trained n times on the CIFAR10 training dataset.

[0111] In the fifth stage, the network model parameters are saved after pruning.

[0112] Table 1 Pruning results of CIFAR10 dataset on VGG16 model

[0113]

[0114] Referring to Table 1, through the above experimental results, we can see that when the CIFAR10 dataset is pruned by about 60% on VGG16, the accuracy will not change significantly and is within the error range. When the pruning rate reaches 80%, the accuracy will decrease to a certain extent, but it can be seen that the model size has been reduced a lot, and has been reduced to 2.72M. It is worth noting that the VGG16 model we use here is not the most original 13-layer convolution plus three-layer fully connected structure. Since the parameters of the fully connected layer are too large, we will use the GAP layer to replace the first two fully connected layers, and only retain the last fully connected layer for classification.

[0115] In addition, although the specific embodiment here is pruning the CIFAR10 dataset on the VGG16 model, this solution is not limited to this dataset and this network structure - it can be used for other network structures.

[0116] Please refer to Figure 7 , an embodiment of the present invention provides a neural network compression device 70, comprising:

[0117] The first processing module 71 is used to perform the following operations in sequence for each specified convolution layer in the trained neural network:

[0118] Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer;

[0119] Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold;

[0120] Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer;

[0121] The second processing module 72 is used to obtain a compressed neural network model after pruning all designated convolutional layers.

[0122] Optionally, the encoding layer in the first processing module 71 includes:

[0123] A global average pooling layer and two fully connected layers, the features of the convolutional layer are transformed by the global average pooling layer, and the features of the transformed convolutional layer are extracted by the fully connected layers.

[0124] Optionally, the activation layer in the first processing module 71 is used to adjust the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0125]

[0126] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0127] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0128] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0129] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0130] λ∑ αg(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0131] Optional, reference Figure 7 , the neural network compression device 70 further includes:

[0132] The third processing module 73 is used to use the training image set and the verification image set to train the compressed neural network after pruning all the designated convolutional layers for z iterations.

[0133] Through the various modules of the above-mentioned neural network compression device, it is possible to adapt to various network structures through the combination of coding layers and activation layers, has good universality, and makes full use of the characteristics of nonlinear transformation functions. By adding the activation layer parameters to the constraints of the loss function, parameter adaptation is achieved, the importance of each channel is better distinguished, and the pruning effect is optimized. It can optimize the network structure, reduce computing resources and storage resources, and provide support for running high-performance neural networks on resource-constrained devices.

[0134] Please refer to Figure 8 , an embodiment of the present invention further provides a network device 80, including a transceiver 81 and a processor 82;

[0135] The processor 82 is used to perform the following operations in sequence for each specified convolution layer in the trained neural network:

[0136] Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer;

[0137] Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold;

[0138] Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer;

[0139] The processor 82 is also used to obtain a compressed neural network model after pruning all designated convolutional layers.

[0140] Optionally, the encoding layer includes a global average pooling layer and two fully connected layers, and the features of the convolution layer are transformed by the global average pooling layer, and the features of the transformed convolution layer are extracted by the fully connected layer.

[0141] Optionally, the activation layer adjusts the sparsity of the output vector of the activation layer by a first adjustment parameter α in an activation function, and the activation function is:

[0142]

[0143] Wherein, x is the input of the neural network, and α is the first adjustment parameter.

[0144] Optionally, the value of the first adjustment parameter α is constrained by a loss function of the neural network.

[0145] Optionally, the loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α);

[0146] W is the weight of the neural network, f(x, W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x, W), y) is the loss generated by the neural network prediction;

[0147] λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

[0148] Optionally, the processor 82 is further configured to use a training image set and a verification image set to train the compressed neural network for z iterations after pruning all designated convolutional layers.

[0149] Through the various modules of the above-mentioned network devices, it is possible to adapt to various network structures through the combination of coding layers and activation layers, has good universality, and makes full use of the characteristics of nonlinear transformation functions. By adding the activation layer parameters to the constraints of the loss function, parameter adaptation is achieved, the importance of each channel is better distinguished, and the pruning effect is optimized. It can optimize the network structure, reduce computing resources and storage resources, and provide support for running high-performance neural networks on resource-constrained devices.

[0150] Please refer to Fig. 9 The embodiment of the present invention further provides a network device 90, including a processor 91, a memory 92, and a computer program stored in the memory 92 and executable on the processor 91. When the computer program is executed by the processor 91, each process of the above-mentioned neural network compression method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0151] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned neural network compression method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0154] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A neural network compression method, characterized in that: The method comprises: For each specified convolutional layer in the trained neural network, perform the following operations in sequence: A coding layer and an activation layer are added after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer; Using a training image set and a verification image set, training the neural network to be compressed for m iterations until the sparsity of the feature vector output by the activation layer meets a first threshold; According to the feature vector output by the activation layer and the second threshold, the redundant channels in the specified convolution layer are deleted to obtain the pruned specified convolution layer; Obtain a compressed neural network model after pruning all specified convolutional layers; The activation layer adjusts the sparsity of the output vector of the activation layer through the first adjustment parameter α in the activation function, and the activation function is: Wherein, x is the input of the neural network, and α is the first adjustment parameter; The value of the first adjustment parameter α is constrained by the loss function of the neural network; The loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α); W is the weight of the neural network, f(x,W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x,W),y) is the loss generated by the neural network prediction; λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

2. The neural network compression method according to claim 1, characterized in that: The encoding layer includes a global average pooling layer and two fully connected layers. The features of the convolution layer are transformed through the global average pooling layer, and the features of the transformed convolution layer are extracted through the fully connected layer.

3. The neural network compression method according to claim 1, characterized in that: Also includes: The compressed neural network after pruning all designated convolutional layers is trained for z iterations using a training image set and a validation image set.

4. A neural network compression device, characterized in that: The device comprises: The first processing module is used to perform the following operations in sequence for each specified convolutional layer in the trained neural network: Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer; Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold; Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer; The second processing module is used to obtain a compressed neural network model after pruning all designated convolutional layers; The activation layer in the first processing module is used to adjust the sparsity of the output vector of the activation layer through the first adjustment parameter α in the activation function, and the activation function is: Wherein, x is the input of the neural network, and α is the first adjustment parameter; The value of the first adjustment parameter α is constrained by the loss function of the neural network; The loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α); W is the weight of the neural network, f(x,W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x,W),y) is the loss generated by the neural network prediction; λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

5. A network device, characterized in that: including a transceiver and a processor; The processor is used to perform the following operations in sequence for each specified convolutional layer in the trained neural network: Used to add a coding layer and an activation layer respectively after the specified convolution layer to obtain a neural network to be compressed; the coding layer is used to transform and extract the features output by the specified convolution layer; the activation layer is used to normalize and map the features output by the coding layer; Used to train the neural network to be compressed for m iterations using a training image set and a verification image set until the sparsity of the feature vector output by the activation layer meets a first threshold; Used to delete redundant channels in the specified convolution layer according to the feature vector output by the activation layer and the second threshold, and obtain the pruned specified convolution layer; The processor is further used to obtain a compressed neural network model after pruning all designated convolutional layers; The activation layer adjusts the sparsity of the output vector of the activation layer through the first adjustment parameter α in the activation function, and the activation function is: Wherein, x is the input of the neural network, and α is the first adjustment parameter; The value of the first adjustment parameter α is constrained by the loss function of the neural network; The loss function is: L = ∑ (x,y) l(f(x,W),y)+λ∑ α g(α); W is the weight of the neural network, f(x,W) is the output of the neural network, y is the label of the sample, ∑ (x,y) l(f(x,W),y) is the loss generated by the neural network prediction; λ∑ α g(α) is a constraint term for the α value, and the second adjustment parameter λ is used to balance the loss generated by the neural network prediction and the constraint term for the α value, and g(α) is a monotonically decreasing function.

6. A network device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the neural network compression method as claimed in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the neural network compression method according to any one of claims 1 to 3.