Neural network pruning and application method, system, device and storage medium

Abstract

The application discloses a neural network pruning and application method, system, device and storage medium, and the related method comprises the following steps: in the training process, multiplying the output of each channel of each convolution layer for computer vision processing or video coding processing by a channel importance variable respectively, inputting to the next convolution layer, and optimizing the channel importance variable corresponding to each channel of each convolution layer in the training process; after the training is completed, removing the channels with the channel importance variable less than a set importance threshold value to obtain a pruned neural network; and using the pruned neural network for computer vision processing or video coding processing. Through the pruning of the neural network, the number of channels of the neural network is reduced, the inference speed of the neural network is accelerated, and the consumption of computing resources and storage resources can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a neural network pruning and application method, system, device and storage medium. BACKGROUND

[0002] The convolutional neural network is currently widely applied in various fields of computer vision, such as image recognition, object detection, image segmentation and the like, and is also widely applied in the field of video coding, and has achieved very good effects in the above fields.

[0003] However, the current convolutional neural network has high computational complexity, resulting in high consumption of computing resources and storage resources, and the processing speed also needs to be improved. SUMMARY

[0004] The purpose of the present application is to provide a neural network pruning and application method, system, device and storage medium, which can improve the processing speed and reduce the consumption of computing resources and storage resources.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A neural network pruning and application method comprises:

[0007] For a neural network used for computer vision processing or video coding processing, in the training process, the input training image is convoluted by the convolutional layer in the neural network, the image features output by each channel of each convolutional layer are multiplied by a current channel importance variable respectively, and then input to the next convolutional layer, and in the training process, the total loss function is constructed in combination with the processing result of the input training image and the current channel importance variable, and the channel importance variable corresponding to each channel of each convolutional layer and the parameters of the neural network are optimized;

[0008] After training, the channels with channel importance variables less than a set importance threshold are removed, and a pruned neural network is obtained;

[0009] The pruned neural network is used for computer vision processing or video coding processing.

[0010] A neural network pruning and application system comprises:

[0011] The neural network training unit is configured to train a neural network used for computer vision processing or video coding processing. During the training process, the input training image is convoluted by a convolution layer in the neural network. The image features output by each channel of each convolution layer are multiplied by a current channel importance variable and input to a next convolution layer. During the training process, a total loss function is constructed based on the processing result of the input training image and the current channel importance variable. The channel importance variable corresponding to each channel of each convolution layer and the parameters of the neural network are optimized.

[0012] The pruning unit is configured to remove a channel whose channel importance variable is less than a set importance threshold after the training of the neural network is completed, so as to obtain a pruned neural network.

[0013] The application unit is configured to use the pruned neural network to perform computer vision processing or video coding processing.

[0014] A processing device includes one or more processors, and a memory configured to store one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0016] A readable storage medium stores a computer program. When the computer program is executed by a processor, the foregoing method is implemented.

[0017] The technical solution provided by the foregoing application can be seen as follows. The neural network is pruned to reduce the number of channels of the neural network, accelerate the inference speed of the neural network, and reduce the consumption of computing resources and storage resources. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0019] Figure 1 A flowchart of a neural network pruning and application method provided by an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a neural network pruning and application method provided by an embodiment of the present application;

[0021] Figure 3 A process schematic diagram of a single convolution layer after being pruned provided by an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a neural network pruning and application system provided by an embodiment of the present application is shown in the figure;

[0023] Figure 5 A schematic diagram of a processing device provided by an embodiment of the present application is shown in the figure; DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0025] First, the terms possibly used in the present application are explained as follows:

[0026] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0027] A neural network pruning and application method, system, device and storage medium provided by the present application are described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If the specific conditions are not specified in the embodiments of the present application, the conventional conditions in the art or the recommended conditions of the manufacturer are followed.

[0028] Embodiment one

[0029] The embodiments of the present application provide a neural network pruning and application method, as shown in the figure, which mainly includes the following steps: Figure 1

[0030] Step 1, neural network training.

[0031] ​In the training process of the neural network, the input training image is convoluted by the convolution layer in the neural network, the image features output by each channel of each convolution layer are multiplied by a current channel importance variable respectively, and then input to the next convolution layer, and in the training process, a total loss function is constructed in combination with the processing result of the input training image and the current channel importance variable, and the channel importance variable corresponding to each channel of each convolution layer and the parameters of the neural network are optimized.

[0032] In the embodiment of the application, the neural network mainly refers to a neural network used for computer vision processing or video coding processing, and the specific neural network structure is not limited in the application; the processing result of the training image as described above generally refers to the final processing result, and specifically, if the neural network is used for computer vision processing, the processing result is an image recognition result, an object detection result or an image segmentation result, and if the neural network is used for video coding processing, the processing result is a reconstructed image.

[0033] In the embodiment of the application, an importance index loss term is added in the training process, and a total loss function is formed in combination with an original loss function term; the channel importance variable and the parameters of the neural network are optimized by using the total loss function. The original loss function term is calculated by using the processing result of the input training image, and can be implemented according to a conventional technology, and will not be described herein. The importance index loss term is calculated by using the current channel importance variable.

[0034] Preferably, in order to avoid the problem that some convolution layers are pruned too much and some convolution layers are pruned too little, a sparsity index is introduced, that is, the importance index loss term is calculated by using the channel importance variable and the sparsity index.

[0035] Finally, the total loss function is expressed as:

[0036] L total =L ori +λ*L prune

[0037] L prune =(g-s) 2

[0038] Wherein, L total is the total loss function, L ori is the original loss function term, L prune is the importance index loss term, g is the current channel importance variable, and the last updated value is used each time to calculate L prune in the above formula, and s is the sparsity index.

[0039] 2. Neural network pruning.

[0040] In this embodiment of the invention, after training is completed, channels whose channel importance variables are less than a set importance threshold are removed to obtain the pruned neural network. The pruning process includes: 1) setting an importance threshold based on acceleration and performance metrics; 2) comparing the channel importance variable corresponding to each channel of each convolutional layer with the set importance threshold; if the channel importance variable corresponding to a channel of a certain convolutional layer is less than the set importance threshold, then the corresponding channel in the corresponding convolutional layer is removed, and the corresponding filter in the next convolutional layer is also removed.

[0041] 3. Deploy and apply.

[0042] In this embodiment of the invention, the pruned neural network is used for computer vision processing or video encoding processing. This stage can be performed according to the original process of the neural network.

[0043] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0044] The neural network in this embodiment of the invention can be a neural network applied to various processing fields of computer vision (such as image recognition, object detection, image segmentation, etc.), or it can be a neural network applied to video encoding processing. The following description uses a neural network applied to video encoding processing as an example.

[0045] like Figure 2 As shown, the neural network used for video encoding mainly includes an encoder network and a decoder network. Its input is the image frames of the original video, and its output is the reconstructed image frames. Both the encoder and decoder networks contain several convolutional layers. During training, the output of each channel of each convolutional layer is multiplied by a channel importance variable. Assuming a single convolutional layer has 256 channels, there are 256 corresponding channel importance variables. The channel importance variable is initialized to 1, representing the highest importance, and after training, it becomes a scalar between [0,1]. During pruning, an importance threshold is set based on acceleration and performance metrics, and all channels below the importance threshold are removed.

[0046] Considering that neural networks may become unbalanced after pruning, a soft pruning index is set for different convolutional layers, so that the final number of pruned layers fluctuates around the pruning index. Specifically, this invention proposes a sparsity index as the pruning index, and at the same time, adjusts the loss function to enhance the network's understanding of sparsity during training and adjust the importance accordingly, so as to provide conditions for pruning.

[0047] In this embodiment of the invention, the sparsity index of each layer can be determined according to the complexity formula, and finally, in the original loss function L... oriThe loss term L is added with the importance index prune , to obtain a new loss function L as follows total :

[0048] L total =L ori +λ*L prune

[0049] L prune =(g-s) 2

[0050] Wherein, g is the current channel importance variable, and each time the loss function is calculated, it is the value updated last time, and s is the sparsity index.

[0051] In the embodiment of the application, the channel importance variable g is a scalar, which is a value between 0 and 1. If it is understood from the overall neural network, it can be understood as a matrix. Assuming that the neural network has 4 layers and each layer has 10 channels, the channel importance variable of the neural network can be written as a 4x10 matrix, and the element in the ith row and jth column is the channel importance variable of the 5th channel in the 1st layer. Similarly, assuming that the neural network has 4 layers and each layer has 10 channels, s can also be regarded as a matrix. Since the same layer of convolution needs to be cut together, the values in the same layer are the same. In the embodiment of the application, the computing power (i.e. complexity) of the network can be calculated according to the neural network. Through the algorithm of bisection search, a threshold value common to the entire network is found. The threshold value will cut different numbers of channels for each layer, and the number of channels cut is the sparsity index of each layer. Taking the sparsity index s = 0.5 as an example, after training with the new loss function, the value of the importance index variable will be similar to a Gaussian distribution with 0.5 as the mean value.

[0052] The process of pruning after training will be introduced below taking a single convolutional layer as an example. As shown in FIG. 1, the i-th convolutional layer on the left has N channels. Each channel will be multiplied by a channel importance variable during training. After training, the channel importance variable presents values such as 0.3, 0.7, 0.8 and 1.0. Assuming that the importance threshold is set to 0.5 according to the acceleration index and the performance index, channels 3 and 4 below 0.5 will be cut off, and finally only N-2 channels are retained. Figure 3

[0053] After determining the channels to be removed through the above method, the same channels in the neural network can be removed by writing a script, and the corresponding filters in the next convolutional layer are also removed, to obtain the pruned neural network. Then, the pruned neural network can perform processing of related video data (or image data) according to the original working process.

[0054] ​Those skilled in the art can clearly understand from the description of the foregoing embodiments that the foregoing embodiments can be implemented by software, or can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the foregoing embodiments can be embodied in the form of a software product, which can be stored in a nonvolatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment of the application.

[0055] Embodiment two

[0056] The application further provides a neural network pruning and application system, which is mainly used for implementing the method provided in the foregoing embodiments, as shown in the accompanying drawings. Figure 4 The system mainly includes:

[0057] a neural network training unit, configured to train a neural network used for computer vision processing or video coding processing, and in a training process, perform convolution processing on an input training image through a convolution layer in the neural network, multiply image features output by each channel of each convolution layer by a current channel importance variable, and then input the image features to a next convolution layer, and in the training process, combine a processing result of the input training image and the current channel importance variable to construct a total loss function, and optimize the channel importance variable corresponding to each channel of each convolution layer and parameters of the neural network;

[0058] a pruning unit, configured to remove a channel whose channel importance variable is less than a set importance threshold after the training of the neural network is completed, and obtain a pruned neural network;

[0059] an application unit, configured to perform computer vision processing or video coding processing by using the pruned neural network.

[0060] In the embodiment of the application, an importance index loss term is added in a training process, and a total loss function is formed in combination with an original loss function term; the channel importance variable and the parameters of the neural network are optimized by using the total loss function; the original loss function term is calculated by using a processing result of an input training image, and the importance index loss term is calculated by using a current channel importance variable.

[0061] In the embodiment of the application, the total loss function is expressed as:

[0062] L total = L ori + λ * L prine

[0063] L prune = (g-s) 2

[0064] Among them, L total Let L be the total loss function. ori For the original loss function term, L prune , where g is the current channel importance variable and s is the sparsity index;

[0065] In this embodiment of the invention, after training is complete, removing channels whose channel importance variables are less than a set importance threshold to obtain the pruned neural network includes:

[0066] Importance thresholds are set based on acceleration and performance metrics;

[0067] The channel importance variable corresponding to each channel of each convolutional layer is compared with the set importance threshold. If the channel importance variable corresponding to a certain channel of a certain convolutional layer is less than the set importance threshold, the corresponding channel in the corresponding convolutional layer is removed, and the corresponding filter in the next convolutional layer is also removed.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0069] Example 3

[0070] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0071] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0072] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0073] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0074] The output device can be a display terminal;

[0075] The memory can be a random access memory (RAM) or a non-volatile memory such as a disk memory.

[0076] Embodiment Four

[0077] The application further provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0078] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0079] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for neural network pruning and application, characterized in that, The application relates to a neural network for computer vision processing or video coding processing. During training, the input training image is convoluted through a convolutional layer in the neural network, the image features output by each channel of each convolutional layer are multiplied by a current channel importance variable, and the multiplied results are input into a next convolutional layer; during training, a total loss function is constructed by combining the processing results of the input training image and the current channel importance variable, and the channel importance variable corresponding to each channel of each convolutional layer and the parameters of the neural network are optimized. After training, channels with channel importance variables less than a set importance threshold are removed to obtain a pruned neural network; the importance threshold is set according to an acceleration index and a performance index; the channel importance variable corresponding to each channel of each convolutional layer is compared with the set importance threshold; if the channel importance variable corresponding to a channel of a convolutional layer is less than the set importance threshold, the corresponding channel in the convolutional layer is removed, and the corresponding filter in the next convolutional layer is removed. The pruned neural network is used for computer vision processing or video coding processing.

2. The method of claim 1, wherein, During training, an importance index loss item is added, and a total loss function is formed by combining an original loss function item; the channel importance variable and the parameters of the neural network are optimized by using the total loss function; the original loss function item is calculated by using the processing results of the input training image, and the importance index loss item is calculated by using the current channel importance variable.

3. The method of claim 2, wherein the method further comprises: The total loss function is expressed as: L total = L ori + λ*L prune L prune = (g - s) 2 wherein L total is the total loss function, L ori is the original loss function term, L prune is the importance indicator loss term, g is the current channel importance variable, and s is the sparsity indicator.

4. A neural network pruning and application system, comprising: The application relates to a neural network for computer vision processing or video coding processing. During training, the input training image is convoluted through a convolutional layer in the neural network, the image features output by each channel of each convolutional layer are multiplied by a current channel importance variable, and the multiplied results are input into a next convolutional layer; during training, a total loss function is constructed by combining the processing results of the input training image and the current channel importance variable, and the channel importance variable corresponding to each channel of each convolutional layer and the parameters of the neural network are optimized. After training, channels with channel importance variables less than a set importance threshold are removed to obtain a pruned neural network; the importance threshold is set according to an acceleration index and a performance index; the channel importance variable corresponding to each channel of each convolutional layer is compared with the set importance threshold; if the channel importance variable corresponding to a channel of a convolutional layer is less than the set importance threshold, the corresponding channel in the convolutional layer is removed, and the corresponding filter in the next convolutional layer is removed. The pruned neural network is used for computer vision processing or video coding processing.

5. The neural network pruning and application system of claim 4, wherein, An importance index loss item is added in the training process, and combined with an original loss function item to form a total loss function; the total loss function is used to optimize the channel importance variable and the parameters of the neural network; wherein the original loss function item is calculated by using the processing result of the input training image, and the importance index loss item is calculated by using the current channel importance variable.

6. The neural network pruning and application system of claim 5, wherein, The total loss function is expressed as: L total = L ori + λ * L prune L prune = (g - s) 2 wherein L total is the total loss function, L ori is the original loss function term, L prune is the importance indicator loss term, g is the current channel importance variable, and s is the sparsity indicator.

7. A processing device, characterized by The method comprises: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-3.

8. A readable storage medium, storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-3.

Images