Pruning method, device and equipment of neural network and medium

Abstract

Embodiments of the present disclosure relate to a pruning processing method, device and equipment of a neural network and a medium, wherein the method comprises: obtaining a first sparse constraint matrix corresponding to a weight matrix of a layer to be pruned of an original neural network based on a first pruning manner, and obtaining a first sparse neural network after pruning processing according to the first sparse constraint matrix; obtaining a second sparse constraint matrix corresponding to a weight matrix of a layer to be pruned of the first sparse neural network based on a second pruning manner, and obtaining a second sparse neural network after pruning processing according to the second sparse constraint matrix. It can be seen that the embodiments of the present disclosure can adaptively prune the original neural network, improve the efficiency of pruning processing, and make the pruning processing reasonable, better balance the actual speed benefit of pruning and the accuracy of the neural network after pruning, improve the pruning performance, and at the same time occupy less mobile terminal computing power resources, storage resources and the like, improve the performance of data processing of the mobile terminal.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for pruning neural networks. Background Technology

[0002] The application of neural network-based artificial intelligence technology on mobile terminals has led to the rapid development of smart mobile terminals, meeting various application needs. Key technologies include data processing based on trained neural network model data in application areas such as video processing, speech recognition, image recognition and understanding, and game vision. Given the limited computing resources of mobile terminals and considering that most convolutional neural networks have a certain degree of parameter redundancy, pruning is used to remove redundant convolutional kernels or neurons on convolutional kernels in each layer of the neural network, resulting in neural networks with lower computational and storage requirements on mobile terminals.

[0003] To further improve the computing performance of mobile terminals, the acceleration advantages of structured pruning and the performance advantages of unstructured pruning were combined, and two pruning modes with different granularities were applied simultaneously in the pruning algorithm. However, in specific experiments, it was found that the computation process of applying two different pruning modes simultaneously resulted in difficulties in sparse matrix convergence due to computational interference, leading to poor actual pruning performance and low pruning efficiency. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, device, and medium for pruning neural networks.

[0005] In a first aspect, embodiments of this disclosure provide a method for pruning neural networks, the method comprising:

[0006] The first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned is obtained based on the first pruning method, and the first sparse neural network after pruning is obtained based on the first sparse constraint matrix.

[0007] The second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network is obtained based on the second pruning method, and the second sparse neural network after pruning is obtained based on the second sparse constraint matrix.

[0008] In one optional implementation, it further includes:

[0009] Two pruning methods, a first pruning method and a second pruning method, are executed sequentially in series. The first pruning method is a connection pruning method, and the second pruning method is a pattern pruning method; or...

[0010] The first pruning method is a pattern pruning method, and the second pruning method is a connection pruning method.

[0011] In one optional implementation, the first pruning method is a connection pruning method, and the step of obtaining the first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned based on the first pruning method includes:

[0012] The initial constraint matrix of the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned is obtained based on the alternating direction multiplier method.

[0013] The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the connection pruning method. After a preset number of iterations of training, the first candidate neural network with updated weights and the corresponding first updated weight matrix are obtained.

[0014] The first sparse constraint matrix corresponding to the first updated weight matrix is ​​obtained based on the alternating direction multiplier method.

[0015] In one optional implementation, obtaining the pruned first sparse neural network based on the first sparse constraint matrix includes:

[0016] The first sparse neural network is obtained by multiplying the first update weight matrix corresponding to the first candidate neural network by the first sparse constraint matrix.

[0017] In one optional implementation, the second pruning method is a pattern pruning method, and obtaining the second sparse constraint matrix corresponding to the weight matrix of the first sparse neural network layer to be pruned based on the second pruning method includes:

[0018] The initial constraint matrix of the pattern pruning method corresponding to the initial weight matrix of the first sparse neural network to be pruned layer is obtained based on the alternating direction multiplier method.

[0019] The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the pattern pruning method. After a preset number of iterations of training, the second candidate neural network with updated weights and the corresponding second updated weight matrix are obtained.

[0020] The second sparse constraint matrix corresponding to the second updated weight matrix is ​​obtained based on the alternating direction multiplier method.

[0021] In one optional implementation, obtaining the pruned second sparse neural network based on the second sparse constraint matrix includes:

[0022] The second candidate neural network is obtained by multiplying the second update weight matrix by the second sparse constraint matrix.

[0023] In one optional implementation, the feature is that it further includes:

[0024] The parameters of the second sparse neural network are trained based on sample data from the application scenario to generate a target neural network, which then processes the input data to obtain output data. The input data includes one or more combinations of multimedia data, audio data, video data, and image data.

[0025] In one optional implementation, the step of training the parameters of the second sparse neural network to generate the target neural network based on sample data from the application scenario includes:

[0026] Obtain the face sample images used to train the original neural network;

[0027] Obtain the rendered image of the key parts corresponding to the face sample image;

[0028] During the training of the second sparse neural network based on the face sample image and the key part rendering image, the parameters of the second sparse neural network are obtained based on the preset objective function constraint to obtain the target neural network, so as to obtain the target image by rendering the test image based on the target neural network.

[0029] Secondly, embodiments of this disclosure also provide a neural network pruning processing apparatus, the apparatus comprising:

[0030] The first acquisition module is used to acquire the first sparse constraint matrix corresponding to the weight matrix of the original neural network to be pruned layer based on the first pruning method, and to acquire the first sparse neural network after pruning based on the first sparse constraint matrix.

[0031] The second acquisition module is used to acquire the second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network based on the second pruning method, and to acquire the second sparse neural network after pruning based on the second sparse constraint matrix.

[0032] Thirdly, this disclosure provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the above-described method.

[0033] Fourthly, this disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0034] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.

[0035] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0036] The first sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the original neural network is obtained based on the first pruning method, and the pruned first sparse neural network is obtained based on the first sparse constraint matrix. The second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network is obtained based on the second pruning method, and the pruned second sparse neural network is obtained based on the second sparse constraint matrix. It is evident that this embodiment can adaptively prune the original neural network, improving the efficiency of the pruning process and making the pruning process more rational. It better balances the actual speed gain of pruning and the accuracy of the neural network after pruning, improving pruning performance. Simultaneously, it avoids mutual interference between the first and second pruning methods, making convergence easier during the pruning process. When the second sparse neural network obtained through this pruning method runs on a mobile terminal, it consumes less computing and storage resources, thereby improving the performance of the mobile terminal in data processing applications such as video processing, speech recognition, and image recognition and understanding. Attached Figure Description

[0037] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0038] Figure 1 A flowchart illustrating a neural network pruning method provided in an embodiment of this disclosure;

[0039] Figure 2 A flowchart illustrating another neural network pruning method provided in this embodiment of the disclosure;

[0040] Figure 3 A schematic diagram of a process for obtaining a first sparse neural network is provided for an embodiment of this disclosure;

[0041] Figure 4A schematic diagram of a process for obtaining a second sparse neural network is provided for an embodiment of this disclosure;

[0042] Figure 5 A schematic diagram of the structure of a neural network pruning device provided in an embodiment of this disclosure;

[0043] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0044] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0045] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0046] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0047] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0048] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0049] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0050] To address the aforementioned problems, this disclosure provides a method for pruning neural networks, which will be described below with reference to specific embodiments.

[0051] Figure 1 This is a flowchart illustrating a neural network pruning method provided in an embodiment of this disclosure. The method can be executed by a neural network pruning device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0052] Step 101: Obtain the first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned based on the first pruning method, and obtain the first sparse neural network after pruning based on the first sparse constraint matrix.

[0053] To ensure model performance, the original neural network may contain a certain amount of redundant parameters. Pruning can remove redundant convolutional kernels (i.e., structured pruning) or neurons on convolutional kernels (i.e., unstructured pruning) from each convolutional layer while maintaining accuracy, resulting in a "slimmed-down model" with lower computational and storage resource consumption. This accelerates the inference process of the neural network and facilitates its edge deployment. However, some pruning methods using alternating direction method of multiplies (ADMM) on the original neural network can encounter convergence difficulties. In this embodiment, the convergence difficulty in the pruning process is addressed, specifically including:

[0054] In this embodiment, the original neural network is a neural network model that needs to be pruned. The neural network model can be set according to the application scenario and / or user needs, etc. This embodiment does not impose any restrictions. The original neural network can be a neural network model that has been trained and optimized by L2 regularization.

[0055] In this embodiment, a first pruning method and a second pruning method with different pruning granularities can be obtained according to the coarseness of the pruning granularity. By sequentially executing the first pruning method and the second pruning method, the original neural network is pruned. The coarseness relationship between the pruning granularity of the first pruning method and the second pruning method is not limited in this embodiment. For example, the pruning granularity of the first pruning method can be smaller than that of the second pruning method, or the pruning granularity of the first pruning method can be larger than that of the second pruning method.

[0056] For example, if ADMM is used to prune the original neural network, the first pruning method can be defined as connectivity pruning, and the second pruning method as pattern pruning. Connectivity pruning prunes the channels of the convolutional kernel. If it is determined through connectivity pruning that channel A needs to be pruned, then the computation between channel A and the corresponding channel of the input feature map is pruned. Pattern pruning prunes the neurons in the channels of the convolutional kernel. This pattern pruning method has corresponding preset patterns. When performing pattern pruning, pruning is performed according to the preset pattern. For example, when a channel includes 9 neurons, and these 9 neurons are represented as a 3*3 rectangular block, the preset pattern can be to retain 4 of the 9 neurons, including the neuron located at the center of the rectangular block.

[0057] In one optional implementation, two pruning methods, a first pruning method and a second pruning method, can be executed sequentially in series. Taking the connection of the pruning method and the pattern pruning method as an example, there are various methods for this sequential execution, such as:

[0058] In one optional implementation, the first pruning method is a connection pruning method, and the second pruning method is a pattern pruning method. In this embodiment, the connection pruning method with coarser granularity is performed first, followed by the pattern pruning method with finer granularity.

[0059] In another optional implementation, the first pruning method is a pattern pruning method, and the second pruning method is a connection pruning method. In this embodiment, the pattern pruning method with finer granularity is performed first, followed by the connection pruning method with coarser granularity.

[0060] When pruning the original neural network using methods such as ADMM, initialization can be performed according to application requirements to obtain the weight matrix of the pruned layer. This weight matrix represents the weight of each element in the pruned layer. Similarly, in this embodiment, a first sparse constraint matrix corresponding to the weight matrix can be obtained based on the first pruning method. The original neural network is then trained using this first sparse constraint matrix to obtain the first sparse neural network after pruning.

[0061] Step 102: Obtain the second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network based on the second pruning method, and obtain the second sparse neural network after pruning based on the second sparse constraint matrix.

[0062] Furthermore, in this embodiment, initialization can be performed according to the application scenario requirements of the first sparse neural network pruning process, thereby obtaining the weight matrix of the first sparse neural network pruning layer. This weight matrix can represent the weight of each element in the first sparse neural network pruning layer. Then, based on the second pruning method, a second sparse constraint matrix corresponding to the weight matrix is ​​obtained, and the first sparse neural network is sparsely trained using the second sparse constraint matrix as a constraint to obtain a second sparse neural network after pruning the first sparse neural network.

[0063] In summary, the neural network pruning method of this disclosure obtains a first sparse constraint matrix corresponding to the weight matrix of the original neural network's pruning layer based on a first pruning method, and obtains a pruned first sparse neural network based on the first sparse constraint matrix; it then obtains a second sparse constraint matrix corresponding to the weight matrix of the first sparse neural network's pruning layer based on a second pruning method, and obtains a pruned second sparse neural network based on the second sparse constraint matrix. It is evident that this disclosure can adaptively prune the original neural network, improving the efficiency of the pruning process and making it more rational. It better balances the actual speed gain of pruning with the accuracy of the neural network after pruning, improving pruning performance. Simultaneously, it avoids mutual interference between the first and second pruning methods, making convergence easier during the pruning process. When the second sparse neural network obtained through this pruning method runs on a mobile terminal, it consumes less computing and storage resources, thereby improving the performance of the mobile terminal in data processing applications such as video processing, speech recognition, and image recognition and understanding.

[0064] In related technologies, pruning processes based on algorithms such as ADMM are indirect pruning processes based on sparse constraints. In this process, sparse constraint matrices for connection pruning and pattern pruning are obtained, and these two types of sparse constraints are multiplied to obtain a sparsification matrix for sparsifying the original neural network. Practical experience has shown that the convergence of sparse constraints in this ADMM-based pruning process is poor. Further experiments and comparative analysis revealed that one reason for this problem is the simultaneous execution of connection pruning and pattern pruning, which affects the convergence of pattern pruning. In this embodiment, the convergence of the pruning process is improved by sequentially executing connection pruning and pattern pruning.

[0065] Specifically, based on the above embodiments, Figure 2 This is a flowchart illustrating another neural network pruning method provided in an embodiment of this disclosure. In this embodiment, the first pruning method is connection pruning, and the second pruning method is pattern pruning, such as... Figure 2As shown, the method includes the following steps:

[0066] Step 201: Obtain the initial constraint matrix of the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned based on the alternating direction multiplier method.

[0067] In this embodiment, the pruning process of the original neural network can be decomposed into connection pruning and pattern pruning, which are executed sequentially in series. Similar to the previous embodiments, the original neural network can be a high-performance neural network model trained with L2 regularization.

[0068] In this embodiment, the original neural network layer to be pruned has a corresponding initial weight matrix. Based on the application requirements of the current scenario, initialization is performed using the alternating direction multiplier method to obtain an initial constraint matrix for the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned. In an optional implementation, this initial constraint matrix for the connection pruning method includes two matrices, one of which can represent the initial connection pruning mask.

[0069] Step 202: Train the first candidate neural network with updated weights and the corresponding first updated weight matrix based on the preset original loss function and the initial constraint matrix of the connection pruning method as the target loss function. After a preset number of iterations of training, obtain the first candidate neural network with updated weights and the corresponding first updated weight matrix.

[0070] In this embodiment, a corresponding original loss function is preset for the original neural network, and the target loss function can be determined by calculation based on the original loss function and the initial constraint matrix of the connection pruning method.

[0071] Then, the original neural network is trained using an initial constraint matrix with connection pruning, and the target loss function is used as the constraint condition for the training process. After a preset M (M is a positive integer) iterations, the first candidate neural network with updated weights and the corresponding first updated weight matrix are obtained. Understandably, this iterative process guides the weight matrix towards optimization towards a sparser matrix.

[0072] Step 203: Obtain the first sparse constraint matrix corresponding to the first updated weight matrix based on the alternating direction multiplier method.

[0073] Furthermore, the first updated weight matrix is ​​calculated using the alternating direction multiplier method to obtain the first sparse constraint matrix corresponding to the first updated weight matrix. The first sparse constraint matrix is ​​obtained by updating the initial constraint matrix of the connection pruning method. Therefore, the first sparse constraint matrix may include two matrices, one of which can represent the target connection pruning mask.

[0074] Step 204: Multiply the first update weight matrix corresponding to the first candidate neural network by the first sparse constraint matrix to obtain the first sparse neural network.

[0075] In one optional implementation, after obtaining the first update weights corresponding to the first candidate neural network, the first update weight matrix of each layer of the first candidate neural network is multiplied by the pruning mask in the first sparse constraint matrix to obtain the first sparse neural network.

[0076] Figure 3 This is a schematic diagram of a process for obtaining a first sparse neural network according to an embodiment of the present disclosure, such as... Figure 3 As shown, in one optional implementation, the original neural network model is model_original, and the initial weight matrix W is obtained by initialization based on the alternating direction multiplier method and the application scenario requirements. 0 The corresponding initial constraint matrix Y of the connection pruning method 0 and V 0 According to the original loss function L original Preset second coefficient α2, initial weight matrix W 0 And the initial constraint matrix Y of the connection pruning method 0 and V 0 Determine the target loss function L, where:

[0077] L = Lo rig inal+α2∑||W i t -Y i t +V i t ||.

[0078] In the first iteration, W i t =W 0 Y i t =Y 0 V i t =V 0Then, using the target loss function L, model_original is trained based on the current constraint matrix, iterating M times (M is a positive integer). After each iteration, the subscripts i of W, Y, and V are incremented by 1 until the subscripts of W, Y, and V equal M. After M iterations, the neural network weight matrix is ​​updated once, the subscripts of W, Y, and V are reset, and the superscripts of W, Y, and V are incremented by 1. Then, the constraint matrix of the connection pruning method is updated based on the latest weight matrix, and the current neural network model is trained based on this constraint matrix. This process is repeated K times (K is a positive integer) to obtain the first sparse constraint matrix. This first sparse constraint matrix includes the matrix Y representing the target connection pruning mask. K At the end of the iteration, the first candidate neural network model_sparseA is obtained. The weight matrix of each layer of model_sparseA is multiplied by the corresponding Y. K The first sparse neural network model_sparseB is obtained.

[0079] Step 205: Obtain the initial constraint matrix of the pattern pruning method corresponding to the initial weight matrix of the first sparse neural network layer to be pruned based on the alternating direction multiplier method.

[0080] In this embodiment, the first sparse neural network layer to be pruned has a corresponding initial weight matrix. Based on the requirements of the current application scenario, the alternating direction multiplier method should be initialized to obtain an initial constraint matrix for the pattern pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned. In an optional implementation, this initial constraint matrix for the pattern pruning method may include two matrices, one of which can represent the initial pattern pruning mask.

[0081] Step 206: Train the second candidate neural network with updated weights and the corresponding second updated weight matrix based on the preset original loss function and the initial constraint matrix of the pattern pruning method as the target loss function. After a preset number of iterations of training, obtain the second candidate neural network with updated weights and the corresponding second updated weight matrix.

[0082] In this embodiment, a corresponding original loss function is preset for the first sparse neural network, and the target loss function can be determined by calculation based on the original loss function and the initial constraint matrix of the pattern pruning method.

[0083] Furthermore, a first sparse neural network is trained based on the initial constraint matrix of the pattern pruning method, and the target loss function is used as the constraint condition for the training process. After a preset N (N is a positive integer) iterations of training, a second candidate neural network with updated weights and the corresponding second updated weight matrix are obtained. Understandably, this iterative process guides the weight matrix towards a sparse matrix. N can be the same as or different from M in the above steps; this embodiment does not impose any restrictions.

[0084] Step 207: Obtain the second sparse constraint matrix corresponding to the second updated weight matrix based on the alternating direction multiplier method.

[0085] Furthermore, the second updated weight matrix is ​​calculated using the alternating direction multiplier method to obtain the second sparse constraint matrix corresponding to the second updated weight matrix. This second sparse constraint matrix is ​​obtained by updating the initial constraint matrix of the pattern pruning method. In an optional implementation, the second sparse constraint matrix may include two matrices, one of which can represent the target pattern pruning mask.

[0086] Step 208: Multiply the second update weight matrix corresponding to the second candidate neural network by the second sparse constraint matrix to obtain the second sparse neural network.

[0087] After obtaining the second updated weights corresponding to the second candidate neural network, the second updated weight matrix of each layer of the second candidate neural network is multiplied by the pruning mask in the second sparse constraint matrix to obtain the second sparse neural network.

[0088] Figure 4 This is a schematic diagram of a process for obtaining a second sparse neural network according to an embodiment of the present disclosure, such as... Figure 4 As shown, in one optional implementation, the initial weight matrix W is obtained by initialization based on the alternating direction multiplier method and the application scenario requirements. 0′ The corresponding pattern pruning method's initial constraint matrix Z 0 and U 0 According to the original loss function L original ′、Preset first coefficient α1、Initial weight matrix W 0′ And the initial constraint matrix Z of the pattern pruning method 0 and U 0 Determine the target loss function L′, where:

[0089]

[0090] In the first iteration, W i t′ =W 0′ , Then, using the target loss function L, a first sparse neural network is trained based on the current constraint matrix, iterating N times (N is a positive integer). After each iteration, the subscripts i of W, Z, and U are incremented by 1 until the subscripts of W, Z, and U equal N. After N iterations, the neural network weight matrix is ​​updated once, resetting the subscripts of W, Z, and U and incrementing the superscripts of W, Z, and U by 1. Then, the constraint matrix of the pruning mode is updated based on the latest weight matrix, and the current neural network model is trained based on this constraint matrix. This process is repeated L times (L is a positive integer) to obtain a second sparse constraint matrix. Taking the case where L is the same as K in the above steps as an example, this second sparse constraint matrix includes a matrix Z representing the target connection pruning mask. K At the end of the iteration, a second candidate neural network, model_sparseC, is obtained. The weight matrix of each layer of model_sparseC is then multiplied by the corresponding Z. K The second sparse neural network model_sparseD is obtained.

[0091] Step 209: Train the second sparse neural network to obtain the target neural network.

[0092] After processing with connection pruning and pattern pruning methods, in order to improve the accuracy of the neural network model, the second sparse neural network is trained with training data to restore accuracy, thereby obtaining the target neural network.

[0093] In another optional implementation, the pruning process of the original neural network can be decomposed into a pattern pruning method and a connection pruning method executed sequentially. First, the pattern pruning method is performed on the original neural network to obtain a first sparse neural network. Then, the connection pruning method is performed on the first sparse neural network to obtain a second sparse neural network. Finally, the second sparse neural network is trained to obtain the target neural network. The specific execution process is similar to that in the above embodiment and will not be described in detail here.

[0094] Furthermore, convergence tests can be performed on two cases based on preset experimental configurations: executing connection pruning first and then pattern pruning, and executing pattern pruning first and then connection pruning. The experimental configuration can be set according to analysis requirements, etc., and this embodiment does not impose any restrictions. An example is illustrated below:

[0095] In one optional implementation, the preset experimental configuration uses the CIFAR-10 dataset and the ResNet20 network. In this implementation, a higher score indicates better convergence. The convergence score of the pruning method that performs connection pruning first and then pattern pruning is 91.05, while the convergence score of the pruning method that performs pattern pruning first and then connection pruning is 90.82. It can be seen that the pruning method that performs connection pruning first and then pattern pruning has better convergence. This is because the sparsity constraint pruning in pattern pruning has poor convergence. Performing connection pruning first effectively reduces the complexity of the model and the solution space. Then, performing pattern pruning, which prunes less data, makes it easier to converge.

[0096] In another optional implementation, the preset experimental configuration uses peak signal-to-noise ratio (PSNR) as the evaluation metric and super-resolution validation set for performance evaluation. In this implementation, a higher PSNR score indicates better convergence. The PSNR score for the pruning method that simultaneously performs connection pruning and pattern pruning is 27.48, the PSNR score for the pruning method that performs connection pruning first and then pattern pruning is 27.53, and the PSNR score for the pruning method that performs pattern pruning first and then connection pruning is 27.51. It can be seen that performing the pruning methods sequentially has a significant convergence advantage, and the pruning method that performs connection pruning first and then pattern pruning has better convergence.

[0097] In summary, the neural network pruning method of this disclosure effectively reduces the number of parameters and computational load of the neural network while ensuring the accuracy of the neural network model. Furthermore, by first performing connection pruning, which is more likely to converge, the complexity and solution space of the neural network are effectively reduced. Then, by performing pattern pruning, the amount of pruning in pattern pruning is reduced, resulting in better convergence of the neural network. This significantly improves the convergence performance of the pruning process and makes the pruning performance more stable.

[0098] Based on the above embodiments, the second sparse neural network can be further trained to improve the accuracy of the neural network after pruning, specifically including:

[0099] The parameters of the second sparse neural network are trained based on the sample data of the application scenario to generate the target neural network. The target neural network then processes the input data to obtain the output data. The input data includes one or more combinations of multimedia data, audio data, video data, and image data.

[0100] In this embodiment, the application scenarios include, but are not limited to, any one of video processing language recognition, image recognition and understanding, and game vision. The sample data is data that meets the training requirements of the application scenario. This sample data can be data used to train the original neural network, or other data that meets the requirements of the application scenario; this embodiment does not impose any limitations. The sample data is used to train the parameters of the second sparse neural network obtained through pruning, thereby improving the accuracy of the neural network and generating the corresponding target neural network. This target neural network can process the input data in the application scenario and generate corresponding output data. It is understood that different application scenarios have corresponding input data, which includes one or more combinations of multimedia data, audio data, video data, and image data.

[0101] Taking the application scenario of rendering key facial features as an example, in this application scenario, the input data of the target neural network includes image data. The steps above, which train the parameters of the second sparse neural network to generate the target neural network based on the sample data of the application scenario, include:

[0102] First, obtain the face sample images used to train the original neural network.

[0103] Furthermore, the rendering images of key parts corresponding to the face sample images are obtained.

[0104] In this embodiment, face sample images and their corresponding key part rendering images used to train the original neural network can be used as sample data, and the sample data can be used to train the second sparse neural network to improve accuracy.

[0105] Furthermore, during the training of the second sparse neural network based on the face sample image and the rendered image of the key part, the parameters of the second sparse neural network are obtained based on the preset objective function constraint to obtain the target neural network, so as to obtain the target image by rendering the test image based on the target neural network.

[0106] In this embodiment, a preset objective function is used as a loss function to constrain the training of the second sparse neural network. This objective function can be selected according to the application scenario requirements, and this embodiment does not impose any restrictions. Furthermore, face sample images can be used as input data for the second sparse neural network, and key feature rendered images can be used as the output images of the second sparse neural network. The parameters of the second sparse neural network are trained based on the constraints of the objective function. After training, the corresponding target neural network is obtained. This target neural network can render the input test images to generate the corresponding target images.

[0107] In summary, the neural network pruning method of this disclosure can prune the original neural network and perform further precision training in application scenarios such as video processing language recognition, image recognition and understanding, and game vision, thereby reducing the number of neural network parameters and computing power consumption in the application scenario, while ensuring the accuracy of the neural network.

[0108] Figure 5 This is a schematic diagram of a neural network pruning processing device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and is generally integrated into an electronic device.

[0109] like Figure 5 As shown, the device includes:

[0110] The first acquisition module 501 is used to acquire the first sparse constraint matrix corresponding to the weight matrix of the original neural network to be pruned layer based on the first pruning method, and to acquire the first sparse neural network after pruning based on the first sparse constraint matrix.

[0111] The second acquisition module 502 is used to acquire the second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network based on the second pruning method, and to acquire the second sparse neural network after pruning based on the second sparse constraint matrix.

[0112] In one optional implementation, it further includes:

[0113] The decomposition module is used to set two pruning methods, a first pruning method and a second pruning method, to be executed sequentially in series. The first pruning method is a connection pruning method, and the second pruning method is a pattern pruning method; or...

[0114] The first pruning method is a pattern pruning method, and the second pruning method is a connection pruning method.

[0115] In one optional implementation, the first acquisition module 501 is configured to:

[0116] The initial constraint matrix of the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned is obtained based on the alternating direction multiplier method.

[0117] The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the connection pruning method. After a preset number of iterations of training, the first candidate neural network with updated weights and the corresponding first updated weight matrix are obtained.

[0118] The first sparse constraint matrix corresponding to the first updated weight matrix is ​​obtained based on the alternating direction multiplier method.

[0119] In an optional implementation, the first acquisition module 501 is further configured to:

[0120] The first sparse neural network is obtained by multiplying the first update weight matrix corresponding to the first candidate neural network by the first sparse constraint matrix.

[0121] In one optional implementation, the second acquisition module 502 is configured to:

[0122] The initial constraint matrix of the pattern pruning method corresponding to the initial weight matrix of the first sparse neural network to be pruned layer is obtained based on the alternating direction multiplier method.

[0123] The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the pattern pruning method. After a preset number of iterations of training, the second candidate neural network with updated weights and the corresponding second updated weight matrix are obtained.

[0124] The second sparse constraint matrix corresponding to the second updated weight matrix is ​​obtained based on the alternating direction multiplier method.

[0125] In one optional implementation, the second acquisition module 502 includes:

[0126] The second candidate neural network is obtained by multiplying the second update weight matrix by the second sparse constraint matrix.

[0127] In one optional implementation, it further includes:

[0128] The training module is used to train the parameters of the second sparse neural network to generate a target neural network based on sample data of the application scenario, so as to process the input data according to the target neural network to obtain output data. The input data includes one or more combinations of multimedia data, audio data, video data, and image data.

[0129] In one optional implementation, the training module is used for:

[0130] Obtain the face sample images used to train the original neural network;

[0131] Obtain the rendered image of the key parts corresponding to the face sample image;

[0132] During the training of the second sparse neural network based on the face sample image and the rendered image of the key parts, the parameters of the second sparse neural network are obtained based on the preset objective function constraints to obtain the target neural network. The test image is then rendered using the target neural network to obtain the target image. The neural network pruning processing device provided in this disclosure can execute the neural network pruning processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method.

[0133] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the neural network pruning method described in this disclosure.

[0134] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, implement the neural network pruning method described in this disclosure.

[0135] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0136] The following is a detailed reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device 600 in the embodiments of this disclosure. The electronic device 600 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0137] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0138] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0139] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the neural network pruning method of embodiments of this disclosure.

[0140] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0141] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0143] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes these programs, the electronic device: obtains a first sparse constraint matrix corresponding to the weight matrix of the original neural network's pruned layer based on a first pruning method, and obtains a pruned first sparse neural network based on the first sparse constraint matrix; obtains a second sparse constraint matrix corresponding to the weight matrix of the first sparse neural network's pruned layer based on a second pruning method, and obtains a pruned second sparse neural network based on the second sparse constraint matrix. It is evident that the embodiments of this disclosure can adaptively prune the original neural network, improving the efficiency of the pruning process and rationalizing it. This better balances the actual speed gain of pruning with the accuracy of the neural network after pruning, improving pruning performance. Simultaneously, it avoids mutual interference between the first and second pruning methods, making convergence easier during the pruning process. When the second sparse neural network obtained through this pruning method runs on a mobile terminal, it consumes less computing and storage resources, thereby improving the performance of the mobile terminal in data processing applications such as video processing, speech recognition, and image recognition and understanding.

[0144] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0147] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0148] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0150] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multimodal and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0151] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for pruning neural networks, characterized in that, include: The first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned is obtained based on the first pruning method, and the first sparse neural network after pruning is obtained based on the first sparse constraint matrix; wherein, the first pruning method is a connection pruning method. The second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network is obtained based on the second pruning method, and the pruned second sparse neural network is obtained based on the second sparse constraint matrix; wherein, the second pruning method is a pattern pruning method, the first pruning method and the second pruning method are executed in series, the second sparse neural network is trained to generate the target neural network, and the input data of the target neural network includes one or more combinations of audio data, video data and image data; The step of obtaining the first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned based on the first pruning method includes: The initial constraint matrix of the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned is obtained based on the alternating direction multiplier method. The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the connection pruning method. After a preset number of iterations of training, the first candidate neural network with updated weights and the corresponding first updated weight matrix are obtained. The first sparse constraint matrix corresponding to the first updated weight matrix is ​​obtained based on the alternating direction multiplier method.

2. The method according to claim 1, characterized in that, The step of obtaining the pruned first sparse neural network based on the first sparse constraint matrix includes: The first sparse neural network is obtained by multiplying the first update weight matrix corresponding to the first candidate neural network by the first sparse constraint matrix.

3. The method according to claim 2, characterized in that, The step of obtaining the second sparse constraint matrix corresponding to the weight matrix of the first sparse neural network layer to be pruned based on the second pruning method includes: The initial constraint matrix of the pattern pruning method corresponding to the initial weight matrix of the first sparse neural network to be pruned layer is obtained based on the alternating direction multiplier method. The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the pattern pruning method. After a preset number of iterations of training, the second candidate neural network with updated weights and the corresponding second updated weight matrix are obtained. The second sparse constraint matrix corresponding to the second updated weight matrix is ​​obtained based on the alternating direction multiplier method.

4. The method according to claim 3, characterized in that, The step of obtaining the pruned second sparse neural network based on the second sparse constraint matrix includes: The second candidate neural network is obtained by multiplying the second update weight matrix by the second sparse constraint matrix.

5. The method according to any one of claims 1-4, characterized in that, Also includes: The parameters of the second sparse neural network are trained based on sample data from the application scenario to generate a target neural network, which then processes the input data to obtain output data.

6. The method according to claim 5, characterized in that, The step of training the parameters of the second sparse neural network to generate the target neural network based on sample data from the application scenario includes: Obtain the face sample images used to train the original neural network; Obtain the rendered image of the key parts corresponding to the face sample image; During the training of the second sparse neural network based on the face sample image and the key part rendering image, the parameters of the second sparse neural network are obtained based on the preset objective function constraint to obtain the target neural network, so as to obtain the target image by rendering the test image based on the target neural network.

7. A pruning device for a neural network, characterized in that, The device includes: The first acquisition module is used to acquire the first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned based on the first pruning method, and to acquire the first sparse neural network after pruning based on the first sparse constraint matrix; wherein, the first pruning method is a connection pruning method. The second acquisition module is used to acquire a second sparse constraint matrix corresponding to the weight matrix of the layer to be pruned in the first sparse neural network based on a second pruning method, and to acquire a second sparse neural network after pruning based on the second sparse constraint matrix; wherein, the second pruning method is a pattern pruning method, the first pruning method and the second pruning method are executed in series, the second sparse neural network is trained to generate a target neural network, and the input data of the target neural network includes one or more combinations of audio data, video data, and image data; The step of obtaining the first sparse constraint matrix corresponding to the weight matrix of the original neural network layer to be pruned based on the first pruning method includes: The initial constraint matrix of the connection pruning method corresponding to the initial weight matrix of the original neural network layer to be pruned is obtained based on the alternating direction multiplier method. The target loss function is used as the target loss function for training based on the preset original loss function and the initial constraint matrix of the connection pruning method. After a preset number of iterations of training, the first candidate neural network with updated weights and the corresponding first updated weight matrix are obtained. The first sparse constraint matrix corresponding to the first updated weight matrix is ​​obtained based on the alternating direction multiplier method.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images