Global pruning method of neural network, device thereof, electronic device and storage medium

Abstract

The application discloses a global pruning method of a neural network, a device thereof, an electronic device and a storage medium, and relates to the field of artificial intelligence. The global pruning method comprises the following steps: calculating the L1 norm of each convolution kernel in the neural network; calculating the target pruning number of a convolution layer based on the L1 norm of each convolution kernel; calculating the geometric median of each convolution kernel in each convolution layer; determining the target pruning convolution kernel based on the target pruning number of each convolution layer and the geometric median of each convolution kernel, and performing global pruning on the neural network based on the target pruning convolution kernel. The application solves the technical problem that, in the related art, there is a lack of effective pruning strategy for convolution layers with different sensitivities when pruning the neural network.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and other related technologies. Specifically, it relates to a global pruning method for neural networks, an apparatus, an electronic device, and a storage medium thereof. Background Technology

[0002] Neural networks, as an important artificial intelligence technology, have been widely used in fields such as image recognition, natural language processing, medicine, and finance. Although the learning ability of large neural networks is well-known, in fact, not all neural networks are useful after the training process. Large neural networks often contain many convolutional layers, each containing hundreds or thousands of filters. Fine-grained pruning of the convolutional layer filters can remove redundant parts of the network that have good performance but consume a lot of resources.

[0003] In related technologies, when pruning neural networks, pruning can be performed through convolutional kernel similarity or by calculating the norm. However, the above pruning methods all suffer from inaccurate redundancy filter determination and different pruning sensitivities among convolutional layers. This can lead to excessive filter removal in convolutional layers with high pruning sensitivity, resulting in a significant decrease in accuracy.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a global pruning method, apparatus, electronic device, and storage medium for neural networks, to at least solve the technical problem in the related art of lacking effective pruning strategies for convolutional layers with different sensitivities when pruning neural networks.

[0006] According to one aspect of the present invention, a global pruning method for a neural network is provided, comprising: calculating the L1 norm of each convolutional kernel in the neural network, wherein the neural network comprises at least two convolutional layers, each convolutional layer comprises a plurality of convolutional kernels, the L1 norm being used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels; calculating a target pruning number for the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; calculating the geometric median of each convolutional kernel in each convolutional layer, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, the other convolutional kernels referring to convolutional kernels other than the target convolutional kernel; determining a target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and performing global pruning on the neural network based on the target pruning convolutional kernel.

[0007] Optionally, before calculating the L1 norm of each convolutional kernel in the neural network, the method further includes: numbering the convolutional kernels in the neural network to obtain convolutional kernel numbers.

[0008] Optionally, the step of calculating the L1 norm of each convolutional kernel in the neural network includes: determining the convolutional channels of each convolutional kernel and obtaining the convolutional kernel weight coefficients on each convolutional channel, wherein one convolutional kernel corresponds to N convolutional channels, and N is a positive integer greater than or equal to 1; and summing the absolute values ​​of the convolutional kernel weight coefficients on each convolutional channel to obtain the L1 norm of each convolutional kernel.

[0009] Optionally, the step of calculating the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel includes: counting the total number of convolutional kernels in the neural network and determining a preset pruning rate; multiplying the total number of convolutional kernels and the preset pruning rate to obtain the total number of pruned convolutional kernels; sorting the L1 norms to obtain a norm sorting result; obtaining the number of convolutional kernels to be pruned based on the total number of pruned kernels and the norm sorting result; locating the convolutional kernels to be pruned in the convolutional layer based on the number of convolutional kernels to be pruned, and determining the target pruning number of the convolutional layer.

[0010] Optionally, after calculating the geometric median of each convolutional kernel in each convolutional layer, the method includes: sorting the geometric median of each convolutional kernel in each convolutional layer to obtain a geometric median sorting result; determining redundant convolutional kernels in each convolutional layer based on the target pruning number of the convolutional layer and the geometric median sorting result, and setting the weight coefficient of the redundant convolutional kernels to zero.

[0011] Optionally, after determining the redundant convolutional kernels in each convolutional layer based on the target pruning number of the convolutional layer and the geometric median sorting result, and setting the weight coefficients of the redundant convolutional kernels to zero, the method further includes: initializing the weight coefficients of all the convolutional kernels and iteratively training the neural network; determining the redundant convolutional kernels in each iteration cycle and setting the weight coefficients of the redundant convolutional kernels in each iteration cycle to zero; and performing M iterations of training on the neural network to obtain a sparse neural network, where M is an integer greater than 1.

[0012] Optionally, the step of determining the target pruned convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel includes: calculating the geometric median of each convolutional kernel in each convolutional layer in the sparse neural network to obtain the target geometric median; sorting the target geometric medians corresponding to the convolutional kernels in each convolutional layer to obtain the sorting result of the target geometric medians of each convolutional layer; and determining the target pruned convolutional kernel based on the target pruning number of each convolutional layer and the sorting result of the target geometric median.

[0013] According to another aspect of the present invention, a global pruning apparatus for a neural network is also provided, comprising: a first calculation unit, configured to calculate the L1 norm of each convolutional kernel in the neural network, wherein the neural network comprises at least two convolutional layers, each convolutional layer comprises a plurality of convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels; a second calculation unit, configured to calculate a target pruning number for the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; a third calculation unit, configured to calculate the geometric median of each convolutional kernel in each convolutional layer, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, wherein the other convolutional kernels refer to convolutional kernels other than the target convolutional kernel; and a determination unit, configured to determine a target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and to perform global pruning on the neural network based on the target pruning convolutional kernel.

[0014] Optionally, the global pruning device of the neural network further includes: a first numbering module, used to number the convolution kernels in the neural network to obtain convolution kernel numbers.

[0015] Optionally, the first calculation unit includes: a first determining module, configured to determine the convolutional channels of each convolutional kernel and obtain the convolutional kernel weight coefficients on each convolutional channel, wherein one convolutional kernel corresponds to N convolutional channels, and N is a positive integer greater than or equal to 1; and a first accumulating module, configured to accumulate the absolute values ​​of the convolutional kernel weight coefficients on each convolutional channel to obtain the L1 norm of each convolutional kernel.

[0016] Optionally, the second calculation unit includes: a first statistics module, used to count the total number of convolutional kernels in the neural network and determine a preset pruning rate; a first calculation module, used to multiply the total number of convolutional kernels and the preset pruning rate to obtain the total number of pruned convolutional kernels; a first sorting module, used to sort the L1 norm to obtain a norm sorting result; a first obtaining module, used to obtain the number of the convolutional kernel to be pruned based on the total number of pruned kernels and the norm sorting result; and a first positioning module, used to position the convolutional kernel to be pruned into the convolutional layer based on the number of the convolutional kernel to be pruned, and determine the target number of pruning kernels in the convolutional layer.

[0017] Optionally, the global pruning device of the neural network further includes: a second sorting module, used to sort the geometric median of each convolutional kernel in each convolutional layer to obtain a geometric median sorting result; and a second determining module, used to determine redundant convolutional kernels in each convolutional layer based on the target pruning number of the convolutional layer and the geometric median sorting result, and set the weight coefficients of the redundant convolutional kernels to zero.

[0018] Optionally, the global pruning device for the neural network further includes: a first initialization module, used to initialize the weight coefficients of all the convolutional kernels and perform iterative training on the neural network; a third determination module, used to determine the redundant convolutional kernels in each iteration cycle and set the weight coefficients of the redundant convolutional kernels in each iteration cycle to zero; and a first training module, used to perform M iterations of training on the neural network to obtain a sparse neural network, where M is an integer greater than 1.

[0019] Optionally, the determining unit includes: a second calculation module, configured to calculate the geometric median of each convolutional kernel in each convolutional layer in the sparse neural network to obtain a target geometric median; a third sorting module, configured to sort the target geometric medians corresponding to the convolutional kernels in each convolutional layer to obtain a sorting result of the target geometric medians of each convolutional layer; and a fourth determining module, configured to determine the target pruned convolutional kernel based on the target pruning number and the sorting result of the target geometric medians of each convolutional layer.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described global pruning methods for neural networks.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described global pruning methods for neural networks.

[0022] In this disclosure, the following steps are performed: First, the L1 norm of each convolutional kernel in the neural network is calculated, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernel. Then, the target pruning number of the convolutional layer is calculated based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer. Then, the geometric median of each convolutional kernel in each convolutional layer is calculated, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and other convolutional kernels refer to convolutional kernels other than the target convolutional kernel. Finally, the target pruning convolutional kernel is determined based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and the neural network is globally pruned based on the target pruning convolutional kernel.

[0023] In this disclosure, when pruning a neural network, the L1 norm is first calculated to preliminarily evaluate the importance of the convolutional kernels in the neural network, obtaining the number of pruning operations for each convolutional layer. Then, the geometric median of the convolutional kernels is calculated to perform a secondary evaluation of each convolutional kernel, determining the convolutional kernels to be pruned, and performing global pruning on the neural network. This solves the problems of inaccurate determination of redundant filters and different sensitivities of each convolutional layer to pruning, improving the accuracy of neural network pruning. Furthermore, it solves the technical problem in related technologies of lacking effective pruning strategies for convolutional layers with different sensitivities when pruning neural networks. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a flowchart of an optional global pruning method for a neural network according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of an optional method for calculating the target number of pruning branches in the convolutional layer according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional global pruning device for a neural network according to an embodiment of the present invention;

[0028] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) according to an embodiment of the present invention, which describes a global pruning method for a neural network. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the global pruning method and apparatus for neural networks in this disclosure can be used in the field of artificial intelligence for pruning neural networks, and can also be used in any field other than artificial intelligence for pruning neural networks. This disclosure does not limit the application field of the global pruning method and apparatus for neural networks.

[0032] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) disclosed herein are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information must be sent to the aforementioned user or organization through the interface, and the relevant information will be obtained only after receiving consent from the aforementioned user or organization.

[0033] The following embodiments of the present invention can be applied to various global pruning systems / applications / devices for neural networks. The present invention uses L1 norm and geometric median to evaluate the convolutional kernels of the neural network twice. First, the L1 norm is calculated to make a preliminary assessment of the importance of the convolutional kernels, thereby obtaining the pruning rate of each convolutional layer. Then, the geometric median of each convolutional kernel is calculated to make a second evaluation of the neural network convolutional kernels, obtaining the convolutional kernels to be pruned in each convolutional layer, and performing global pruning on the neural network. Based on the two convolutional kernel evaluations, the pruning accuracy is improved, and the problem of different sensitivities among different convolutional layers is solved.

[0034] The present invention will now be described in detail with reference to various embodiments.

[0035] Example 1

[0036] According to an embodiment of the present invention, an embodiment of a global pruning method for a neural network is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 1 This is a flowchart of an optional global pruning method for a neural network according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0038] Step S101: Calculate the L1 norm of each convolutional kernel in the neural network, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernel;

[0039] Step S102: Calculate the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer;

[0040] Step S103: Calculate the geometric median of each convolutional kernel in each convolutional layer, where the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and other convolutional kernels refer to convolutional kernels other than the target convolutional kernel;

[0041] Step S104: Determine the target pruning convolution kernel based on the target pruning number of each convolutional layer and the geometric median of each convolution kernel, and perform global pruning on the neural network based on the target pruning convolution kernel.

[0042] Through the above steps, firstly, the L1 norm of each convolutional kernel in the neural network is calculated. The neural network contains at least two convolutional layers, and each convolutional layer contains multiple convolutional kernels. The L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernel. Then, the target pruning number of the convolutional layer is calculated based on the L1 norm of each convolutional kernel. The target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer. Next, the geometric median of each convolutional kernel in each convolutional layer is calculated. The geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer. Other convolutional kernels refer to convolutional kernels other than the target convolutional kernel. Finally, the target pruning convolutional kernel is determined based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel. Global pruning of the neural network is then performed based on the target pruning convolutional kernel.

[0043] In this embodiment, when pruning the neural network, the L1 norm is first calculated to preliminarily evaluate the importance of the convolutional kernels in the neural network, obtaining the number of pruning operations for each convolutional layer. Then, the geometric median of the convolutional kernels is calculated to perform a secondary evaluation of each convolutional kernel, determining the convolutional kernels to be pruned, and performing global pruning on the neural network. This solves the problems of inaccurate determination of redundant filters and different sensitivities of each convolutional layer to pruning, improving the accuracy of neural network pruning. Furthermore, it solves the technical problem in related technologies where there is a lack of effective pruning strategies for convolutional layers with different sensitivities when pruning neural networks.

[0044] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0045] It should be noted that the embodiments of the present invention are applied to the scenario of pruning neural networks in the field of artificial intelligence. The neural network is composed of multiple convolutional kernels, which are distributed in various convolutional layers. The number of convolutional kernels in each convolutional layer is different, resulting in different sensitivities in each convolutional layer of the neural network.

[0046] It should be noted that not all parts of a trained neural network are usable. During iterative training, a large number of redundant filters, i.e., redundant convolutional kernels, may be generated. Redundant convolutional kernels will consume too much network resources. Therefore, pruning the neural network can effectively solve the problem of redundant convolutional kernel resource consumption.

[0047] Optionally, before calculating the L1 norm of each convolutional kernel in the neural network, the method further includes: numbering the convolutional kernels in the neural network to obtain convolutional kernel numbers.

[0048] It should be noted that, in order to distinguish the massive number of convolutional kernels in a neural network, each convolutional kernel can be numbered. This is done on a layer-by-layer basis, assigning a number to each convolutional kernel in each layer, resulting in C. l,j'l' represents the number of each convolutional layer, and 'j' represents the number of the convolutional kernel in the convolutional layer. Alternatively, it can be represented as 'conv'(the convolutional layer to which the kernel belongs, and its position number within the convolutional layer). For example, the fourth convolutional kernel in the first convolutional layer is C. 1,4 Or conv(1,4).

[0049] Step S101: Calculate the L1 norm of each convolutional kernel in the neural network.

[0050] It should be noted that a neural network contains at least two convolutional layers, and each convolutional layer contains multiple convolutional kernels. The L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels. It can be used to characterize the importance of each convolutional kernel. Based on the L1 norm, the neural network can be initially evaluated to extract the number of convolutional kernels with lower importance in each convolutional layer. By calculating the L1 norm, the problem of ignoring the different sensitivities of each convolutional layer by performing a fixed number of prunings on each convolutional layer can be avoided.

[0051] Optionally, the step of calculating the L1 norm of each convolutional kernel in the neural network includes: determining the convolutional channels of each convolutional kernel and obtaining the convolutional kernel weight coefficients on each convolutional channel, wherein one convolutional kernel corresponds to N convolutional channels, and N is a positive integer greater than or equal to 1; and summing the absolute values ​​of the convolutional kernel weight coefficients on each convolutional channel to obtain the L1 norm of each convolutional kernel.

[0052] It should be noted that in a neural network, each convolutional layer includes multiple convolutional channels, and each convolutional kernel in each channel corresponds to a weight coefficient. During the training of the neural network, weight coefficients are assigned to the convolutional kernels. When calculating the L1 norm, the absolute values ​​of the individual convolutional kernel weight coefficients for the multiple channels corresponding to the kernel are summed to obtain the L1 norm of each convolutional kernel in the convolutional layer. This L1 norm is used as a preliminary criterion for screening the importance of the convolutional kernel. The formula for calculating the L1 norm is as follows: The superscript (l) indicates the l-th convolutional layer, the subscript j indicates the j-th convolutional kernel, and c∈[1,n] i ] represents the c-th convolutional channel of the convolutional kernel, K c This represents the weight of a single convolutional kernel on the c-th convolutional channel.

[0053] Step S102: Calculate the target number of pruning branches for the convolutional layer based on the L1 norm of each convolutional kernel.

[0054] It should be noted that the L1 norm can be used to evaluate the importance of each convolutional kernel. Based on the L1 norm of each convolutional kernel, the number of convolutional kernels to be pruned in each convolutional layer can be determined, i.e., the target number of pruning kernels.

[0055] Figure 2This is a flowchart of an optional method for calculating the target pruning number of a convolutional layer according to an embodiment of the present invention, such as... Figure 2 As shown, the steps for calculating the target number of pruning branches for a convolutional layer based on the L1 norm of each convolutional kernel include:

[0056] Step S201: Count the total number of convolutional kernels in the neural network and determine the preset pruning rate;

[0057] Step S202: Multiply the total number of convolution kernels by the preset pruning rate to obtain the total number of pruned convolution kernels;

[0058] Step S203: Sort the L1 norms to obtain the norm sorting results;

[0059] Step S204: Obtain the convolution kernel number to be pruned based on the total number of prunes and the norm sorting result;

[0060] Step S205: Based on the convolution kernel number to be pruned, locate the convolution kernel to be pruned in the convolutional layer and determine the target number of prunes in the convolutional layer.

[0061] It should be noted that after calculating the L1 norm of each convolutional kernel in each convolutional layer, sorting the norms from smallest to largest yields a preliminary global evaluation of the convolutional kernels. Based on the preset model pruning rate Pr (obtained from user input) and the total number of convolutional kernels a, the kernel with the smallest L1 norm, P, needs to be pruned in this pruning process. r ×a convolutional kernels. Using a single convolutional layer as the evaluation dimension, based on the initial evaluation ranking, P to be pruned is... r ×a convolutional kernels are located in each convolutional layer, and the pruning rate of each convolutional layer is calculated. and the number of convolution kernels to be clipped in each layer For example, if the total number of convolution kernels to be pruned is 10, according to After sorting and locating, we find that the first convolutional layer has 2 convolutional kernels to be pruned, and the second convolutional layer has 8 convolutional kernels to be pruned. Assuming that each of these two convolutional layers has 16 convolutional kernels, we can obtain... It is 2. It is 1 / 8. It is 8. It is 1 / 2.

[0062] Step S103: Calculate the geometric median of each convolutional kernel in each convolutional layer.

[0063] It should be noted that the geometric median of the convolutional kernel is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer. Other convolutional kernels refer to convolutional kernels other than the target convolutional kernel. A smaller Euclidean distance indicates that the convolutional kernel is highly similar to other convolutional kernels and can be replaced by other convolutional kernels, i.e., a redundant convolutional kernel. Calculating the redundancy of convolutional kernels can remove convolutional kernels with high redundancy and avoid redundant convolutional kernels occupying network resources.

[0064] Optionally, after calculating the geometric median of each convolutional kernel in each convolutional layer, the method includes: sorting the geometric median of each convolutional kernel in each convolutional layer to obtain a geometric median sorting result; determining redundant convolutional kernels in each convolutional layer based on the target pruning number of the convolutional layer and the geometric median sorting result, and setting the weight coefficients of the redundant convolutional kernels to zero.

[0065] It should be noted that after calculating the geometric median of the convolutional kernels, the geometric medians of the convolutional kernels in each convolutional layer are sorted. Based on the sorting results and the pre-calculated target pruning number for each convolutional layer, the kernel with the smallest geometric median in the l-th layer is selected. Each convolutional kernel is identified as a redundant convolutional kernel. The weight coefficients of the redundant convolutional kernels are set to 0, and the model is iterated. The weight coefficients of the redundant convolutional kernels in each iteration cycle are set to 0, thus obtaining a model with sparse convolutional kernel dimensions.

[0066] Optionally, after determining the redundant convolutional kernels in each convolutional layer based on the target pruning number and geometric median sorting result of the convolutional layer, and setting the weight coefficients of the redundant convolutional kernels to zero, the method further includes: initializing the weight coefficients of all convolutional kernels and iteratively training the neural network; determining the redundant convolutional kernels in each iteration cycle and setting the weight coefficients of the redundant convolutional kernels in each iteration cycle to zero; and performing M iterations of training on the neural network to obtain a sparse neural network, where M is an integer greater than 1.

[0067] It should be noted that when training a neural network iteratively, the weight coefficients of the convolution kernels need to be initialized before each iteration. Then, the geometric median of each convolution kernel is calculated, and redundant convolution kernels are selected based on the geometric median. The weight coefficients of the redundant convolution kernels are set to 0. After multiple iterations of training, a sparse neural network is obtained.

[0068] Step S104: Determine the target pruning convolution kernel based on the target pruning number of each convolutional layer and the geometric median of each convolution kernel, and perform global pruning on the neural network based on the target pruning convolution kernel.

[0069] Optionally, the step of determining the target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel includes: calculating the geometric median of each convolutional kernel in each convolutional layer in the sparse neural network to obtain the target geometric median; sorting the target geometric medians corresponding to the convolutional kernels in each convolutional layer to obtain the sorting result of the target geometric medians of each convolutional layer; and determining the target pruning convolutional kernel based on the sorting result of the target pruning number of each convolutional layer and the target geometric median.

[0070] It should be noted that by calculating the target number of prunes for each convolutional layer and the geometric median of each convolutional kernel, the neural network can be screened twice to determine the convolutional kernels to be pruned and to locate the convolutional kernels in the convolutional layer. The number of convolutional kernels pruned will also be different for convolutional layers with different sensitivities. Global pruning of the neural network based on the convolutional kernels to be pruned in each convolutional layer can yield a pruned model. The compact model performs the same function as the original model, but can save a lot of computing resources and improve computational efficiency.

[0071] The above-mentioned pruning method for neural networks can be implemented using neural network pseudocode:

[0072] Input: trained model M, dataset D, learning rate α, preset pruning rate Pr;

[0073] Output: Pruned compact model M';

[0074] The specific pseudocode is as follows:

[0075] do for each epoch;

[0076] do for each layer;

[0077] Calculate the sum of the absolute values ​​of the weights of each convolutional layer.

[0078] Determine the number of convolution kernels to be clipped in each layer.

[0079] Find the layer with the smallest d value in the l-th layer. Each convolutional kernel is identified as a redundant convolutional kernel, and its weights are all set to 0.

[0080] end for;

[0081] end for;

[0082] We obtain a model M with sparse convolution kernels;

[0083] P r After removing ×a convolution kernels, the pruned compact model M' is obtained.

[0084] The following is a detailed description with reference to another embodiment.

[0085] Example 2

[0086] The global pruning device for a neural network provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0087] Figure 3 This is a schematic diagram of an optional global pruning device for a neural network according to an embodiment of the present invention, such as... Figure 3 As shown, the global pruning device includes: a first calculation unit 31, a second calculation unit 32, a third calculation unit 33, and a determination unit 34, wherein,

[0088] The first computing unit 31 is used to calculate the L1 norm of each convolutional kernel in the neural network, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels.

[0089] The second calculation unit 32 is used to calculate the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer.

[0090] The third calculation unit 33 is used to calculate the geometric median of each convolutional kernel in each convolutional layer. The geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer. Other convolutional kernels refer to convolutional kernels other than the target convolutional kernel.

[0091] The determination unit 34 is used to determine the target pruning convolution kernel based on the target pruning number of each convolutional layer and the geometric median of each convolution kernel, and to perform global pruning on the neural network based on the target pruning convolution kernel.

[0092] The aforementioned global pruning device calculates the L1 norm of each convolutional kernel in the neural network through a first computing unit 31, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels; calculates the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel through a second computing unit 32, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; calculates the geometric median of each convolutional kernel in each convolutional layer through a third computing unit 33, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and other convolutional kernels refer to convolutional kernels other than the target convolutional kernel; and determines the target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel through a determining unit 34, and performs global pruning on the neural network based on the target pruning convolutional kernel.

[0093] In this embodiment, when pruning the neural network, the L1 norm is first calculated to preliminarily evaluate the importance of the convolutional kernels in the neural network, obtaining the number of pruning operations for each convolutional layer. Then, the geometric median of the convolutional kernels is calculated to perform a secondary evaluation of each convolutional kernel, determining the convolutional kernels to be pruned, and performing global pruning on the neural network. This solves the problems of inaccurate determination of redundant filters and different sensitivities of each convolutional layer to pruning, improving the accuracy of neural network pruning. Furthermore, it solves the technical problem in related technologies where there is a lack of effective pruning strategies for convolutional layers with different sensitivities when pruning neural networks.

[0094] It should be noted that the embodiments of the present invention are applied to the scenario of pruning neural networks in the field of artificial intelligence. The neural network is composed of multiple convolutional kernels, which are distributed in various convolutional layers. The number of convolutional kernels in each convolutional layer is different, resulting in different sensitivities in each convolutional layer of the neural network.

[0095] It should be noted that not all parts of a trained neural network are usable. During iterative training, a large number of redundant filters, i.e., redundant convolutional kernels, may be generated. Redundant convolutional kernels will consume too much network resources. Therefore, pruning the neural network can effectively solve the problem of redundant convolutional kernel resource consumption.

[0096] Optionally, the global pruning device for the neural network further includes: a first numbering module, used to number the convolution kernels in the neural network to obtain convolution kernel numbers.

[0097] It should be noted that, in order to distinguish the massive number of convolutional kernels in a neural network, each convolutional kernel can be numbered. This is done on a layer-by-layer basis, assigning a number to each convolutional kernel in each layer, resulting in C. l,j 'l' represents the number of each convolutional layer, and 'j' represents the number of the convolutional kernel in the convolutional layer. Alternatively, it can be represented as 'conv'(the convolutional layer to which the kernel belongs, and its position number within the convolutional layer). For example, the fourth convolutional kernel in the first convolutional layer is C. 1,4 Or conv(1,4).

[0098] Optionally, the first calculation unit includes: a first determining module, used to determine the convolution channels of each convolution kernel and obtain the convolution kernel weight coefficients on each convolution channel, wherein one convolution kernel corresponds to N convolution channels, and N is a positive integer greater than or equal to 1; and a first accumulating module, used to accumulate the absolute values ​​of the convolution kernel weight coefficients on each convolution channel to obtain the L1 norm of each convolution kernel.

[0099] It should be noted that a neural network contains at least two convolutional layers, and each convolutional layer contains multiple convolutional kernels. The L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels. It can be used to characterize the importance of each convolutional kernel. Based on the L1 norm, the neural network can be initially evaluated to extract the number of convolutional kernels with lower importance in each convolutional layer. By calculating the L1 norm, the problem of ignoring the different sensitivities of each convolutional layer by performing a fixed number of prunings on each convolutional layer can be avoided.

[0100] It should be noted that in a neural network, each convolutional layer includes multiple convolutional channels, and each convolutional kernel in each channel corresponds to a weight coefficient. During the training of the neural network, weight coefficients are assigned to the convolutional kernels. When calculating the L1 norm, the absolute values ​​of the individual convolutional kernel weight coefficients for the multiple channels corresponding to the kernel are summed to obtain the L1 norm of each convolutional kernel in the convolutional layer. This L1 norm is used as a preliminary criterion for screening the importance of the convolutional kernel. The formula for calculating the L1 norm is as follows: The superscript (l) indicates the l-th convolutional layer, the subscript j indicates the j-th convolutional kernel, and c∈[1,n] i ] represents the c-th convolutional channel of the convolutional kernel, K c This represents the weight of a single convolutional kernel on the c-th convolutional channel.

[0101] Optionally, the second calculation unit includes: a first statistics module for counting the total number of convolutional kernels in the neural network and determining a preset pruning rate; a first calculation module for multiplying the total number of convolutional kernels by the preset pruning rate to obtain the total number of pruned convolutional kernels; a first sorting module for sorting the L1 norm to obtain a norm sorting result; a first obtaining module for obtaining the number of convolutional kernels to be pruned based on the total number of pruned kernels and the norm sorting result; and a first positioning module for positioning the convolutional kernels to be pruned into the convolutional layer based on the number of the convolutional kernels to be pruned, and determining the target number of pruning kernels in the convolutional layer.

[0102] It should be noted that after calculating the L1 norm of each convolutional kernel in each convolutional layer, sorting the norms from smallest to largest yields a preliminary global evaluation of the convolutional kernels. Based on the preset model pruning rate Pr (obtained from user input) and the total number of convolutional kernels a, the kernel with the smallest L1 norm, P, needs to be pruned in this pruning process. r ×a convolutional kernels. Using a single convolutional layer as the evaluation dimension, based on the initial evaluation ranking, P to be pruned is... r ×a convolutional kernels are located in each convolutional layer, and the pruning rate of each convolutional layer is calculated. and the number of convolution kernels to be clipped in each layer For example, if the total number of convolution kernels to be pruned is 10, according to After sorting and locating, we find that the first convolutional layer has 2 convolutional kernels to be pruned, and the second convolutional layer has 8 convolutional kernels to be pruned. Assuming that each of these two convolutional layers has 16 convolutional kernels, we can obtain... It is 2. It is 1 / 8. It is 8. It is 1 / 2.

[0103] It should be noted that the geometric median of the convolutional kernel is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer. Other convolutional kernels refer to convolutional kernels other than the target convolutional kernel. A smaller Euclidean distance indicates that the convolutional kernel is highly similar to other convolutional kernels and can be replaced by other convolutional kernels, i.e., a redundant convolutional kernel. Calculating the redundancy of convolutional kernels can remove convolutional kernels with high redundancy and avoid redundant convolutional kernels occupying network resources.

[0104] Optionally, the global pruning device for the neural network further includes: a second sorting module, used to sort the geometric median of each convolutional kernel in each convolutional layer to obtain a geometric median sorting result; and a second determining module, used to determine redundant convolutional kernels in each convolutional layer based on the target pruning number of the convolutional layer and the geometric median sorting result, and set the weight coefficients of the redundant convolutional kernels to zero.

[0105] Optionally, the global pruning device for the neural network further includes: a first initialization module for initializing the weight coefficients of all convolutional kernels and iteratively training the neural network; a third determination module for determining redundant convolutional kernels in each iteration cycle and setting the weight coefficients of redundant convolutional kernels in each iteration cycle to zero; and a first training module for training the neural network M times to obtain a sparse neural network, where M is an integer greater than 1.

[0106] It should be noted that after calculating the geometric median of the convolutional kernels, the geometric medians of the convolutional kernels in each convolutional layer are sorted. Based on the sorting results and the pre-calculated target pruning number for each convolutional layer, the kernel with the smallest geometric median in the l-th layer is selected. Each convolutional kernel is identified as a redundant convolutional kernel. The weight coefficients of the redundant convolutional kernels are set to 0, and the model is iterated. The weight coefficients of the redundant convolutional kernels in each iteration cycle are set to 0, thus obtaining a model with sparse convolutional kernel dimensions.

[0107] It should be noted that when training a neural network iteratively, the weight coefficients of the convolution kernels need to be initialized before each iteration. Then, the geometric median of each convolution kernel is calculated, and redundant convolution kernels are selected based on the geometric median. The weight coefficients of the redundant convolution kernels are set to 0. After multiple iterations of training, a sparse neural network is obtained.

[0108] Optionally, the determining unit includes: a second calculation module, used to calculate the geometric median of each convolutional kernel in each convolutional layer in the sparse neural network to obtain the target geometric median; a third sorting module, used to sort the target geometric medians corresponding to the convolutional kernels in each convolutional layer to obtain the sorting result of the target geometric medians of each convolutional layer; and a fourth determining module, used to determine the target pruned convolutional kernel based on the target pruning number and the sorting result of the target geometric median of each convolutional layer.

[0109] It should be noted that by calculating the target number of prunes for each convolutional layer and the geometric median of each convolutional kernel, the neural network can be screened twice to determine the convolutional kernels to be pruned and to locate the convolutional kernels in the convolutional layer. The number of convolutional kernels pruned will also be different for convolutional layers with different sensitivities. Global pruning of the neural network based on the convolutional kernels to be pruned in each convolutional layer can yield a pruned model. The compact model performs the same function as the original model, but can save a lot of computing resources and improve computational efficiency.

[0110] The aforementioned global pruning device for neural networks may also include a processor and a memory. The first computing unit 31, the second computing unit 32, the third computing unit 33, the determination unit 34, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0111] The processor described above contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and global pruning of the neural network can be performed by adjusting kernel parameters.

[0112] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0113] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described global pruning methods for neural networks.

[0114] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described global pruning methods for neural networks.

[0115] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: calculating the L1 norm of each convolutional kernel in a neural network, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernels; calculating the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; calculating the geometric median of each convolutional kernel in each convolutional layer, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and other convolutional kernels refer to convolutional kernels other than the target convolutional kernel; determining the target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and performing global pruning of the neural network based on the target pruning convolutional kernel.

[0116] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) according to an embodiment of the present invention, which describes a global pruning method for a neural network. Figure 4 As shown, an electronic device may include one or more ( Figure 4 The processor 402 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and the memory 404 for storing data may also be included. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0117] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0118] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0123] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A global pruning method for neural networks, characterized in that, The method, applied to image recognition, includes: Calculate the L1 norm of each convolutional kernel in a neural network, wherein the neural network contains at least two convolutional layers, each of the convolutional layers contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernel; The target pruning number of the convolutional layer is calculated based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; The step of calculating the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel includes: counting the total number of convolutional kernels in the neural network and determining a preset pruning rate; multiplying the total number of convolutional kernels and the preset pruning rate to obtain the total number of pruned convolutional kernels; sorting the L1 norms to obtain a norm sorting result; obtaining the number of the convolutional kernel to be pruned based on the total number of pruned kernels and the norm sorting result; and locating the convolutional kernel to be pruned in the convolutional layer based on the number of the convolutional kernel to be pruned, thereby determining the target pruning number of the convolutional layer. Calculate the geometric median of each of the convolutional kernels in each of the convolutional layers, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and the other convolutional kernels refer to convolutional kernels other than the target convolutional kernel; The target pruning convolutional kernel is determined based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and the neural network is globally pruned based on the target pruning convolutional kernel.

2. The global pruning method according to claim 1, characterized in that, Before calculating the L1 norm of each convolutional kernel in the neural network, the following steps are also included: The convolution kernels in the neural network are numbered to obtain convolution kernel numbers.

3. The global pruning method according to claim 1, characterized in that, The steps for calculating the L1 norm of each convolutional kernel in a neural network include: Determine the convolution channels of each convolution kernel and obtain the convolution kernel weight coefficients on each convolution channel, wherein one convolution kernel corresponds to N convolution channels, and N is a positive integer greater than or equal to 1; The absolute values ​​of the kernel weight coefficients on each of the convolution channels are summed to obtain the L1 norm of each convolution kernel.

4. The global pruning method according to claim 1, characterized in that, After calculating the geometric median of each of the convolutional kernels in each of the convolutional layers, the process includes: The geometric median of each convolutional kernel in each convolutional layer is sorted to obtain a geometric median sorting result; Based on the target pruning number of the convolutional layer and the geometric median sorting result, redundant convolutional kernels in each convolutional layer are determined, and the weight coefficients of the redundant convolutional kernels are set to zero.

5. The global pruning method according to claim 4, characterized in that, After determining the redundant convolutional kernels in each convolutional layer based on the target pruning number and the geometric median sorting result, and setting the weight coefficients of the redundant convolutional kernels to zero, the method further includes: The weight coefficients of all the convolutional kernels are initialized, and the neural network is trained iteratively. The redundant convolutional kernels in each iteration period are determined, and the weight coefficients of the redundant convolutional kernels in each iteration period are set to zero; The neural network is trained M times to obtain a sparse neural network, where M is an integer greater than 1.

6. The global pruning method according to claim 5, characterized in that, The step of determining the target pruned convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel includes: The geometric median of each convolutional kernel in each convolutional layer of the sparse neural network is calculated to obtain the target geometric median; The target geometric median corresponding to the convolution kernel in each convolutional layer is sorted to obtain the sorting result of the target geometric median of each convolutional layer. The target pruned convolutional kernel is determined based on the sorting result of the target pruning number and the target geometric median for each convolutional layer.

7. A global pruning device for a neural network, characterized in that, The device, used for image recognition, includes: The first computing unit is used to calculate the L1 norm of each convolutional kernel in the neural network, wherein the neural network contains at least two convolutional layers, each convolutional layer contains multiple convolutional kernels, and the L1 norm is used to indicate the sum of the absolute values ​​of all weight coefficients corresponding to the convolutional kernel; The second calculation unit is used to calculate the target pruning number of the convolutional layer based on the L1 norm of each convolutional kernel, wherein the target pruning number is used to determine the number of convolutional kernels to be pruned in the convolutional layer; The second calculation unit includes: a first statistics module, used to count the total number of convolutional kernels in the neural network and determine a preset pruning rate; a first calculation module, used to multiply the total number of convolutional kernels and the preset pruning rate to obtain the total number of pruned convolutional kernels; a first sorting module, used to sort the L1 norm to obtain a norm sorting result; a first obtaining module, used to obtain the number of the convolutional kernel to be pruned based on the total number of pruned kernels and the norm sorting result; and a first positioning module, used to position the convolutional kernel to be pruned into the convolutional layer based on the number of the convolutional kernel to be pruned, and determine the target number of pruning kernels in the convolutional layer. The third calculation unit is used to calculate the geometric median of each convolutional kernel in each convolutional layer, wherein the geometric median is used to indicate the Euclidean distance between the target convolutional kernel and other convolutional kernels in each convolutional layer, and the other convolutional kernels refer to convolutional kernels other than the target convolutional kernel; A determining unit is configured to determine a target pruning convolutional kernel based on the target pruning number of each convolutional layer and the geometric median of each convolutional kernel, and to perform global pruning on the neural network based on the target pruning convolutional kernel.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the global pruning method of the neural network according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the global pruning method of the neural network according to any one of claims 1 to 6.

Images