Pruning method and device of neural network, electronic equipment and storage medium

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for pruning a neural network. The method includes: determining the initial measure values ​​of each output channel of the i-th layer in the neural network to be pruned and the target parameter values ​​of the i-th layer; normalizing the initial measure values ​​of each output channel of the i-th layer according to the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel of the i-th layer; and pruning the i-th layer according to the target measure values ​​of each output channel of the i-th layer to obtain a pruned neural network; where i is a positive integer. Thus, by pruning the i-th layer using the target measure values ​​of each output channel of the i-th layer, global pruning processing for cross-layer comparison can be achieved, making the inference performance of the pruned small model (i.e., the pruned neural network) more significant, while also improving the accuracy of the pruned small model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for pruning neural networks. Background Technology

[0002] Deep learning algorithms based on neural networks have been widely used in fields such as machine vision, natural language processing, autonomous driving, and robotics. However, their complex structure and large number of parameters place higher demands on computer resources such as computing power and memory, which severely limits the deployment of neural networks on resource-constrained edge devices.

[0003] Among these technologies, a series of model compression algorithms have been proposed, such as using structured pruning algorithms to reduce model size, thereby reducing the neural network's demand for computing power and memory resources. However, although structured pruning algorithms can use layer-by-layer pruning to prune the neural network, the inference performance of the pruned small model obtained through this method is not significant, and the accuracy of the pruned small model still needs to be improved. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for pruning neural networks, which can make the inference performance of the pruned small model more significant and improve the accuracy of the pruned small model.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a method for pruning a neural network, the method comprising:

[0007] Determine the initial measure values ​​of each output channel in the i-th layer of the neural network to be pruned, as well as the target parameter values ​​of the i-th layer.

[0008] The initial measure values ​​of each output channel of the i-th layer are normalized based on the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel of the i-th layer.

[0009] The i-th layer is pruned based on the target metric values ​​of each output channel of the i-th layer to obtain the pruned neural network; where i is a positive integer.

[0010] Secondly, embodiments of this application provide a pruning device for a neural network, which includes a determining unit, a normalizing unit, and a pruning unit, wherein:

[0011] The determining unit is configured to determine the initial measure values ​​of each output channel of the i-th layer in the neural network to be pruned and the target parameter values ​​of the i-th layer.

[0012] The normalization unit is configured to normalize the initial measure values ​​of each output channel of the i-th layer according to the target parameter values ​​of the i-th layer, so as to obtain the target measure values ​​of each output channel of the i-th layer.

[0013] The pruning unit is configured to prune the i-th layer according to the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network; where i is a positive integer.

[0014] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the neural network pruning method described in the first aspect.

[0015] Fourthly, the chip provided in the embodiments of this application is used to implement the neural network pruning method described in the first aspect above.

[0016] Specifically, the chip includes a processor for retrieving and running a computer program from memory, causing a device equipped with the chip to perform the neural network pruning method described in the first aspect above.

[0017] Fifthly, the computer-readable storage medium provided in the embodiments of this application stores a computer program that, when executed by at least one processor, implements the neural network pruning method described in the first aspect above.

[0018] Sixthly, the computer program product provided in the embodiments of this application includes computer program instructions that cause a computer to execute the neural network pruning method described in the first aspect.

[0019] In a seventh aspect, the computer program provided in the embodiments of this application, when run on a computer, causes the computer to execute the neural network pruning method described in the first aspect above.

[0020] This application provides a method for pruning a neural network. First, the initial measure values ​​of each output channel in the i-th layer of the neural network to be pruned and the target parameter values ​​of the i-th layer are determined. Then, the initial measure values ​​of each output channel in the i-th layer are normalized based on the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel. Finally, the i-th layer is pruned based on the target measure values ​​of each output channel to obtain the pruned neural network. In this way, normalizing the initial measure values ​​of each output channel in the i-th layer using the target parameter values ​​of the i-th layer reduces the sensitivity of the target measure values ​​of each output channel to the target parameter values ​​of the i-th layer, allowing the target measure values ​​of each output channel to be applied to global pruning for cross-layer comparisons. Thus, by pruning the i-th layer using the target measure values ​​of each output channel, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly better inference performance, improving the accuracy of the pruned small model. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a scenario for the structured pruning algorithm.

[0022] Figure 2 This is a schematic diagram illustrating a scenario using the unstructured pruning algorithm.

[0023] Figure 3 This is a schematic diagram of a scenario where output channels to be deleted in a neural network to be pruned.

[0024] Figure 4 This is a schematic diagram of a vectorized output channel in a neural network to be pruned.

[0025] Figure 5 This is a flowchart illustrating a neural network pruning method provided in an embodiment of this application.

[0026] Figure 6 This is a flowchart illustrating another neural network pruning method provided in an embodiment of this application.

[0027] Figure 7 This is a schematic diagram of the structural composition of a neural network pruning device provided in an embodiment of this application.

[0028] Figure 8 This is a schematic structural diagram of an electronic device provided in an embodiment of this application.

[0029] Figure 9 This is a schematic structural diagram of a chip provided in an embodiment of this application. Detailed Implementation

[0030] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] It should also be noted that the terms "first, second, and third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In related technologies, deep learning algorithms based on neural networks (such as convolutional neural networks) have flourished in recent years and have been widely used in fields such as machine vision, natural language processing, autonomous driving, and robotics, achieving good results. However, the complexity of their structure and the large number of parameters place higher demands on computer resources such as computing power and memory, thus limiting the deployment and application of neural networks on edge devices (such as mobile phones and cameras) where computing power and memory resources are severely limited.

[0035] To address the aforementioned issues, a series of model compression algorithms have been proposed to reduce the computational and memory requirements of neural networks. These include quantization, pruning, and distillation algorithms. Among these, pruning algorithms significantly reduce the number of parameters in the pruned neural network by removing redundant parameters, thereby enabling the deployment and application of neural networks on edge devices where computational and memory resources are severely limited.

[0036] For example, suppose the neural network has L layers, n i n is used to represent the number of input channels in the i-th layer of a neural network. i+1 F is used to represent the number of output channels in the i-th layer. i,mUsed to represent the filters for each output channel of the i-th layer, where 1 ≤ m ≤ n i+1， Then F i,m The parameter dimension is Layer i W (i) It can be represented as {F i,m ,1≤m≤n i+1 The parameter dimension of the entire neural network can be represented as follows: Where K represents the kernel size of the filter.

[0037] For example, such as Figure 1 As shown, the filter F of the i-th layer in the neural network to be pruned i,m The kernel size K is equal to 3, and the number of input channels in the i-th layer is n. i The value equals 3. Let F be the filter for output channel 1 of the i-th layer in the neural network to be pruned. i,1 Then F i,1 The parameter dimension is The parameter dimension of the i-th layer in the neural network to be pruned can be expressed as:

[0038] It's important to note that pruning algorithms can be categorized into structured pruning and unstructured pruning. Structured pruning algorithms target a specific layer in the neural network, removing redundant output channels to reduce the number of parameters. Unstructured pruning algorithms, on the other hand, don't completely delete output channels; instead, they set some elements (parameters) within those channels to zero. Therefore, structured and unstructured pruning algorithms differ in their granularity.

[0039] For example, such as Figure 1 As shown, assume that the number of output channels in the i-th layer of the neural network to be pruned is n. i+1 The output channels of the i-th layer are denoted as output channel 1, output channel 2, ..., output channel n, respectively. i+1 -1. Output channel n i+1 After pruning the i-th layer using the structured pruning algorithm, the pruned neural network has at least two output channels, i and n, removed from the i-th layer. i+1 -1( Figure 1 (As shown in the black area). Thus, by deleting output channels, the number of parameters in the i-th layer of the pruned neural network can be reduced compared to the i-th layer of the unpruned neural network.

[0040] For example, such as Figure 2As shown, assume that the number of output channels in the i-th layer of the neural network to be pruned is n. i+1 The output channels of the i-th layer are denoted as output channel 1, output channel 2, ..., output channel n, respectively. i+1 -1. Output channel n i+1 After pruning the i-th layer using an unstructured pruning algorithm, the resulting pruned neural network will have some elements of some output channels set to zero in the i-th layer. For example... Figure 2 Output channel 1, output channel 2, output channel n i+1 -1 and output channel n i+1 The black area in the diagram illustrates this. Thus, by setting some elements in some output channels to zero, the number of parameters in the i-th layer of the pruned neural network can be reduced compared to the i-th layer of the unpruned neural network.

[0041] It should be noted that although unstructured pruning algorithms can more accurately locate redundant parameters, structured pruning algorithms have better hardware support and are easier to implement on hardware devices. Therefore, structured pruning algorithms have a wider range of applications in practice.

[0042] In related technologies, structured pruning algorithms based on geometric median can determine the filter in the i-th layer of the neural network to be pruned that is close to the geometric median. Thus the filter The corresponding output channel is determined to be a redundant output channel. Deleting this redundant output channel has little impact on the inference performance of the i-th layer in the pruned neural network. (Filter) The calculation formula is as follows:

[0043]

[0044] or,

[0045]

[0046] in,

[0047] Furthermore, the structured pruning algorithm based on the geometric median can first determine the measure value of each output channel in the i-th layer of the neural network to be pruned; secondly, it can calculate the sparsity S of the neural network to be pruned. i Sparsity and the number of output channels n of the i-th layer i+1 The product S i ×n i+1 ; Sort the measure values ​​of each output channel of the i-th layer from smallest to largest again, and then sort the S-th layer by the result. i ×n i+1The metric values ​​are used as thresholds; then the metric values ​​of each output channel in the i-th layer are compared with the thresholds; finally, the output channels that need to be deleted in the i-th layer are determined, that is, the output channels corresponding to the metric values ​​of each output channel in the i-th layer that are less than the thresholds are deleted.

[0048] For example, such as Figure 3 As shown, assume that the measure value of each output channel in the i-th layer of the neural network to be pruned is M. i,m (For example Figure 3 M in i,6 M i,8 ), 1≤m≤n i+1 The metric values ​​M of each output channel of the i-th layer i,m Sort the data in ascending order, then the Sth digit of the sorted data... i ×n i+1 Each measure value M i,m′ As a threshold, the metric value M of each output channel of the i-th layer is... i,m Medium less than threshold M i,m′ Measure value (e.g.) Figure 3 M in i,6 M i,8 M i,20 M i,15 M i,33 Delete the corresponding output channels.

[0049] It should be understood that the metric value of each output channel can be used to characterize the importance of each output channel. For example, in the i-th layer of a neural network to be pruned, the output channel with a larger metric value is more important than the output channel with a smaller metric value.

[0050] It should be noted that the sparsity S of the neural network to be pruned i This is used to represent the sparsity of each layer in the neural network to be pruned, and generally, the sparsity S of each layer in the neural network to be pruned is... i All are equal, that is, S i =S, 1≤i≤L.

[0051] In this embodiment of the application, for the i-th layer in the neural network to be pruned, the filter F of each output channel can be... i,m Vectorized into filter vector V i,m For example, such as Figure 4 As shown, assume the number of output channels in the i-th layer is n. i+1 The output channels of the i-th layer are denoted as output channel 1, output channel 2, ..., output channel n, respectively. i+1 -1. Output channel n i+1 Then the filter F of output channel 1 can be adjusted. i,1 Filter F of output channel 2 i,2..., output channels n i+1 -1 filter Output channel n i+1 Filter Vectorization is performed to obtain the filter vector V for output channel 1. i,1 The filter vector V of output channel 2 i,2 ..., output channels n i+1 -1 filter vector Output channel n i+1 Filter vector Assume V i,m′ V is the filter vector for each output channel. i,m One of them, V can be calculated. i,m′ Other filter vectors {V i,m ,m∈[1,n i+1 The Euclidean distance (L2 distance) between each of ∩m≠m′} is calculated, and all Euclidean distances are summed to determine V. i,m′ The corresponding output channel metric value M i,m′ V i,m′ The corresponding output channel metric value M i,m′ The calculation formula is as follows:

[0052]

[0053] It should be understood that although all layers in the neural network to be pruned use the same sparsity, the number of output channels in each layer may not be the same. Therefore, the threshold for each layer needs to be calculated layer-wise. Some output channels may not be important in a particular layer (i.e., the measure value of the output channel determined by the aforementioned method is small), but that layer has a significant effect on the entire neural network to be pruned. Therefore, if we can step outside of that layer and determine which output channels in that layer can be deleted from the perspective of the entire neural network to be pruned, we can perform pruning better and more effectively. In other words, when pruning a layer, we need to retain important output channels and delete channels that have the least inference effect on the entire neural network and are truly redundant. This algorithm can also be called the Global Pruning algorithm.

[0054] It should also be understood that when pruning a certain layer from the perspective of the entire neural network to be pruned, the metric value of the output channel calculated by formula (3) is related to the number of output channels n of the layer where the output channel is located. i+1 It is quite sensitive when the number of output channels n of different layers i+1When the difference is significant, the measure values ​​of each output channel across different layers become incomparable. In deep neural networks, as the number of layers in the network to be pruned increases, the number of output channels n... i+1 This will also increase significantly, resulting in a large difference in the measure values ​​of each output channel between shallow and deep neural networks to be pruned. However, this does not mean that the output channels of shallow neural networks to be pruned are unimportant. On the contrary, some shallower layers in the neural network to be pruned, such as the first layer, can have a significant impact on the overall inference performance of the neural network.

[0055] In summary, the metric values ​​of output channels in related technologies are highly sensitive to the number of output channels in the layer containing that output channel. When the difference in the number of output channels between different layers is large, the difference between the metric values ​​of each output channel in the shallow and deep neural networks to be pruned is also large. Therefore, it is impossible to compare the metric values ​​of each output channel in the shallow and deep neural networks to be pruned. In this case, pruning layer by layer using the metric values ​​of each output channel of the neural network to be pruned does not result in significant inference performance of the pruned small model, and the accuracy of the pruned small model still needs to be improved.

[0056] Based on this, this application provides a method for pruning a neural network. First, the initial measure values ​​of each output channel in the i-th layer of the neural network to be pruned and the target parameter values ​​of the i-th layer are determined. Then, the initial measure values ​​of each output channel in the i-th layer are normalized according to the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel in the i-th layer. Finally, the i-th layer is pruned according to the target measure values ​​of each output channel in the i-th layer to obtain the pruned neural network. In this way, by normalizing the initial measure values ​​of each output channel in the i-th layer using the target parameter values ​​of the i-th layer, the sensitivity of the target measure values ​​of each output channel in the i-th layer to the target parameter values ​​of the i-th layer is reduced, allowing the target measure values ​​of each output channel in the i-th layer to be applied to global pruning for cross-layer comparisons. Thus, by pruning the i-th layer using the target measure values ​​of each output channel in the i-th layer, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly better inference performance, improving the accuracy of the pruned small model.

[0057] To facilitate understanding of the technical solutions of the embodiments of this application, the technical solutions of this application are described in detail below through specific embodiments. The above-mentioned related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0058] In one embodiment of this application, see Figure 5This illustrates a flowchart of a neural network pruning method provided in an embodiment of this application. Figure 5 As shown, the method may include the following steps.

[0059] S510, determine the initial measure values ​​of each output channel of the i-th layer in the neural network to be pruned and the target parameter values ​​of the i-th layer.

[0060] Where i is a positive integer.

[0061] It should be noted that, in the embodiments of this application, the neural network pruning method can be applied to a neural network pruning device or an electronic device integrating such a device. The electronic device can be implemented in various forms, such as smartphones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, digital cameras, etc., and is not limited thereto.

[0062] It should also be noted that each output channel in the i-th layer of the neural network to be pruned has its own initial measure value. When the number of output channels in the i-th layer is n... i+1 At that time, the number of initial measure values ​​for each output channel of the i-th layer is also n. i+1 For example, the output channels of the i-th layer can be denoted as output channel 1, output channel 2, ..., output channel n. i+1 -1. Output channel n i+1 Let M be the initial measure value of each output channel of the i-th layer. i,m , 1≤m≤n i+1 Then the initial measure value of output channel 1 is M. i,1 The initial measure value of output channel 2 is M. i,2 ..., output channels n i+1 The initial measure value of -1 is Output channel n i+1 The initial measure value is

[0063] It should also be noted that the target parameter value of the i-th layer in the neural network to be pruned can be the same as or different from the target parameter values ​​of other layers except the i-th layer. This application does not limit this.

[0064] In some embodiments, the target parameter value may include the number of output channels. For example, the target parameter value for the i-th layer is the number of output channels n of the i-th layer. i+1 .

[0065] It should be noted that the target parameter value can also be other values. There can be one or more target parameter values; this application does not limit the number of such values.

[0066] In some embodiments, determining the initial measure value of each output channel of the i-th layer in the neural network to be pruned may include: determining the filter vector of each output channel of the i-th layer; and determining the initial measure value of each output channel of the i-th layer based on the filter vector of each output channel of the i-th layer.

[0067] It should be noted that the filter vectors of each output channel in the i-th layer can be obtained by vectorizing the filters of each output channel in the i-th layer. For example, the filter F of each output channel in the i-th layer can be vectorized. i,m Vectorized into filter vector V i,m Thus, the filter vector V of output channel 1 is obtained. i,1 The filter vector V of output channel 2 i,2 ..., output channels n i+1 -1 filter vector Output channel n i+1 Filter vector Based on the filter vector V of each output channel of the i-th layer i,m This allows us to obtain the initial measure values ​​M for each output channel of the i-th layer. i,m .

[0068] Further, in some embodiments, determining the initial measure value of each output channel in the i-th layer based on the filter vector of each output channel in the i-th layer may include: calculating the distance between the filter vector of the first output channel and the filter vectors of each output channel in the i-th layer other than the first output channel; accumulating the calculated distances to determine the initial measure value of the first output channel; wherein the first output channel is one of the output channels in the i-th layer.

[0069] It should be noted that the distance between the filter vector of the first output channel and the filter vectors of each output channel in the i-th layer other than the first output channel can be Euclidean distance or other distances, and the embodiments of this application do not limit this.

[0070] For example, suppose the filter vector of the first output channel is V. i,m′ The filter vector V of the first output channel i,m′ The filter vectors {V} of each output channel in the i-th layer, excluding the first output channel. i,m ,m∈[1,n i+1 Taking the distance between ∩m≠m′} as the Euclidean distance as an example, the initial measure value M of the first output channel... i,m′The calculation formula is shown in the aforementioned formula (3). In addition, since the first output channel is one of the output channels of the i-th layer, the initial measure value for each output channel in the i-th layer can be calculated according to the aforementioned formula (3).

[0071] S520, normalize the initial measure values ​​of each output channel of the i-th layer according to the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel of the i-th layer.

[0072] For example, when the target parameter value of the i-th layer is the number of output channels n of the i-th layer. i+1 At that time, based on the number of output channels n of the i-th layer i+1 The initial measure value M of each output channel of the i-th layer can be obtained. i,m After normalization, the target metric values ​​of each output channel of the i-th layer are obtained.

[0073] For example, taking the first output channel as an example, the target metric value of the first output channel The calculation formula is as follows:

[0074]

[0075] It should be noted that when the target parameter value of the i-th layer is other values, the initial measure values ​​of each output channel of the i-th layer can also be normalized based on other values.

[0076] S530, the i-th layer is pruned according to the target metric values ​​of each output channel of the i-th layer to obtain the pruned neural network.

[0077] It should be noted that since the target metric values ​​of each output channel of the i-th layer are obtained by normalizing the initial metric values ​​of each output channel of the i-th layer using the target parameter values ​​of the i-th layer, the target metric values ​​of each output channel of the i-th layer are not sensitive to the target parameter values ​​of the i-th layer. In this case, the target metric values ​​of each output channel of the i-th layer can be applied to global pruning for cross-layer comparison.

[0078] This application provides a neural network pruning method. It normalizes the initial measure values ​​of each output channel of the i-th layer using the target parameter values ​​of the i-th layer, obtaining the target measure values ​​for each output channel of the i-th layer. This reduces the sensitivity of the target measure values ​​of each output channel of the i-th layer to the target parameter values ​​of the i-th layer, allowing the target measure values ​​of each output channel of the i-th layer to be applied to global pruning for cross-layer comparisons. Thus, by pruning the i-th layer using the target measure values ​​of each output channel of the i-th layer, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly improved inference performance and increases the accuracy of the pruned small model.

[0079] In another embodiment of this application, based on the neural network pruning method described in the foregoing embodiments, this application embodiment can prune the i-th layer according to the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network. For example... Figure 6 As shown, the method may include:

[0080] S610, Determine the target sparsity and number of parameters of the neural network to be pruned;

[0081] S620, determine the target threshold of the i-th layer based on the target sparsity, the number of parameters, and the target parameter values ​​of the i-th layer;

[0082] S630, the i-th layer is pruned according to the target threshold of the i-th layer and the target metric value of each output channel of the i-th layer to obtain the pruned neural network.

[0083] It should be noted that the target sparsity S of the neural network to be pruned global This is used to represent the sparsity of the neural network to be pruned. Generally, the target sparsity S of each layer in the neural network to be pruned is... i All are equal, and S global It can be equal to the target sparsity S of each layer in the neural network to be pruned. i S i =S global ,1≤i≤L.

[0084] It should also be noted that the number of parameters N in the neural network to be pruned weight This represents the sum of the number of parameters in each layer of the neural network to be pruned. The number of parameters in each layer of the neural network to be pruned may be equal or unequal, and this embodiment does not limit this.

[0085] For example, when the target parameter value of the i-th layer is the number of output channels n of the i-th layer. i+1 At that time, the target sparsity S of the neural network to be pruned can be used as a reference. global The number of parameters N in the neural network to be pruned weight and the number of output channels n of the i-th layer i+1 Determine the target threshold for the i-th layer.

[0086] For example, it can be based on the target threshold of the i-th layer. and the target metric values ​​of each output channel of the i-th layer Pruning is performed on the i-th layer to obtain the pruned neural network.

[0087] In some embodiments, determining the target threshold of the i-th layer based on the target sparsity, the number of parameters, and the target parameter value of the i-th layer may include: determining a global threshold based on the target sparsity and the number of parameters; determining an initial threshold based on the target sparsity and the target parameter value of the i-th layer; determining the minimum value between the global threshold and the initial threshold, and using the minimum value as the target threshold of the i-th layer.

[0088] For example, when the target parameter value of the i-th layer is the number of output channels n of the i-th layer. i+1 At that time, it can be based on the target sparsity S global And the number of parameters N weight Determine the global threshold t global It can be based on the target sparsity S global The initial threshold is determined based on the number of output channels in the i-th layer. At the global threshold t global and initial threshold The minimum value is determined in the middle, and the minimum value is used as the target threshold of the i-th layer. At this point, the target threshold of the i-th layer... The calculation formula is as follows:

[0089]

[0090] In some embodiments, determining a global threshold based on the target sparsity and the number of parameters may include: determining the target metric value of each output channel of all layers in the neural network to be pruned, wherein all layers in the neural network to be pruned include the i-th layer; performing a product calculation on the target sparsity and the number of parameters to determine a first product; and determining a global threshold based on the target metric value of each output channel of all layers and the first product.

[0091] It should be noted that the method for determining the target metric values ​​of each output channel in all layers is similar to the method for determining the target metric values ​​of each output channel in the i-th layer. Specifically, the initial metric values ​​of each output channel in each layer and the target parameter values ​​of each layer can be determined. Based on the target parameter values ​​of each layer, the initial metric values ​​of each output channel in that layer are normalized to obtain the target metric values ​​of each output channel in that layer.

[0092] It should also be noted that the first product is the target sparsity S. global And the number of parameters N weight The product between them, i.e., the first product, is S. global ×N weight .

[0093] Furthermore, determining the global threshold based on the target metric values ​​of each output channel of all layers and the first product can include: sorting the target metric values ​​of each output channel of all layers from smallest to largest to obtain a first sorting result; and using the p-th target metric value in the first sorting result as the global threshold.

[0094] Where p represents the first product, and p is a positive integer.

[0095] For example, the first product is S global ×N weight Assuming the Sth order in the first sorting result global ×N weight The target measure is 1≤c≤L, The global threshold t is used to represent the target metric value of the output channel k″ in the c-th layer. global for

[0096] In some embodiments, determining an initial threshold based on the target sparsity and the target parameter value of the i-th layer may include: multiplying the target sparsity and the target parameter value of the i-th layer to determine a second product; and determining the initial threshold based on the target metric value of each output channel of the i-th layer and the second product.

[0097] It should be noted that when the target parameter value of the i-th layer is n, the number of output channels of the i-th layer... i+1 When the second product is the target sparsity S, global and the number of output channels n of the i-th layer i+1 The product between them, i.e., the second product, is S. global ×n i+1 .

[0098] Further, determining the initial threshold based on the target metric value and the second product of each output channel of the i-th layer can include: sorting the target metric values ​​of each output channel of the i-th layer from smallest to largest to obtain a second sorting result; and using the q-th target metric value in the second sorting result as the initial threshold.

[0099] Where q represents the second product, and q is a positive integer.

[0100] For example, the second product is S global ×n i+1 Assuming the Sth order in the second sorting result global ×n i+1 The target measure is 1≤i≤L, The initial threshold is used to represent the target metric value of output channel k′ in the i-th layer. for

[0101] In some embodiments, pruning the i-th layer based on the target threshold of the i-th layer and the target metric values ​​of each output channel of the i-th layer may include: comparing the target threshold of the i-th layer with the target metric values ​​of each output channel of the i-th layer; if j target metric values ​​of each output channel of the i-th layer are less than the target threshold of the i-th layer, then deleting the output channels corresponding to each of the j target metric values.

[0102] Where j is an integer and 0 ≤ j < n i+1 n i+1 The number of output channels for the i-th layer.

[0103] It should be noted that if any of the target metric values ​​in the output channels of the i-th layer equals the target threshold of the i-th layer, whether or not to delete the output channel corresponding to that target metric value depends on the specific scenario. In some scenarios, it is necessary to delete the output channel corresponding to that target metric value; in other scenarios, it is not necessary to delete the output channel corresponding to that target metric value.

[0104] Furthermore, pruning the i-th layer based on the target threshold of the i-th layer and the target metric values ​​of each output channel of the i-th layer can be implemented in the following two possible ways.

[0105] One possible implementation is that if the global threshold is less than the initial threshold, the target threshold of the i-th layer is the global threshold, and the i-th layer can be pruned according to the global threshold and the target metric values ​​of each output channel of the i-th layer.

[0106] Specifically, the global threshold is compared with the target metric values ​​of each output channel of the i-th layer; if j target metric values ​​of each output channel of the i-th layer are less than the global threshold, then the output channels corresponding to the j target metric values ​​are deleted.

[0107] For example, suppose the target metric value of the first output channel is... It is one of j target measure values, that is, the target measure value of the first output channel. Less than the global threshold Then delete the first output channel.

[0108] Another possible implementation is that if the initial threshold is less than the global threshold, then the target threshold of the i-th layer is the initial threshold, and the i-th layer can be pruned according to the initial threshold and the target metric values ​​of each output channel of the i-th layer.

[0109] Specifically, the initial threshold is compared with the target metric values ​​of each output channel of the i-th layer; if j target metric values ​​of each output channel of the i-th layer are less than the initial threshold, then the output channels corresponding to the j target metric values ​​are deleted.

[0110] For example, suppose the target metric value of the first output channel is... It is one of j target measure values, that is, the target measure value of the first output channel. Less than the initial threshold Then delete the first output channel.

[0111] It should be noted that if the global threshold is equal to the initial threshold, the target threshold of the i-th layer can be either the global threshold or the initial threshold. In this case, the i-th layer can be pruned based on either the global threshold and the target metric values ​​of each output channel of the i-th layer, or it can be pruned based on either the initial threshold and the target metric values ​​of each output channel of the i-th layer. This application does not limit this.

[0112] This application provides a neural network pruning method. Based on the target sparsity, the number of parameters, and the target parameter values ​​of the i-th layer, a target threshold for the i-th layer is determined. The i-th layer is then pruned based on this target threshold and the target metric values ​​of each output channel, resulting in a pruned neural network. By pruning the i-th layer using its target threshold and the target metric values ​​of each output channel, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly improved inference performance and accuracy.

[0113] In another embodiment of this application, based on the neural network pruning method of the foregoing embodiments, the embodiments of this application can use the initial measure value M of each output channel of the i-th layer in the neural network to be pruned. i,m Improvements were made to enable global comparisons across layers. Specifically, considering the initial measure value M of each output channel in the i-th layer... i,m For the number of output channels n of the i-th layer i+1 It is quite sensitive, so the number of output channels n of the i-th layer can be used. i+1 The initial measure value M for each output channel of the i-th layer i,m After normalization, the target metric values ​​of each output channel of the i-th layer are obtained. At this time, the target metric values ​​of each output channel of the i-th layer The number of output channels n of the i-th layer i+1 The impact is relatively small. The target metric values ​​of each output channel in the i-th layer. The calculation formula is similar to the aforementioned formula (4).

[0114] In this embodiment of the application, the target metric value is based on the normalized output channel of the i-th layer. The following steps can be used to perform a cross-layer global comparison of the neural network to be pruned:

[0115] The first step is to set a target sparsity S for the entire neural network to be pruned. global And set the target sparsity S of each layer i =S global ,1≤i≤L;

[0116] The second step involves obtaining the initial threshold using the method described in the preceding embodiments.

[0117] The third step is to sort the target metric values ​​of each output channel of all layers in the neural network to be pruned from smallest to largest, and then sort them according to the target sparsity S. global And the number of parameters N weight The number of channels S to be deleted in the entire neural network to be pruned is obtained. global ×N weight The Sth sorted number global ×N weight Each target measure is used as the global threshold t. global ;

[0118] The fourth step is to process the i-th layer W... (i) When performing pruning, the target threshold used for pruning the i-th layer. It should be the initial threshold obtained in the second step. and the global threshold t obtained in the third step global The minimum value in the range, at which point the target threshold of the i-th layer is reached. The calculation formula is the aforementioned formula (5);

[0119] Fifth step, use the target threshold of the i-th layer in the aforementioned formula (5). Pruning can be performed on the i-th layer of the neural network to be pruned. Specifically, the target threshold of the i-th layer is compared with the target metric values ​​of each output channel of the i-th layer. If j target metric values ​​of each output channel of the i-th layer are less than the target threshold of the i-th layer, then the output channels corresponding to each of the j target metric values ​​are deleted, where j is an integer and 0 ≤ j < n. i+1 n i+1 The number of output channels for the i-th layer.

[0120] It should be noted that the embodiments of this application propose an improved target metric value, which can be normalized by using the number of output channels of the i-th layer to normalize the initial metric value of each output channel of the i-th layer, thereby obtaining a normalized target metric value based on the geometric median, and this target metric value can be used for global pruning of cross-layer comparisons.

[0121] It should also be noted that, in this embodiment, the initial measure value based on the geometric median is normalized to obtain a normalized target measure value based on the geometric median, which enables the target measure value to be subject to global pruning through cross-layer comparison. Moreover, by using global pruning, the inference performance of the pruned small model (i.e., the pruned neural network) is improved, so that the pruned small model can achieve similar accuracy to the unpruned large model (the neural network to be pruned) through a certain amount of fine-tuning.

[0122] It should also be noted that the initial measurement value of the output channel is normalized using the number of output channels in this embodiment. It is understood that other target parameter values ​​can also be used to normalize the initial measurement value of the output channel, and this embodiment does not limit this.

[0123] It should also be noted that the embodiments of this application used normalized target measure values ​​based on geometric median for cross-layer global comparison. It is understood that other methods can also be used in the global pruning of cross-layer comparison, and the embodiments of this application do not limit this.

[0124] It should also be noted that when pruning other layers in the neural network to be pruned, a method similar to the pruning method for the i-th layer provided in the embodiments of this application can also be used.

[0125] This application provides a neural network pruning method. It normalizes the initial measure values ​​of each output channel in the i-th layer using the number of output channels in the i-th layer, obtaining the target measure values ​​for each output channel. This reduces the sensitivity of the target measure values ​​of each output channel in the i-th layer to the number of output channels, allowing them to be applied to global pruning for cross-layer comparisons. Thus, by pruning the i-th layer using the target measure values ​​of each output channel, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly better inference performance, improving the accuracy of the pruned small model.

[0126] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Moreover, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application.

[0127] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0128] In another embodiment of this application, based on the same inventive concept as the foregoing embodiments, see [link to previous embodiment]. Figure 7 It shows a schematic diagram of the composition structure of the neural network pruning device provided in the embodiments of this application, such as... Figure 7 As shown, the neural network pruning device 700 may include a determining unit 710, a normalizing unit 720, and a pruning unit 730, wherein:

[0129] The determining unit 710 is configured to determine the initial measure values ​​of each output channel of the i-th layer in the neural network to be pruned and the target parameter values ​​of the i-th layer.

[0130] The normalization unit 720 is configured to normalize the initial measure values ​​of each output channel of the i-th layer according to the target parameter values ​​of the i-th layer, so as to obtain the target measure values ​​of each output channel of the i-th layer.

[0131] The pruning unit 730 is configured to prune the i-th layer according to the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network; where i is a positive integer.

[0132] In some embodiments, the target parameter value includes the number of output channels.

[0133] In some embodiments, the determining unit 710 is further configured to determine the target sparsity and the number of parameters of the neural network to be pruned; and to determine the target threshold of the i-th layer based on the target sparsity, the number of parameters and the target parameter value of the i-th layer; and the pruning unit 730 is further configured to prune the i-th layer based on the target threshold of the i-th layer and the target metric value of each output channel of the i-th layer to obtain the pruned neural network.

[0134] In some embodiments, the determining unit 710 is further configured to determine a global threshold based on the target sparsity and the number of parameters; determine an initial threshold based on the target sparsity and the target parameter value of the i-th layer; determine the minimum value among the global threshold and the initial threshold, and use the minimum value as the target threshold of the i-th layer.

[0135] In some embodiments, see Figure 7 The neural network pruning device 700 may further include a computing unit 740, wherein:

[0136] The determining unit 710 is further configured to determine the target metric value of each output channel of all layers in the neural network to be pruned, wherein all layers in the neural network to be pruned include the i-th layer;

[0137] The computing unit 740 is configured to perform a product calculation on the target sparsity and the number of parameters to determine the first product;

[0138] The determination unit 710 is also configured to determine the global threshold based on the target metric value of each output channel of all layers and the first product.

[0139] In some embodiments, see Figure 7 The neural network pruning device 700 may further include a sorting unit 750, wherein:

[0140] Sorting unit 750 is configured to sort the target metric values ​​of each output channel of all layers from smallest to largest to obtain the first sorting result;

[0141] The determining unit 710 is further configured to use the p-th target measure value in the first sorting result as a global threshold; where p represents the first product and p is a positive integer.

[0142] In some embodiments, the calculation unit 740 is further configured to perform a product calculation on the target sparsity and the target parameter value of the i-th layer to determine a second product; the determination unit 710 is further configured to determine an initial threshold based on the target metric value of each output channel of the i-th layer and the second product.

[0143] In some embodiments, the sorting unit 750 is further configured to sort the target metric values ​​of each output channel of the i-th layer from smallest to largest to obtain a second sorting result; the determining unit 710 is further configured to use the q-th target metric value in the second sorting result as an initial threshold; wherein q represents the second product and q is a positive integer.

[0144] In some embodiments, see Figure 7 The neural network pruning device 700 may further include a comparison unit 760 and a deletion unit 770, wherein:

[0145] The comparison unit 760 is configured to compare the target threshold of the i-th layer with the target metric values ​​of each output channel of the i-th layer;

[0146] Deletion unit 770 is configured such that if j target measure values ​​in each output channel of the i-th layer are less than the target threshold of the i-th layer, then the output channels corresponding to each of the j target measure values ​​are deleted, where j is an integer and 0 ≤ j < n. i+1 n i+1 The number of output channels for the i-th layer.

[0147] In some embodiments, the determining unit 710 is further configured to determine the filtering vector of each output channel of the i-th layer; and to determine the initial measure value of each output channel of the i-th layer based on the filtering vector of each output channel of the i-th layer.

[0148] In some embodiments, the calculation unit 740 is further configured to calculate the distance between the filter vector of the first output channel and the filter vectors of each output channel in the i-th layer other than the first output channel; accumulate the calculated distances to determine the initial measure value of the first output channel; wherein the first output channel is one of the output channels in the i-th layer.

[0149] This application provides a neural network pruning device that normalizes the initial measure values ​​of each output channel of the i-th layer using the target parameter values ​​of the i-th layer, thereby obtaining the target measure values ​​of each output channel of the i-th layer. This reduces the sensitivity of the target measure values ​​of each output channel of the i-th layer to the target parameter values ​​of the i-th layer, allowing the target measure values ​​of each output channel of the i-th layer to be applied to global pruning for cross-layer comparisons. Thus, by pruning the i-th layer using the target measure values ​​of each output channel of the i-th layer, the resulting pruned small model (i.e., the pruned neural network) exhibits significantly improved inference performance and increases the accuracy of the pruned small model.

[0150] Those skilled in the art should understand that the description of the neural network pruning device in the embodiments of this application can be understood with reference to the description of the neural network pruning method in the embodiments of this application.

[0151] In yet another embodiment of this application, Figure 8 This is a schematic structural diagram of an electronic device 800 provided in an embodiment of this application. Figure 8 The illustrated electronic device 800 includes a processor 810, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0152] In some embodiments, such as Figure 8 As shown, the electronic device 800 may further include a memory 820. The processor 810 can retrieve and run computer programs from the memory 820 to implement the methods described in the embodiments of this application.

[0153] The memory 820 can be a separate device independent of the processor 810, or it can be integrated into the processor 810.

[0154] In some embodiments, such as Figure 8 As shown, the electronic device 800 may also include a transceiver 830, which the processor 810 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0155] The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include an antenna, and the number of antennas may be one or more.

[0156] In some embodiments, this application also provides another electronic device composition, wherein the electronic device may include the neural network pruning device 700 described in any of the foregoing embodiments.

[0157] In yet another embodiment of this application, Figure 9 This is a schematic structural diagram of a chip provided in an embodiment of this application. Figure 9 The chip 900 shown includes a processor 910, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0158] In some embodiments, such as Figure 9 As shown, chip 900 may further include memory 920. Processor 910 can retrieve and run computer programs from memory 920 to implement the methods described in this embodiment.

[0159] The memory 920 can be a separate device independent of the processor 910, or it can be integrated into the processor 910.

[0160] In some embodiments, the chip 900 may further include an input interface 930. The processor 910 can control the input interface 930 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0161] In some embodiments, the chip 900 may further include an output interface 940. The processor 910 can control the output interface 940 to communicate with other devices or chips; specifically, it can output information or data to other devices or chips.

[0162] In some embodiments, the chip can be applied to the electronic device in the present application embodiments, and the chip can implement the corresponding processes implemented by the electronic device in the various methods of the present application embodiments. For the sake of brevity, it will not be described in detail here.

[0163] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0164] It is understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0165] It is also understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0166] It is also understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Dynch Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM), etc. In other words, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0167] This application also provides a computer-readable storage medium for storing computer programs.

[0168] In some embodiments, the computer-readable storage medium may be applied to the electronic device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0169] This application also provides a computer program product, including computer program instructions.

[0170] In some embodiments, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0171] This application also provides a computer program.

[0172] In some embodiments, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0173] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0178] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0180] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0181] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0182] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0183] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0184] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pruning method for neural networks, characterized in that, Applied to electronic devices, the method includes: Determine the initial measure values ​​of each output channel in the i-th layer of the neural network to be pruned, as well as the target parameter values ​​of the i-th layer; The initial measure values ​​of each output channel of the i-th layer are normalized based on the target parameter values ​​of the i-th layer to obtain the target measure values ​​of each output channel of the i-th layer. The i-th layer is pruned according to the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network; where i is a positive integer; the pruned neural network is used to determine the output result corresponding to the input data applied to at least one of the fields of machine vision, natural language processing, autonomous driving and robotics. The step of pruning the i-th layer based on the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network includes: Determine the target sparsity and number of parameters of the neural network to be pruned; The target threshold of the i-th layer is determined based on the target sparsity, the number of parameters, and the target parameter value of the i-th layer; The i-th layer is pruned based on the target threshold of the i-th layer and the target metric values ​​of each output channel of the i-th layer to obtain the pruned neural network.

2. The method according to claim 1, characterized in that, The target parameter value includes the number of output channels.

3. The method according to claim 1, characterized in that, The step of determining the target threshold of the i-th layer based on the target sparsity, the number of parameters, and the target parameter value of the i-th layer includes: A global threshold is determined based on the target sparsity and the number of parameters; Determine the initial threshold based on the target sparsity and the target parameter value of the i-th layer; The minimum value is determined between the global threshold and the initial threshold, and the minimum value is used as the target threshold of the i-th layer.

4. The method according to claim 3, characterized in that, The step of determining the global threshold based on the target sparsity and the number of parameters includes: Determine the target metric value of each output channel of all layers in the neural network to be pruned, wherein all layers in the neural network to be pruned include the i-th layer; The first product is determined by multiplying the target sparsity and the number of parameters. The global threshold is determined based on the target metric value of each output channel of all layers and the first product.

5. The method according to claim 4, characterized in that, The step of determining the global threshold based on the target metric value of each output channel of all layers and the first product includes: The target metric values ​​of each output channel of all layers are sorted from smallest to largest to obtain the first sorting result; The p-th target measure value in the first sorting result is used as the global threshold; where p represents the first product and p is a positive integer.

6. The method according to claim 3, characterized in that, The step of determining the initial threshold based on the target sparsity and the target parameter value of the i-th layer includes: The second product is determined by multiplying the target sparsity and the target parameter value of the i-th layer. The initial threshold is determined based on the target metric value of each output channel of the i-th layer and the second product.

7. The method according to claim 6, characterized in that, The step of determining the initial threshold based on the target metric value of each output channel of the i-th layer and the second product includes: The target metric values ​​of each output channel of the i-th layer are sorted from smallest to largest to obtain a second sorting result; The q-th target measure value in the second sorting result is used as the initial threshold; where q represents the second product and q is a positive integer.

8. The method according to claim 1, characterized in that, The pruning process for the i-th layer based on the target threshold of the i-th layer and the target metric values ​​of each output channel of the i-th layer includes: Compare the target threshold of the i-th layer with the target metric values ​​of each output channel of the i-th layer; If j target measure values ​​in each output channel of the i-th layer are less than the target threshold of the i-th layer, then delete the output channel corresponding to each of the j target measure values. Where j is an integer and , The number of output channels for the i-th layer.

9. The method according to claim 1, characterized in that, The determination of the initial measure values ​​for each output channel of the i-th layer in the neural network to be pruned includes: Determine the filter vector for each output channel of the i-th layer; Based on the filter vectors of each output channel of the i-th layer, determine the initial measure value of each output channel of the i-th layer.

10. The method according to claim 9, characterized in that, The step of determining the initial measure value of each output channel of the i-th layer based on the filter vector of each output channel of the i-th layer includes: Calculate the distance between the filter vector of the first output channel and the filter vectors of all output channels in the i-th layer except the first output channel; The calculated distances are accumulated to determine the initial measurement value of the first output channel; The first output channel is one of the output channels of the i-th layer.

11. A pruning device for a neural network, characterized in that, The pruning device of the neural network includes a determination unit, a normalization unit, and a pruning unit, wherein: The determining unit is configured to determine the initial measure value of each output channel of the i-th layer in the neural network to be pruned and the target parameter value of the i-th layer; The normalization unit is configured to normalize the initial measure values ​​of each output channel of the i-th layer according to the target parameter values ​​of the i-th layer, so as to obtain the target measure values ​​of each output channel of the i-th layer. The pruning unit is configured to prune the i-th layer according to the target metric values ​​of each output channel of the i-th layer to obtain a pruned neural network; where i is a positive integer; wherein the pruned neural network is used to determine the output result corresponding to the input data applied to at least one of the fields of machine vision, natural language processing, autonomous driving and robotics. The determining unit is configured to determine the target sparsity and the number of parameters of the neural network to be pruned; and to determine the target threshold of the i-th layer based on the target sparsity, the number of parameters, and the target parameter value of the i-th layer. The pruning unit is configured to prune the i-th layer according to the target threshold of the i-th layer and the target metric value of each output channel of the i-th layer, so as to obtain the pruned neural network.

12. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method as described in any one of claims 1 to 10.

13. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer storage medium stores a computer program that, when executed by at least one processor, implements the method as described in any one of claims 1 to 10.

Images