Optimization method for neural networks, computer system, and computer-readable medium

Abstract

The present application provides an optimization method, a computer system and a computer readable medium for a neural network. In the method, different two-pruning algorithms are used to prune the neural network sequentially. After each pruning algorithm prunes the neural network, the pruned neural network is retrained. In this way, the operation amount and parameter amount of the neural network can be reduced.

Description

Technical Field

[0001] This invention relates to a neural network technology, and more particularly to an optimization method for neural networks, a computer system, and a computer-readable medium. Background Technology

[0002] In recent years, with the rapid advancements in Artificial Intelligence (AI) technology, the number of parameters and computational complexity of neural network models have also increased significantly. Consequently, compression techniques for neural network models have flourished. Notably, pruning is a crucial technique for model compression. However, existing pruning methods are all single-type pruning approaches. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an optimization method, a computer system, and a computer-readable medium for neural networks, providing a hybrid pruning scheme to achieve model simplification.

[0004] The optimization method for neural networks according to embodiments of the present invention includes (but is not limited to) the following steps: sequentially pruning the neural network using different two-stage pruning algorithms; and retraining the pruned neural network in response to each pruning algorithm.

[0005] The computer system for neural networks according to embodiments of the present invention includes (but is not limited to) a memory and a processor. The memory stores code. The processor is coupled to the memory. The processor is configured to load and execute code to sequentially prune the neural network using different two-pruning algorithms, and to retrain the pruned neural network in response to each pruning algorithm's pruning of the neural network.

[0006] The non-transitory computer-readable storage medium of this invention is used to store code. The processor loads this code to execute the optimization method for neural networks as described above.

[0007] Based on the above, the optimization method for neural networks, computer system, and computer-readable medium according to the embodiments of the present invention utilize various pruning algorithms to achieve a deep learning neural network with low computational cost.

[0008] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a block diagram of a computer system according to an embodiment of the present invention.

[0010] Figure 2This is a flowchart of an optimization method for a neural network according to an embodiment of the present invention.

[0011] Figure 3 This is a flowchart of Channel Pruning via Geometric Median (CPGM) according to an embodiment of the present invention.

[0012] Figure 4 This is a flowchart of a slimming method according to an embodiment of the present invention.

[0013] Figure 5 This is a flowchart of a combination of slimming and ThiNet according to an embodiment of the present invention.

[0014] Figure 6 This is a flowchart of the combination of CPGM and ThiNet according to an embodiment of the present invention.

[0015] Figure 7 This is a schematic diagram of structural and non-structural pruning according to an embodiment of the present invention.

[0016] Figure 8 This is a schematic diagram of a user interface according to an embodiment of the present invention.

[0017] Explanation of symbols in the attached drawings:

[0018] 100. Computer system;

[0019] 110. Memory;

[0020] 130. Processor;

[0021] 150. Input device;

[0022] 170. Monitor;

[0023] S210~S220, S310~S330, S510~S550, S610~S640, steps;

[0024] 401. Redundant channels;

[0025] 402. Non-redundant channel;

[0026] BNT, batch standardization threshold;

[0027] 701. Redundant weights;

[0028] 702. Redundant channels;

[0029] 703. Unpruned passageway;

[0030] 801. Model Setup;

[0031] 802. Setting the pruning ratio;

[0032] 803. Quality threshold setting. Detailed Implementation

[0033] Figure 1 This is a block diagram of a computer system 100 according to an embodiment of the present invention. Please refer to... Figure 1 The computer system 100 includes (but is not limited to) a memory 110 and a processor 130. The computer system 100 may be a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, a medical or product testing instrument, or other computing device.

[0034] The memory 110 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar component. In one embodiment, the memory 110 is used to store code, software modules, configuration settings, data, or files (e.g., training samples, model parameters, pruned sets, or redundant channels).

[0035] Processor 130 is coupled to memory 110. Processor 130 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), neural network accelerator, or other similar components or combinations thereof. In one embodiment, processor 130 is used to execute all or part of the operations of computer system 100, and can load and execute various codes, software modules, files, and data stored in memory 110.

[0036] In some embodiments, the computer system 100 further includes an input device 150. The input device 150 may be a touch panel, mouse, keyboard, trackball, switch, or button. In one embodiment, the input device 150 is used to receive user actions, such as swiping, touching, pressing, or clicking.

[0037] In some embodiments, the computer system 100 further includes a display 170. The display 170 may be a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED), a quantum dot display, or other types of display. In one embodiment, the display 170 is used to display images. The content of the images may be a user interface.

[0038] The method described in the embodiments of the present invention will be explained below in conjunction with various devices, components and modules in the computer system 100. The various processes of this method may be adjusted according to the implementation situation, and are not limited thereto.

[0039] Figure 2 This is a flowchart of an optimization method for a neural network according to an embodiment of the present invention. Please refer to... Figure 2 The processor 130 sequentially prunes the neural network using different two-pruning algorithms (step S210). Specifically, this neural network is trained using a deep learning algorithm. Examples of deep learning algorithms include YOLO (You Only Look Once), AlexNet, ResNet, Region Based Convolutional Neural Networks (R-CNN), or Fast R-CNN. It should be noted that this neural network can be used for image classification, object detection, or other inferences, and this embodiment of the invention does not limit its application. The trained neural network may achieve a preset accuracy standard.

[0040] It's worth noting that trained neural networks have corresponding parameters at each layer (e.g., weights, number of channels, bias, or activation functions). It's conceivable that too many parameters can negatively impact computational efficiency. Pruning is one such compression technique used in neural networks. Pruning removes elements from a neural network that have little or no impact (e.g., channels, filters / kernels, feature maps, layers, neurons, or other parameters).

[0041] Unlike existing technologies, this invention proposes a hybrid pruning scheme. The two or more pruning algorithms can be channel, weight, filter, activation, gradient, hidden layer pruning, or pruning search methods. Compared to a single pruning scheme, this achieves the highest compression ratio and minimal accuracy loss.

[0042] In one embodiment, one of the multiple pruning algorithms used in this invention is a channel pruning (or filter pruning) algorithm. Examples of channel pruning algorithms include ThiNet, network slimming, filter pruning via geometric median (FPGM), or channel pruning via geometric median (CPGM).

[0043] For example, ThiNet prunes the current layer based on statistical information from the next layer, aiming to prune filters that have a small or minimal impact on the output of the current layer. Therefore, the pruned channels will result in a mean squared error less than the error threshold. Then, channels are pruned layer by layer. Finally, the remaining channels in each layer, approximating all channels, are obtained.

[0044] For example, Figure 3 This is a flowchart of channel pruning via geometric mean (CPGM) according to an embodiment of the present invention. Please refer to... Figure 3CPGM is based on Euclidean distances for filter weights and the search for redundant filter weights, as proposed by FPGM. First, processor 130 can set the norm ratio and distance ratio. The norm ratio is a ratio set according to the magnitude of the desired filter weights. In other words, the norm ratio is the ratio of the desired retained size to the total weights. For example, the norm ratio is 90% of the filter weights. The distance ratio is the ratio of the Euclidean distance to the median of the filter weights and the index of the channels to be removed. In other words, the distance ratio is the ratio representing the similarity between two filters / channels (distance is related to similarity). Processor 130 can sort the weights based on the set norm ratio and distance ratio (step S310). For example, the filter weights in the trained neural network are arranged from largest to smallest, and the top 90% of weights are selected based on the norm ratio. Processor 130 can determine the Euclidean distance of the filters (step S320). For example, in multidimensional space, the distance between any tensor and all filters. Next, processor 130 can determine similar filters and store the corresponding channel indices (step S330). For example, the point in each layer with the minimum sum of Euclidean distances is defined as the geometric median. If a filter is close to the geometric median of its layer in multidimensional space (i.e., closer represents greater similarity, and farther represents less similarity), then this filter can be considered informational redundancy and can be replaced. For example, this filter / channel is a redundant filter / channel and can be pruned. The index is the number representing the filter / channel. Processor 130 can assign one or more filter / channel indices to a pruning set. That is, this set includes the redundant filters / channels to be pruned.

[0045] For example, Figure 4 This is a flowchart of a slimming method according to an embodiment of the present invention. Please refer to... Figure 4The processor 130 sorts the scaling factors of channels in each batch normalization layer (layers 1 to N in the figure, where N is a positive integer). The scaling factor is multiplied by the output value of its corresponding channel during forward propagation. During the training of the neural network, the scaling factor is trained along with other weights and is subject to constraints (e.g., L1 norm sparsity penalties). It is worth noting that after training, if the scaling factor is small, the corresponding channel can be considered a redundant channel (classified as a pruning set); if the scaling factor is large, the corresponding channel is disabled / stopped / not considered a redundant channel. As shown in the figure, assuming the global batch normalization threshold (BNT) is 0.15, channels with scaling factors of 0.001, 0.035, 0.1, 0.0134, and 0.122 in the first layer are redundant channels 401, and all channels with the scaling factors listed in the second layer are also redundant channels 401. The channel whose proportional factor is greater than the batch normalization threshold BNT is the non-redundant channel 402. Then, the processor 130 can prune the redundant channel 401 and retain the non-redundant channel 402.

[0046] For example, based on both ThiNet and the greedy algorithm, processor 130 can feed in a validation dataset channel by channel and use the L2 norm function to compare the differences between the output feature maps of pruned and unpruned channels. If the difference is less than a difference threshold, the pruned channels can be considered redundant channels; if the difference is not less than the difference threshold, the pruned channels can be considered non-redundant channels. Unlike the traditional ThiNet, in this embodiment, the sparsity ratio fed to ThiNet is local sparsity, and the local sparsity used in each layer can be different.

[0047] In one embodiment, the channel pruning algorithm includes a first channel pruning algorithm and a second channel pruning algorithm. The processor 130 can obtain a first pruning set based on the first channel pruning algorithm. The first pruning set includes one or more (redundant) channels selected by the first channel pruning algorithm to be pruned. Furthermore, the processor 130 can obtain a second pruning set based on the second channel pruning algorithm. The second pruning set includes one or more (redundant) channels selected by the second channel pruning algorithm to be pruned. That is, the processor 130 uses different channel pruning algorithms to obtain corresponding pruning sets. Then, the processor 130 can determine one or more redundant channels to be pruned based on the first and second pruning sets. For example, it can take the intersection, union, or any of these pruning sets, or a specific number of channels within them. This provides a hybrid channel pruning scheme.

[0048] For example, Figure 5 This is a flowchart illustrating the combination of slimming and ThiNet according to an embodiment of the present invention. Please refer to... Figure 5 For the trained neural network, processor 130 can use a slimming method to determine the scaling factor threshold value of each layer (e.g., the aforementioned batch normalization threshold value) (step S510), and transform the global scaling factor threshold value to the local sparsity corresponding to the sparsity ratio of each layer. Processor 130 can determine the first pruning set of each layer based on the local sparsity of each layer (step S520). Next, processor 130 can use the ThiNet method to select the filter to be pruned based on the local sparsity (i.e., determine the second pruning set) (step S530). Processor 130 can determine the redundant channels to be pruned based on the intersection of the pruning sets of the slimming and ThiNet methods (step S540), and prune the redundant channels accordingly (step S550).

[0049] Figure 6 This is a flowchart illustrating the combination of CPGM and ThiNet according to an embodiment of the present invention. Please refer to... Figure 6 For the trained neural network, the processor 130 can determine the first pruning set using the CPGM method based on the set norm ratio and distance ratio (step S610), and determine the second pruning set using the ThiNet method based on the set distance ratio (step S620). It is worth noting that the ThiNet method focuses on finding the channel influence on the output feature map. The CPGM method, on the other hand, can not only prune the filter weights but also determine the difference between the redundant channel weights and the previous ones. Next, the processor 130 can determine the redundant channels to be pruned based on the intersection of the pruning sets from the ThiNet and CPGM methods (step S630), and prune the redundant channels accordingly (step S640).

[0050] It should be noted that in other embodiments, other channel pruning algorithms or more channel pruning algorithms can also be combined.

[0051] In one embodiment, another of the multiple pruning algorithms used in this invention is a weighted pruning (or element-wise pruning) algorithm. An example of a weighted pruning algorithm is the LotteryTicket Hypothesis.

[0052] Taking the lottery hypothesis as an example, processor 130 randomly initializes a neural network. This neural network includes multiple sub-networks. Processor 130 iteratively trains this neural network, searching for sub-networks that are more likely to win. During this process, processor 130 can create a mask to define a known pruning strategy. This strategy relates to which sub-networks influence the neural network—that is, those that are likely to win. Then, processor 130 can prune sub-networks that do not have a significant impact (those that do not win) based on the mask. For example, processor 130 can sort the weights by size and prune the smallest proportion or number of weights.

[0053] In one embodiment, in response to pruning the neural network using a channel pruning algorithm, the processor 130 may then prune the neural network using a weight pruning algorithm. The channel pruning algorithm is a structured pruning algorithm, while the weight pruning algorithm is an unstructured pruning algorithm. Because unstructured pruning is an irregular type of pruning, accuracy may be difficult to ensure. Prior structured pruning determines that the restored weights are stable values ​​and establishes the overall structure. Therefore, subsequent unstructured pruning can fine-tune the network to achieve better accuracy.

[0054] For example, Figure 7 This is a schematic diagram illustrating structural and non-structural pruning according to an embodiment of the present invention. Please refer to... Figure 7 The processor 130 can employ a structured pruning strategy to prune the trained neural network to obtain a pruned neural network. Then, the processor 130 can employ an unstructured pruning strategy to prune the pruned neural network to obtain a final pruned neural network. Specifically, the structured pruning strategy retains the undone channels 703 and removes redundant channels 702. The unstructured pruning strategy removes redundant weights 701.

[0055] In other embodiments, other unstructured pruning methods (e.g., gradients or stimuli) may also be used, or unstructured pruning may be performed first and structured pruning may be performed later.

[0056] In one embodiment, prior to pruning, processor 130 may reduce the scaling factor of one or more batch normalization layers of the neural network. For example, processor 130 may perform sparse training on the trained neural network. This involves adding an L1 penalty to the loss function used to train the neural network. Batch normalization normalizes each mini-batch until a normal distribution with a mean of 0 and a standard deviation of 1 is formed. Reducing the scaling factor improves the overall correlation between layers and helps, for example, slimming methods to find more suitable channels (e.g., higher accuracy and / or greater reduction).

[0057] In some embodiments, if the scaling factor of the trained neural network has approached a preset value, the processor 130 may omit sparse training or other schemes for converging the scaling factor.

[0058] Please refer to Figure 2 This process involves pruning the neural network using each pruning algorithm and retraining the pruned neural network (step S220). Specifically, after each pruning, the processor 130 can retrain the pruned neural network. Training continues until the neural network (model) converges. The processor 130 can then use another pruning algorithm to prune the pruned neural network. For example, after channel pruning, the neural network can be retrained until convergence before weight pruning. During weight pruning, after a certain number of iterations, the processor 130 can sort the weights from smallest to largest and then remove the smallest weights according to the pruning ratio. Finally, the processor 130 can initialize the remaining weights back to the parameters of the original pre-trained model and retrain the pruned neural network to produce the final lightweight model.

[0059] In other words, whenever a channel is pruned, the preserved channels are initialized, and then the parameters of these preserved channels are trained. Similarly, whenever a weight is pruned, the preserved weights are initialized, and then the parameters of these preserved weights are trained. Retraining for activation pruning, hidden layer pruning, or other pruning can be done in the same way, and will not be elaborated further here.

[0060] It should be noted that the above example uses a combination of two pruning algorithms, but other embodiments can combine more pruning algorithms.

[0061] In one embodiment, processor 130 may receive input operations via input device 150. The input operations are used to set a pruning ratio and select at least one of two or more pruning algorithms to prune according to this pruning ratio. That is, the pruning ratio is the proportion of elements (e.g., channels, weights, or stimuli) to be pruned in each layer or filter out of all elements. For example, a keyboard receives an input operation regarding a 50% pruning ratio. For channel pruning, processor 130 selects 50% of the channels within one or more layers as redundant channels. For weight pruning, processor 130 may remove 50% of the smallest weights.

[0062] In one embodiment, processor 130 can use validation samples with known inference results to determine the accuracy loss of the pruned neural network. For example, if 10 out of 100 validation samples have incorrect inferences, the accuracy loss is 10%. Processor 130 can compare the accuracy loss of the pruned neural network with a quality threshold. This quality threshold is the allowable accuracy loss. For example, the quality threshold is 15%, 20%, or 25%. Processor 130 can change the pruning ratio of at least one of those pruning algorithms based on the comparison result of the accuracy loss and the quality threshold. That is, the quality threshold is used to evaluate whether to change the pruning ratio. For example, if the accuracy loss is lower than the quality threshold, processor 130 can increase the pruning ratio. Or, if the accuracy loss is higher than the quality threshold, processor 130 can decrease the pruning ratio. In one embodiment, processor 130 can use the pruned neural network with the maximum pruning ratio and its accuracy loss lower than the quality threshold as the final lightweight model.

[0063] In one embodiment, input operations received via input device 150 can be used to set a quality threshold and select at least one of two or more pruning algorithms to prune according to this quality threshold. For example, the mouse receives an input operation regarding a quality threshold of 15%.

[0064] In one embodiment, the processor 130 may provide (display) a user interface via a display 170. For example, Figure 8 This is a schematic diagram of a user interface according to an embodiment of the present invention. Please refer to... Figure 8 The user interface includes model settings 801, pruning ratio settings 802, and quality threshold settings 803. Model settings 801 are used to select the type of neural network. For example, third-generation (V3) YOLO or Single Shot MultiBox Detector (SSD) Visual Geometry Group (VGG) 16. Pruning ratio settings 802 are used to select the pruning ratio. For example, 10% to 90%. Quality threshold settings 803 are used to set the quality threshold (i.e., the allowable error rate). For example, 0% to 20%.

[0065] This invention further provides a non-transitory computer-readable storage medium (e.g., hard disk, optical disk, flash memory, solid-state drive (SSD), etc.) for storing code. The processor 130 or other processor of the computer system 100 can load the code and execute corresponding processes of one or more optimized methods of this invention. These processes are described above and will not be repeated here.

[0066] In summary, the optimization method, computer system, and computer-readable medium for neural networks in the embodiments of the present invention employ hybrid pruning to reduce the overall computational load of the neural network. For example, combining channel and weight pruning algorithms can reduce the number of channels and weights. The embodiments of the present invention evaluate pruning strategies from the perspectives of both pruning ratio and accuracy to achieve the requirements of high pruning rate and high accuracy. Furthermore, the embodiments of the present invention provide a user interface, allowing operators to easily understand and become accustomed to the operation.

[0067] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the claims.

Claims

1. An optimization method for neural networks, characterized in that, The neural network is used for image classification and object detection, including: Provide a user interface; The user interface receives a decision on a pruning ratio or a quality threshold. A neural network is pruned sequentially using different two-stage pruning algorithms; and The pruning algorithm prunes the neural network for each iteration, and then retrains the pruned neural network. One of the two pruning algorithms is a one-channel pruning algorithm, which includes a first-channel pruning algorithm and a second-channel pruning algorithm. The steps of pruning the neural network sequentially using different of these two pruning algorithms include: A first pruning set is obtained based on the first channel pruning algorithm, wherein the first pruning set includes at least one channel selected by the first channel pruning algorithm to be pruned; A second pruning set is obtained based on the second-channel pruning algorithm, wherein the second pruning set is at least one channel selected by the second-channel pruning algorithm to be pruned; and At least one redundant channel is determined based on the intersection of the first pruning set and the second pruning set.

2. The optimization method for neural networks as described in claim 1, characterized in that, The other of these two pruning algorithms is a weighted pruning algorithm.

3. The optimization method for neural networks as described in claim 2, characterized in that, The steps for pruning the neural network sequentially using different pruning algorithms include: The neural network is pruned using the channel pruning algorithm and the neural network is pruned using the weight pruning algorithm.

4. The optimization method for neural networks as described in claim 1, characterized in that, Before the steps of pruning the neural network sequentially using different pruning algorithms, the following are further included: The scaling factor of at least one batch of regularization layers of the neural network converges.

5. The optimization method for neural networks as described in claim 1, characterized in that, Including: The algorithm receives an input operation, wherein the input operation is used to set a pruning ratio, and at least one of the two pruning algorithms prunes according to the pruning ratio.

6. The optimization method for neural networks as described in claim 1, characterized in that, Including: Compare the accuracy loss of the pruned neural network with a quality threshold; and The pruning ratio of at least one of the two pruning algorithms is changed based on the comparison result with the quality threshold value.

7. A computer system for neural networks, characterized in that, The neural network is used for image classification and object detection, including: A memory used to store a code; A processor, coupled to the memory, is configured to load and execute the code to: A neural network is pruned sequentially using different two-stage pruning algorithms; and The pruning algorithm prunes the neural network for each iteration, and then retrains the pruned neural network. A display, coupled to the processor, wherein The processor is further configured to: This display provides a user interface; and The user interface receives a decision on a pruning ratio or a quality threshold. One of the two pruning algorithms is a one-channel pruning algorithm, which includes a first-channel pruning algorithm and a second-channel pruning algorithm. The steps of pruning the neural network sequentially using different of these two pruning algorithms include: A first pruning set is obtained based on the first channel pruning algorithm, wherein the first pruning set includes at least one channel selected by the first channel pruning algorithm to be pruned; A second pruning set is obtained based on the second-channel pruning algorithm, wherein the second pruning set is at least one channel selected by the second-channel pruning algorithm to be pruned; and At least one redundant channel is determined based on the intersection of the first pruning set and the second pruning set.

8. The computer system for neural networks as described in claim 7, characterized in that, The other of these two pruning algorithms is a weighted pruning algorithm.

9. The computer system for neural networks as described in claim 8, characterized in that, The processor is further configured to: The neural network is pruned using the channel pruning algorithm and the neural network is pruned using the weight pruning algorithm.

10. The computer system for neural networks as described in claim 7, characterized in that, The processor is further configured to: The scaling factor of at least one batch of regularization layers of the neural network converges.

11. The computer system for neural networks as described in claim 7, characterized in that, At least one of the two pruning algorithms prunes according to the pruning ratio, and the quality threshold value is used to change the pruning ratio.

12. The computer system for neural networks as described in claim 7, characterized in that, The processor is further configured to: Compare the accuracy loss of the pruned neural network with a quality threshold; and The pruning ratio of at least one of the two pruning algorithms is changed based on the comparison result with the quality threshold value.

13. A non-transitory computer-readable storage medium, characterized in that, Used to store code, which a processor loads to execute the optimization method for a neural network as described in any one of claims 1 to 6.

Images