Pruning method and device of neural network
By dividing the neural network training process into stages and using the cosine function to calculate the pruning rate, a neural network model adapted to different terminal computing power is generated, solving the problem of multiple training in existing technologies and achieving efficient model adaptation and shortened training time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YINGCHE XINGCHUANG INTELLIGENT TECH (SHANGHAI) CO LTD
- Filing Date
- 2022-07-21
- Publication Date
- 2026-08-04
AI Technical Summary
Existing neural network pruning methods require multiple training sessions on multiple devices to obtain models that adapt to different computing power, resulting in wasted training time and low efficiency.
The training process of the neural network is divided into multiple stages. In each stage, a model pruning rate is generated based on the preset target pruning rate and the current number of training iterations. The pruning rate is calculated using a cosine function to generate different pruning configurations. The neural network is then trained and pruned to obtain a model that meets the computing power requirements of different terminals.
Multiple neural network models adapted to different terminal computing power are generated in one training process, saving training time. Furthermore, the pruned models do not require subsequent fine-tuning training, thus improving training efficiency.
Smart Images
Figure CN115358384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a method and apparatus for pruning neural networks. Background Technology
[0002] Deep learning has been widely adopted due to its remarkable effectiveness, but its massive computational demands limit its commercialization, especially on edge devices. Edge devices are often not designed for computationally intensive tasks, and simply deploying deep learning-based products there leads to issues with power consumption and latency. Even on servers, increased computation directly increases costs. Therefore, model compression is needed to significantly simplify large original models, resulting in smaller, more compact models with acceptable accuracy loss, thus meeting the computational cost constraints of different devices. Pruning is a common model compression method that removes redundant channels, filters, layers, or neurons to create a lightweight network without affecting performance. However, existing pruning methods only obtain a model with a predetermined pruning rate during the entire training process. Considering that pruned models often need to be deployed on different devices (such as GPUs or edge computing devices), and different devices have varying computational power and model acceleration requirements, multiple pruning training sessions are required to obtain pruned models for deployment on different devices, significantly wasting training time. Therefore, how to obtain a pruned model that meets the needs of different devices through a single pruning training has become an urgent problem to be solved. Summary of the Invention
[0003] This invention provides a method and apparatus for pruning neural networks to solve the above-mentioned problems.
[0004] This invention provides a method for pruning a neural network, comprising: acquiring training images and constructing a neural network model;
[0005] The training process of the neural network model is divided into multiple training stages;
[0006] In each training phase, the model pruning rate corresponding to each training phase is generated based on the preset target pruning rate and the current number of training iterations.
[0007] The pruning configuration corresponding to each training stage is generated based on the pruning rate of the model.
[0008] Based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output at each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power.
[0009] The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0010] According to a neural network pruning method provided by the present invention, the training process of the neural network model is divided into multiple training stages, including:
[0011] The training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed;
[0012] The number of training sessions for each training session is determined based on the number of training sessions and the preset total number of training sessions.
[0013] Accordingly, generating the model pruning rate for each training stage based on the preset target pruning rate and the current training iterations includes:
[0014] Based on the preset target pruning rate, the total number of training stages, the number of training sessions in each stage, and the current number of training sessions, the model pruning rate corresponding to each training stage is calculated using a cosine function.
[0015] According to a neural network pruning method provided by the present invention, the step of calculating the model pruning rate corresponding to each training stage by cosine processing based on a preset target pruning rate, the total number of training stages, the number of training stages, and the current number of training stages includes:
[0016] Obtain the ratio between the current number of training sessions and the number of training sessions in the current stage, and obtain the percentage of the ratio relative to the total number of training stages;
[0017] The proportion is transformed using the cosine function to obtain a first transformation value, which takes the value range of [-1, 1].
[0018] The first conversion value is incremented by one, and half of the first conversion value after incrementing by one is taken as the second conversion value, which has a range of [0,1].
[0019] The product of the second conversion value and the preset target pruning rate is obtained as the model pruning rate corresponding to each training stage.
[0020] According to a neural network pruning method provided by the present invention, the step of generating the pruning configuration corresponding to each training stage based on the model pruning rate includes:
[0021] Regularization is calculated for each filter in each layer of the neural network model to obtain the regularization value for each filter;
[0022] Calculate the mean of the regularization values of all filters in each network layer to obtain the mean regularization value of each network layer, and obtain the layer-level importance score of each network layer based on the mean regularization value of each network layer.
[0023] The regularization value of each filter is calculated by weighting the regularization value using the layer-level importance scores, thus obtaining the weighted regularization value of each filter;
[0024] All filters are sorted according to their weighted regularization values to obtain a filter sequence;
[0025] Based on the pruning rate of the model, a number of filters are determined from the filter sequence as filters to be pruned;
[0026] The pruning rate of each network layer is determined based on the number of filters to be pruned in each network layer, thus forming the pruning configuration corresponding to each training stage.
[0027] According to a neural network pruning method provided by the present invention, the step of training and pruning the neural network model based on the training image and using the pruning configuration to obtain the pruned neural network model output at each training stage includes:
[0028] The parameter mask for each network layer is generated based on the pruning rate of each network layer and the L2 regularization value of each filter in each network layer.
[0029] After the neural network model completes each training session based on the training images, an updated neural network model is obtained. The parameters of each layer in the updated neural network model are pruned using the parameter mask of each network layer, thereby obtaining the pruned neural network model output at each training stage.
[0030] According to a neural network pruning method provided by the present invention, the training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed, including:
[0031] The training process of the neural network model is divided into n training stages for iterative pruning and a training stage for fixed-format pruning, based on the number of terminals to be deployed.
[0032] Accordingly, obtaining the number of training sessions for each training session based on the number of training sessions and the preset total number of training sessions includes:
[0033] The training count for each training stage used for iterative pruning is obtained based on the number of training stages and the preset total number of training stages; wherein, the sum of the training counts for each of the n training stages used for iterative pruning and the training counts for the n training stages used for fixed-form pruning is the preset total number of training stages.
[0034] In each training phase, generating the model pruning rate corresponding to each training phase based on the preset target pruning rate and the current training iterations includes:
[0035] In each training phase for iterative pruning, the model pruning rate corresponding to each training phase for iterative pruning is generated based on the preset target pruning rate and the current number of training iterations.
[0036] The step of training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output at each training stage includes:
[0037] Based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output at each training stage for iterative pruning.
[0038] In the training phase for fixed-format pruning, the pruned neural network model output by the training phase for iterative pruning is trained and pruned using the pruning configuration corresponding to the training phase for iterative pruning, so as to obtain the pruned neural network model corresponding to the training phase for fixed-format pruning.
[0039] The pruned neural network model output from the training phase for fixed-format pruning, together with the pruned neural network models output from the training phase for iterative pruning, constitutes a set of pruned neural network models.
[0040] According to a neural network pruning method provided by the present invention, before generating the model pruning rate corresponding to each training stage based on a preset target pruning rate and the current training iterations, the method further includes:
[0041] If the current number of training iterations is divisible by the number of training iterations in a given stage, the model pruning rate for each training stage is generated based on the preset target pruning rate and the current number of training iterations.
[0042] If the current number of training iterations is not divisible by the number of training iterations in a given stage, the neural network model is pruned using the pruning configuration corresponding to each training stage, resulting in the pruned neural network model output from each training stage.
[0043] The present invention also provides a neural network pruning device, comprising: an image acquisition and model building module, used to acquire training images and build a neural network model;
[0044] The phase division module is used to divide the training process of the neural network model into multiple training phases;
[0045] The pruning rate generation module is used to generate the model pruning rate corresponding to each training stage based on the preset target pruning rate and the current number of training iterations.
[0046] The pruning configuration generation module is used to generate the pruning configuration corresponding to each training stage based on the model pruning rate.
[0047] The pruning model training module is used to train and prune the neural network model based on the training image and using the pruning configuration to obtain the pruned neural network model output at each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power.
[0048] The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0049] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the neural network pruning methods described above.
[0050] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a pruning method for any of the neural networks described above.
[0051] The neural network pruning method and apparatus provided by this invention generate a model pruning rate related to a preset target pruning rate and the current training iterations at each training stage. This allows for the generation of a pruned neural network model at each training stage. After the entire training process, a set of pruned neural network models is obtained, each of which meets the requirements of the terminal with the lowest computing power. This eliminates the need for a complete training pruning process for each terminal, significantly saving training time. Furthermore, each pruned neural network model can be directly deployed without subsequent fine-tuning training, further shortening training time and improving training efficiency. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0053] Figure 1 This is a flowchart illustrating the neural network pruning method provided in this embodiment of the invention. Figure 1 ;
[0054] Figure 2 This is a flowchart illustrating a neural network pruning method provided in another embodiment of the present invention. Figure 2 ;
[0055] Figure 3 This is a structural block diagram of the neural network pruning device provided in an embodiment of the present invention;
[0056] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0058] Figure 1 This is a flowchart illustrating the neural network pruning method provided in this embodiment of the invention. Figure 1 ;like Figure 1 As shown, a neural network pruning method includes the following steps:
[0059] S101: Acquire training images and build a neural network model.
[0060] In this step, the corresponding training images are determined based on the application scenario of the pruned model, and the corresponding neural network model is constructed. For example, in vehicle detection under computer vision, the corresponding training images are vehicle pictures, and the corresponding neural network model can be various commonly used vehicle detection models such as Mask R-CNN model, YOLOv3-TunnelVehicle model, etc. It can also be various commonly used neural network models in the language recognition scenario.
[0061] In this embodiment, the training images corresponding to 3D point cloud object detection are selected, and the constructed neural network model is the Centerpoint model (i.e., a Center-based 3D object detection and tracking model). This model uses keypoints for detection and regression in the first stage; the second stage corrects the detection boxes from the first stage, thereby improving the accuracy of 3D object detection. In other embodiments of the present invention, the training images and neural network models are set according to the application scenario and task requirements, and the present invention does not limit them.
[0062] S102, the training process of the neural network model is divided into multiple training stages.
[0063] In this step, the entire training process is divided into multiple training stages based on the preset number of training generations (i.e., epochs) during the neural network model training process and the number of terminals to be deployed (i.e., terminals that need to deploy the pruned model). For example, if the preset number of epochs is 40 and the number of terminals to be deployed is 4, then the entire training process can be divided into 4 training stages, each with 10 epochs; or it can be divided into 5 training stages, each with 8 epochs. That is, the number of training stages is greater than or equal to the number of terminals to be deployed, and the number of epochs corresponding to each training stage is an integer.
[0064] S103: In each training phase, the model pruning rate corresponding to each training phase is generated based on the preset target pruning rate and the current number of training iterations.
[0065] The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0066] In this step, in order to ensure that the multiple models obtained by pruning can meet the computing power requirements of all terminals to be deployed, it is necessary to limit a maximum pruning rate as the target pruning rate, so that the pruning rate of each pruned model generated during the pruning process is less than or equal to the maximum pruning rate, in order to meet the needs of different terminals to be deployed.
[0067] The pruning rate for each training phase is generated based on a preset target pruning rate and the current training iteration. It's important to note that the current training iteration refers to the training generation (i.e., the current epoch, hereinafter referred to as e_i), and generally e_i is the first epoch of each training phase. In other words, the pruning rate is calculated in the first epoch of each training phase and used to prune the model trained in each epoch of that phase. The specific method for generating the pruning rate is detailed below.
[0068] S104, Generate the pruning configuration corresponding to each training stage based on the model pruning rate.
[0069] In this step, the pruning rate of each network layer in the neural network model is determined as the pruning configuration based on the pruning rate calculated in S103.
[0070] S105, based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output at each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power.
[0071] In this step, the neural network model trained at the end of each training stage is pruned according to the pruning rate of each network layer. After multiple training stages are completed and pruned, multiple pruned neural network models are obtained, thus forming a set of pruned neural network models. Taking the above four training stages as an example, four neural network models with different pruning rates can be obtained.
[0072] Additionally, it should be noted that the neural network model trained and updated in the next training phase is based on the pruned model output from the previous training phase. That is, the pruned model output from the first training phase serves as the basis for training and pruning in the second training phase, and so on.
[0073] This invention provides a neural network pruning method that generates a model pruning rate related to a preset target pruning rate and the current training iterations at each training stage. This ensures that a pruned neural network model is generated at each training stage. After the entire training process, a set of pruned neural network models is obtained, each of which meets the requirements of the terminal with the lowest computing power. This eliminates the need for a complete training pruning process for each terminal, significantly saving training time. Furthermore, each pruned neural network model can be directly deployed without subsequent fine-tuning, further shortening training time and improving training efficiency.
[0074] Furthermore, the training process of the neural network model is divided into multiple training stages, including:
[0075] The training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed.
[0076] Specifically, if the number of terminals to be deployed is a, then the entire training process is divided into n training phases, where n ≥ a.
[0077] The number of training sessions for each training stage is obtained based on the number of terminals to be deployed and the preset total number of training sessions.
[0078] Specifically, the number of training sessions e_stage corresponding to each training stage is the preset total number of training sessions e_total_stage (it should be noted that the preset total number of training sessions e_total_stage here is only the total number of training sessions used for iterative pruning) and is the value after being evenly distributed according to the number of training stages n, that is, e_stage = e_total_stage / n, where e_stage is an integer.
[0079] Accordingly, generating the model pruning rate for each training stage based on the preset target pruning rate and the current training iterations includes:
[0080] Based on the preset target pruning rate, the total number of training stages, the number of training sessions in each stage, and the current number of training sessions, the model pruning rate corresponding to each training stage is calculated using a cosine function.
[0081] Specifically, the step of calculating the model pruning rate corresponding to each training stage based on the preset target pruning rate, the total number of training stages, the number of training iterations in each stage, and the current number of training iterations, through cosine processing, includes:
[0082] Obtain the ratio between the current number of training sessions and the number of training sessions in the current stage, and obtain the percentage of the ratio relative to the total number of training stages.
[0083] The proportion is transformed using the cosine function to obtain a first transformation value, which takes the value range of [-1, 1].
[0084] The first conversion value is incremented by one, and half of the first conversion value after incrementing by one is taken as the second conversion value, which has a range of [0,1].
[0085] The product of the second conversion value and the preset target pruning rate is obtained as the model pruning rate r_stage_i corresponding to each training stage, as shown in the following formula:
[0086]
[0087] In the formula, r_target is the preset target pruning rate, n is the total number of training stages, e_i is the current training iteration, e_stage is the stage training iteration, and r_stage_i is the model pruning rate at the i-th training iteration. Since the model pruning rate is calculated only once in each training stage, r_stage_i is also the model pruning rate at the [i / n]-th training stage.
[0088] This invention provides a neural network pruning method that generates different model pruning rates for each training stage based on a preset target pruning rate and the current training iterations, thereby obtaining multiple pruned neural network models. Furthermore, the generated model pruning rates are all less than or equal to the target pruning rate, ensuring that all pruned neural network models meet the computing power requirements of each deployment terminal, significantly saving training time.
[0089] Further, generating the pruning configuration corresponding to each training stage based on the model pruning rate includes:
[0090] Regularization is calculated for each filter in each layer of the neural network model to obtain the regularization value for each filter.
[0091] In this embodiment, the L1Norm value (i.e., the filter's regularization value) of each filter in each network layer is calculated. In other embodiments of the present invention, the L2Norm value of the filter can also be calculated, and the present invention does not limit this.
[0092] It should be noted that when a network layer to be pruned is specified, regularization is performed only on each filter in the pruned network layer; when no network layer to be pruned is specified, regularization is performed on each filter in all network layers.
[0093] Calculate the mean of the regularization values of all filters in each network layer to obtain the mean regularization value of each network layer, and obtain the layer-level importance score of each network layer based on the mean regularization value of each network layer.
[0094] Specifically, based on the L1Norm value of the filter obtained above, the average value of the L1Norm value of all filters in each network layer is calculated, and the average value is subjected to a softmax transformation to obtain the layer-level importance score_L.
[0095] The regularization value of each filter is calculated by weighting the regularization value using the layer-level importance score, thus obtaining the weighted regularization value of each filter.
[0096] Taking the L1Norm value mentioned above as an example, the weighted regularization value here is L1Norm*score_L.
[0097] All filters are sorted according to their weighted regularization values to obtain a filter sequence.
[0098] Specifically, all filters (i.e. filters in all network layers) are sorted in ascending order of their weighted regularization values to form a filter sequence.
[0099] Based on the pruning rate of the model, multiple filters are determined from the filter sequence as filters to be pruned.
[0100] Specifically, the top r_stage_i*N filters in the filter sequence are identified as filters to be pruned, where N is the total number of filters.
[0101] The pruning rate of each network layer is determined based on the number of filters to be pruned in each network layer, thus forming the pruning configuration corresponding to each training stage.
[0102] Specifically, after determining the filters to be pruned, it is further determined which network layer each filter belongs to. Finally, the number of filters to be pruned in each network layer is counted, and the pruning rate of each layer is calculated based on the number of filters to be pruned. The pruning rates of each network layer constitute the entire pruning configuration.
[0103] The present invention provides a neural network pruning method that refines the pruning rate of each network layer based on the model pruning rate generated in each training stage, thereby greatly reducing redundancy while keeping the accuracy loss within a small range.
[0104] Further, the step of training and pruning the neural network model based on the training images and utilizing the pruning configuration to obtain the pruned neural network model output at each training stage includes:
[0105] The parameter mask for each network layer is generated based on the pruning rate of each network layer and the L2 regularization value of each filter in each network layer.
[0106] After the neural network model completes each training session based on the training images, an updated neural network model is obtained. The parameters of each layer in the updated neural network model are pruned using the parameter mask of each network layer, thereby obtaining the pruned neural network model output at each training stage.
[0107] In this embodiment, a mask is generated for the parameters of each layer of the model according to the pruning configuration. The mask value is 0 or 1, where 0 indicates that the parameter at the corresponding position will be pruned, and 1 indicates that the parameter at the corresponding position will be retained. On a training phase basis, after the model has completed one epoch of training, the above mask is multiplied by the network parameters of each layer to achieve pruning. In each training phase, the model trained in each epoch undergoes the above-described pruning process, and the model pruned in the last epoch of this training phase is used as the pruned neural network model output for this training phase.
[0108] The parameter mask is generated based on the importance of the filters in each layer. The importance is determined by the square root of the L2 regularization value (L2Norm) of each filter. Specifically, the smaller the square root of the L2 regularization value, the less important it is; conversely, the larger the square root of the L2 regularization value, the more important it is.
[0109] The mask generation process involves determining the number of filters to be pruned in each network layer based on the pruning rate of each layer in the pruning configuration. This number is designated as the number of filters to be pruned (num_prune). The square root of the L2 regularization value for each filter is calculated, and the filters are sorted in ascending order of their L2 regularization values. The top num_prune filters are selected as the filters to be pruned, and their parameter masks are set to 0. The parameter masks for the remaining filters are set to 1, thus completing the parameter mask generation.
[0110] This invention provides a neural network pruning method that generates parameter masks for each network layer based on the importance of filters, thereby further reducing model redundancy and ensuring model accuracy.
[0111] In another embodiment of the present invention, since the pruning rate of the model generated in the last training stage is usually large, more training iterations are needed for adjustment. Therefore, the entire training process is divided into a training stage for iterative pruning and a training stage for fixed-format pruning, as detailed below.
[0112] The training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed, including:
[0113] The training process of the neural network model is divided into n training stages for iterative pruning (i.e., multi_stage) and one training stage for fixed-format pruning (i.e., freeze_stage) based on the number of terminals to be deployed. That is, n training stages for iterative pruning and one training stage for fixed-format pruning.
[0114] Accordingly, obtaining the number of training iterations for each training stage based on the number of terminals to be deployed and the preset total number of training iterations includes:
[0115] The number of training iterations for each training stage used for iterative pruning is obtained based on the number of terminals to be deployed and the preset total number of training iterations.
[0116] The sum of the number of training sessions for each of the n training sessions used for iterative pruning and the number of training sessions for the n training sessions used for fixed-form pruning is the preset total number of training sessions.
[0117] For example, if the preset total number of training iterations (i.e., the total number of epochs) is e_total, then the number of training iterations corresponding to the n training stages for iterative pruning can be set to 0.75*e_total, and the number of training iterations corresponding to the training stages for fixed-form pruning is 0.25*e_total. Furthermore, the number of training iterations corresponding to each training stage for iterative pruning is 0.75*e_total / n.
[0118] It should be noted that the 0.75*e_total and 0.25*e_total mentioned above are just illustrative examples. The specific ratio of training stages for iterative pruning to those for fixed-format pruning can be adjusted according to the actual situation. Furthermore, the neural network model first undergoes n training stages for iterative pruning, followed by one training stage for fixed-format pruning; the training stages proceed in a specific order.
[0119] In each training phase, generating the model pruning rate corresponding to each training phase based on the preset target pruning rate and the current training iterations includes:
[0120] In each training phase used for iterative pruning, a model pruning rate is generated based on a preset target pruning rate and the current number of training iterations. That is, the generation of the corresponding model pruning rate for each training phase, as mentioned above, only occurs in the training phase used for iterative pruning; it is not performed in the training phase used for fixed-form pruning.
[0121] The step of training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output at each training stage includes:
[0122] Based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output for each training phase of iterative pruning.
[0123] In the training phase for fixed-format pruning, the pruned neural network model output by the nth training phase for iterative pruning is trained and pruned using the pruning configuration corresponding to the nth training phase for iterative pruning, thus obtaining the pruned neural network model corresponding to the training phase for fixed-format pruning.
[0124] Specifically, after the neural network model undergoes n training phases for iterative pruning, a multi_stage output model is obtained. In the subsequent freeze_stage, the multi_stage output model is first used to perform 0.25*e_total training pruning cycles (taking the number of training cycles corresponding to the above-mentioned format pruning training phase as an example, which is 0.25*e_total), thus obtaining the final pruned neural network model. In other words, in the freeze_stage, after the model is updated after one epoch of training, it is pruned using the pruning configuration generated in the nth training phase for iterative pruning, and the model after the last epoch of training pruning is used as the final pruned neural network model.
[0125] The pruned neural network model output from the training phase for fixed-format pruning, together with the pruned neural network models output from the training phase for iterative pruning, constitutes a set of pruned neural network models. That is, the pruned neural network model output from each of the preceding training phases for iterative pruning, along with the final pruned neural network model, forms the set of pruned neural network models.
[0126] The present invention provides a neural network pruning method that can further improve the training effect of the final pruning model by setting a training stage for fixed-format pruning. This is because the final pruning model usually corresponds to a higher pruning level, which requires more training iterations to train better, and no further fine-tuning is needed.
[0127] Furthermore, before generating the model pruning rate corresponding to each training stage based on the preset target pruning rate and the current training iterations, the method further includes:
[0128] If the current training iterations e_i and the stage training iterations e_stage are divisible, the model pruning rate corresponding to each training stage is generated based on the preset target pruning rate and the current training iterations.
[0129] Specifically, in n training phases used for iterative pruning, if the current training iteration e_i % the phase training iteration e_stage == 0, then a model pruning rate is generated based on the preset target pruning rate and the current training iteration. That is, if the phase training iteration e_stage is 10, then a model pruning rate is generated only when the current training iteration e_i is an integer multiple of 10. In other words, in each multi_stage, a model pruning rate is generated only once, and this model pruning rate is used for model pruning after each epoch of training in that multi_stage.
[0130] It should be noted that in this embodiment, the first training stage is a special case in the specific division process of the training phase. Assuming the number of training stages, e_stage, is 10, and there are a total of 4 training stages, then the first epoch to the 9th epoch constitute the first training stage, the 10th epoch to the 19th epoch constitute the second training stage, the 20th epoch to the 29th epoch constitute the third training stage, and the 30th epoch to the 39th epoch constitute the fourth training stage. At the beginning of the second, third, and fourth training stages, the model pruning rate is calculated. That is, when e_i = 20 / 30 / 40, the condition e_i % e_stage == 0 is satisfied. Then, a pruning configuration is generated based on the generated pruning rate, and this pruning configuration is used to prune the training model at each epoch in the training phase.
[0131] The number of training iterations e_stage for the first stage is 9, which is one epoch less than the 10 for the second training stage, etc. This is because the training pruning in this embodiment is performed on the pre-trained model, and the number of training iterations for the first stage is slightly adjusted. Under this condition, the first training stage does not satisfy e_i%e_stage==0 when e_i=1, but training pruning is still required in the first training stage. Therefore, a judgment condition e_i=1 is added here, meaning that the pruning rate also needs to be calculated at the very beginning of the first training stage.
[0132] The above examples illustrate that the number of training cycles e_stage in the first training phase may be adjusted according to the actual situation of the selected model, and the starting condition for the next stage will also be adjusted accordingly. However, this does not affect the fact that the model pruning rate is calculated and generated at the beginning of each training phase and used for pruning in each training session. For example, if the number of training cycles e_stage in the first training phase is the same as that in other training phases, the criterion for starting a new training phase is e_i % e_stage == 1, that is, whether the remainder of e_i % e_stage is 1; if the number of training cycles e_stage in the first training phase is 2 epochs less than that in other training phases, the criterion for starting a new training phase is e_i = 1 or (e_i + 1) % e_stage == 0. Therefore, this invention does not limit the specific starting condition for a new training phase, which can change with the number of training cycles in the first training phase, as long as the model pruning rate is calculated once at the beginning of each training phase.
[0133] Then, after the model completes each training iteration, pruning is performed based on the model's pruning rate.
[0134] When the current training iterations e_i and the stage training iterations e_stage are not divisible, the neural network model is pruned using the pruning configuration corresponding to each training stage, resulting in the pruned neural network model output for each training stage. In other words, when the current training iterations e_i and the stage training iterations e_stage are not divisible, the neural network model is directly pruned without calculating the pruning rate or performing actual pruning. Furthermore, pruning is performed after each epoch of training, and the model is trained again in the next epoch based on the pruned model from the previous epoch. This process is repeated to output the pruned neural network model for each training stage.
[0135] Figure 2 This is a flowchart illustrating a neural network pruning method provided in another embodiment of the present invention. Figure 2 ;like Figure 2 As shown, a neural network pruning method includes the following steps:
[0136] S201 sets the target pruning rate and divides the training process into two main stages: the iterative pruning stage multi_stage and the fixed-format pruning stage freeze_stage. The iterative pruning stage multi_stage is further divided into n stages.
[0137] S202, in the iterative pruning stage multi_stage, first determine whether to start a new stage, that is, determine whether e_i%e_stage==0? If a new stage is started, proceed to step S203; if no new stage is started, proceed to step S205.
[0138] S203, Generate the model pruning rate r_stage_i corresponding to each training stage based on the target pruning rate and the current number of training iterations;
[0139] S204, Generate the pruning configuration for each layer in the current stage based on the model pruning rate r_stage_i;
[0140] S205, Train the pruned neural network model according to the pruning configuration of the current stage, train for e_total / n epochs until the model fully converges, and save it as the pruned neural network model output of the current stage.
[0141] S206, repeat S202-S205 until each stage outputs the pruned neural network model;
[0142] S207, In the fixed-format pruning stage freeze_stage, the pruning configuration generated in the last stage of the iterative pruning stage multi_stage is used to train and prune the model in the fixed-format pruning stage freeze_stage. The model obtained in the last epoch is used as the pruned neural network model output by the fixed-format pruning stage freeze_stage.
[0143] S208, the pruned neural network model output from S205 and the pruned neural network model output from S207 are combined into a set of pruned neural network models.
[0144] The neural network pruning method provided in this invention generates a model pruning rate related to a preset target pruning rate and the current training iterations at each training stage. This ensures that a pruned neural network model is generated at each training stage. After the entire training process is completed, a set of pruned neural network models is obtained. Each pruned neural network model can meet the requirements of the terminal with the lowest computing power, thus eliminating the need for a complete training pruning process for each terminal, significantly saving training time. Furthermore, each pruned neural network model can be directly deployed without subsequent fine-tuning training, further shortening training time and improving training efficiency.
[0145] The neural network pruning device provided by the present invention will be described below. The neural network pruning device described below can be referred to in correspondence with the neural network pruning method described above.
[0146] Figure 3 This is a structural block diagram of the neural network pruning device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, a neural network pruning device includes an image acquisition and model building module 301, a stage division module 302, a pruning rate generation module 303, a pruning configuration generation module 304, and a pruning model training module 305.
[0147] The image acquisition and model building module 301 is used to acquire training images and build neural network models.
[0148] In this module, the corresponding training images are determined based on the application scenario of the pruned model, and the corresponding neural network model is constructed. For example, in vehicle detection under computer vision, the corresponding training images are vehicle pictures, and the corresponding neural network model can be various commonly used vehicle detection models such as Mask R-CNN model, YOLOv3-TunnelVehicle model, etc. It can also be various commonly used neural network models in the language recognition scenario.
[0149] In this embodiment, the training images corresponding to 3D point cloud object detection are selected, and the constructed neural network model is the Centerpoint model (i.e., a Center-based 3D object detection and tracking model). This model uses keypoints for detection and regression in the first stage; the second stage corrects the detection boxes from the first stage, thereby improving the accuracy of 3D object detection. In other embodiments of the present invention, the training images and neural network models are set according to the application scenario and task requirements, and the present invention does not limit them.
[0150] The phase division module 302 is used to divide the training process of the neural network model into multiple training phases.
[0151] In this module, the entire training process is divided into multiple training stages based on the preset number of training generations (i.e., epochs) and the number of terminals to be deployed (i.e., terminals that need to deploy the pruned model). For example, if the preset number of epochs is 40 and the number of terminals to be deployed is 4, the entire training process can be divided into 4 training stages, each with 10 epochs; or it can be divided into 5 training stages, each with 8 epochs. In other words, the number of training stages is greater than or equal to the number of terminals to be deployed, and the number of epochs corresponding to each training stage is an integer.
[0152] The pruning rate generation module 303 is used to generate the model pruning rate corresponding to each training stage based on the preset target pruning rate and the current number of training iterations.
[0153] The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0154] In this module, in order to ensure that the multiple models obtained by pruning can meet the computing power requirements of all terminals to be deployed, it is necessary to limit a maximum pruning rate so that the pruning rate of each pruned model generated during the pruning process is less than or equal to the maximum pruning rate, so as to meet the needs of different terminals to be deployed.
[0155] The pruning rate for each training phase is generated based on the preset target pruning rate and the current training iteration. It's important to note that the current training iteration refers to the training epoch (i.e., the current epoch, hereinafter referred to as e_i). Generally, e_i represents the last epoch of each training phase. Taking the four training phases mentioned above as an example, e_i represents the 10th epoch, the 20th epoch, the 30th epoch, and the 40th epoch. In other words, the pruning rate is calculated for the last epoch of each training phase and used to prune the model generated at each epoch.
[0156] The pruning configuration generation module 304 is used to generate the pruning configuration corresponding to each training stage based on the model pruning rate.
[0157] In this module, the pruning rate of each network layer in the neural network model is determined as the pruning configuration based on the pruning rate calculated in the pruning rate generation module 303.
[0158] The pruning model training module 305 is used to train and prune the neural network model based on the training image and using the pruning configuration to obtain the pruned neural network model output at each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power.
[0159] In this module, the neural network model trained at the end of each training stage is pruned according to the pruning rate of each network layer. After multiple training stages are completed and pruned, multiple pruned neural network models are obtained, thus forming a set of pruned neural network models. Taking the above four training stages as an example, four neural network models with different pruning rates can be obtained.
[0160] Additionally, it should be noted that the neural network model trained and updated in the next training phase is based on the pruned model output from the previous training phase. That is, the pruned model output from the first training phase serves as the basis for training and pruning in the second training phase, and so on.
[0161] The neural network pruning device provided in this invention generates a model pruning rate related to a preset target pruning rate and the current training iterations at each training stage. This allows for the generation of a pruned neural network model at each training stage. After the entire training process, a set of pruned neural network models is obtained, each capable of meeting the needs of the terminal with the lowest computing power. This eliminates the need for a complete training pruning process for each terminal, significantly saving training time. Furthermore, each pruned neural network model can be directly deployed without subsequent fine-tuning, further shortening training time and improving training efficiency.
[0162] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a neural network pruning method, which includes: acquiring training images and constructing a neural network model; dividing the training process of the neural network model into multiple training stages; in each training stage, generating a model pruning rate corresponding to each training stage based on a preset target pruning rate and the current training iterations; generating a pruning configuration corresponding to each training stage based on the model pruning rate; training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output in each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used for deployment on terminals with different computing power. The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0163] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0164] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the neural network pruning method provided by the above-described method. The neural network pruning method includes: acquiring training images and constructing a neural network model; dividing the training process of the neural network model into multiple training stages; in each training stage, generating a model pruning rate corresponding to each training stage based on a preset target pruning rate and the current number of training iterations; generating a pruning configuration corresponding to each training stage based on the model pruning rate; training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output in each training stage, thereby forming a set of pruned neural network models. Each pruned neural network model in the set of pruned neural network models has a different pruning rate and is used for deployment on terminals with different computing power. The preset target pruning rate is determined based on the minimum computing power of each terminal.
[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method of pruning a neural network, the method comprising: include: Acquire training images and build a neural network model; The training process of the neural network model is divided into multiple training stages; In each training phase, the model pruning rate corresponding to each training phase is generated based on the preset target pruning rate and the current number of training iterations. The pruning configuration corresponding to each training stage is generated based on the pruning rate of the model. Based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output in each training stage, thereby forming a set of pruned neural network models. The neural network model trained and updated in the next training stage is based on the pruned model output in the previous training stage. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power. The preset target pruning rate is determined based on the minimum computing power of each terminal.
2. The pruning method of a neural network according to claim 1, characterized in that, The training process of the neural network model is divided into multiple training stages, including: The training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed; The number of training sessions for each training session is determined based on the number of training sessions and the preset total number of training sessions. Accordingly, generating the model pruning rate for each training stage based on the preset target pruning rate and the current training iterations includes: Based on the preset target pruning rate, the total number of training stages, the number of training sessions in each stage, and the current number of training sessions, the model pruning rate corresponding to each training stage is calculated using a cosine function.
3. The method of pruning a neural network according to claim 2, wherein, The step of calculating the model pruning rate for each training stage based on the preset target pruning rate, the total number of training stages, the number of training iterations in each stage, and the current number of training iterations, using cosine processing, includes: Obtain the ratio between the current number of training sessions and the number of training sessions in the current stage, and obtain the percentage of the ratio relative to the total number of training stages; The proportion is transformed using the cosine function to obtain a first transformation value, which takes the value range of [-1, 1]. The first conversion value is incremented by one, and half of the first conversion value after incrementing by one is taken as the second conversion value, which has a range of [0,1]. The product of the second conversion value and the preset target pruning rate is obtained as the model pruning rate corresponding to each training stage.
4. The method of pruning a neural network according to claim 1, wherein, The step of generating the pruning configuration corresponding to each training stage based on the model pruning rate includes: Regularization is calculated for each filter in each layer of the neural network model to obtain the regularization value for each filter; Calculate the mean of the regularization values of all filters in each network layer to obtain the mean regularization value of each network layer, and obtain the layer-level importance score of each network layer based on the mean regularization value of each network layer. The regularization value of each filter is calculated by weighting the regularization value using the layer-level importance scores, thus obtaining the weighted regularization value of each filter; All filters are sorted according to their weighted regularization values to obtain a filter sequence; Based on the pruning rate of the model, a number of filters are determined from the filter sequence as filters to be pruned; The pruning rate of each network layer is determined based on the number of filters to be pruned in each network layer, thus forming the pruning configuration corresponding to each training stage.
5. The method of pruning a neural network according to claim 4, wherein, The step of training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output at each training stage includes: The parameter mask for each network layer is generated based on the pruning rate of each network layer and the L2 regularization value of each filter in each network layer. After the neural network model completes each training session based on the training images, an updated neural network model is obtained. The parameters of each layer in the updated neural network model are pruned using the parameter mask of each network layer, thereby obtaining the pruned neural network model output at each training stage.
6. The method of pruning a neural network according to claim 2, wherein, The training process of the neural network model is divided into multiple training stages based on the number of terminals to be deployed, including: The training process of the neural network model is divided into n training stages for iterative pruning and a training stage for fixed-format pruning, based on the number of terminals to be deployed. Accordingly, obtaining the number of training sessions for each training session based on the number of training sessions and the preset total number of training sessions includes: The training count for each training stage used for iterative pruning is obtained based on the number of training stages and the preset total number of training stages; wherein, the sum of the training counts for each of the n training stages used for iterative pruning and the training counts for the n training stages used for fixed-form pruning is the preset total number of training stages. In each training phase, generating the model pruning rate corresponding to each training phase based on the preset target pruning rate and the current training iterations includes: In each training phase for iterative pruning, the model pruning rate corresponding to each training phase for iterative pruning is generated based on the preset target pruning rate and the current number of training iterations. The step of training and pruning the neural network model based on the training images and using the pruning configuration to obtain the pruned neural network model output at each training stage includes: Based on the training images and using the pruning configuration, the neural network model is trained and pruned to obtain the pruned neural network model output at each training stage for iterative pruning. In the training phase for fixed-format pruning, the pruned neural network model output by the training phase for iterative pruning is trained and pruned using the pruning configuration corresponding to the training phase for iterative pruning, so as to obtain the pruned neural network model output by the training phase for fixed-format pruning. The pruned neural network model output from the training phase for fixed-format pruning, together with the pruned neural network models output from the training phase for iterative pruning, constitutes a set of pruned neural network models.
7. The method of pruning a neural network according to claim 2, wherein, Before generating the model pruning rate corresponding to each training stage based on the preset target pruning rate and the current training iterations, the method further includes: If the current number of training iterations is divisible by the number of training iterations in a given stage, the model pruning rate for each training stage is generated based on the preset target pruning rate and the current number of training iterations. If the current number of training iterations is not divisible by the number of training iterations in a given stage, the neural network model is pruned using the pruning configuration corresponding to each training stage, resulting in the pruned neural network model output from each training stage.
8. A pruning device for a neural network model, characterized in that, include: The image acquisition and model building module is used to acquire training images and build neural network models. The phase division module is used to divide the training process of the neural network model into multiple training phases; The pruning rate generation module is used to generate the model pruning rate corresponding to each training stage based on the preset target pruning rate and the current number of training iterations. The pruning configuration generation module is used to generate the pruning configuration corresponding to each training stage based on the model pruning rate. The pruning model training module is used to train and prune the neural network model based on the training image and using the pruning configuration to obtain the pruned neural network model output in each training stage, thereby forming a set of pruned neural network models. The neural network model trained and updated in the next training stage is based on the pruned model output in the previous training stage. Each pruned neural network model in this set of pruned neural network models has a different pruning rate and is used to deploy on terminals with different computing power. The preset target pruning rate is determined based on the minimum computing power of each terminal.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the pruning method of the neural network as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the pruning method of the neural network as described in any one of claims 1 to 7.