A neural network pruning subnetwork search method based on relative information entropy of convolutional layers

Abstract

This scheme discloses a neural network pruning subnetwork search method based on the relative information entropy of convolutional layers. This method innovatively utilizes the information entropy of model weights to compare the redundancy of different convolutional layers. Each convolutional layer is treated differently based on its redundancy level. Layers with low redundancy are pruned with a relatively large proportion, while layers with high redundancy are pruned with a relatively small proportion. This results in subnetworks with better network performance, which can be applied to neural network pruning requirements within any proportion range (between 0 and 1). Based on the set pruning retention ratio requirements, a large number of subnetworks that meet the requirements can be searched, and the optimal subnetwork can be selected from them using an effective adaptive batch normalization evaluator.

Description

Technical Field

[0001] This invention belongs to the field of neural network technology, and in particular relates to a neural network pruning subnet search method based on the relative information entropy of convolutional layers. Background Technology

[0002] Neural network pruning is a commonly used neural network optimization technique that reduces the size and improves efficiency by removing some convolutional kernels or weights from the neural network. Neural network pruning methods can generally be divided into two types: structural pruning and weight pruning. Structural pruning involves removing entire neurons or layers to reduce the model's structural size. Weight pruning involves removing certain weights from the network to reduce the number of parameters in the model. Furthermore, pruning methods can also be classified according to the pruning strategy employed.

[0003] Neural network pruning has the following advantages: it can reduce model size, improve inference speed, and reduce storage space; it can improve the model's generalization ability and prevent overfitting; it can improve training efficiency and reduce the risk of overfitting.

[0004] Neural network pruning has been widely applied to various deep learning tasks, such as image classification, object detection, speech recognition, and natural language processing. Specifically, pruning methods can be applied to almost all types of neural networks, including convolutional neural networks, recurrent neural networks, and transformer models.

[0005] The general steps of neural network pruning are as follows:

[0006] 1) Establish a baseline model: First, establish a baseline model, which is usually trained using all available data. This model will also serve as the benchmark for subsequent pruning operations.

[0007] 2) Choosing a pruning method: When choosing a pruning method, you can consider using global pruning or local pruning. Global pruning refers to pruning the entire neural network, while local pruning refers to pruning only certain layers or channels.

[0008] 3) Define pruning criteria: Before pruning, it is necessary to define pruning criteria, which can be weight values, gradients, feature importance, etc.

[0009] 4) Pruning operation: Prune the weights, layers or channels of the neural network according to the pruning criteria.

[0010] 5) Adjusting the pruned model: Pruning reduces the size of the neural network, so the pruned model needs to be adjusted. This process usually includes fine-tuning and retraining.

[0011] 6) Evaluate the model: Evaluate the pruned model using a test set. This process helps determine whether the performance of the pruned model meets expectations.

[0012] In general, the steps of neural network pruning include selecting a pruning method, defining pruning criteria, performing pruning operations, adjusting the model, and evaluating the model. Throughout this process, it is crucial to maintain the model's accuracy and stability while effectively reducing its size and computational complexity.

[0013] The conventional pruning methods mentioned above cannot balance model accuracy and algorithm complexity, and usually require a lot of computational resources and time. To overcome these shortcomings, researchers have been conducting extensive research. Recently, a pruning method based on subnet search and subnet evaluation [Li, Bailin, et al. "Eagleeye: Fast sub-net evaluation for efficient neural network pruning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.] has good practicality and achieves a good balance between algorithm complexity and model accuracy. The authors use a simple random sampling strategy to generate different subnets, and then use an adaptive batch normalization evaluator to evaluate the accuracy of these subnets. The steps are as follows:

[0014] I. Establish a benchmark model;

[0015] 2. Use random numbers to search for subnets; quickly evaluate subnet accuracy and select the optimal subnet;

[0016] 3. Fine-tune the subnets to restore model accuracy.

[0017] This method uses adaptive batch normalization (BN) to evaluate the accuracy of the pruned model. This technique adapts the BN statistics to the pruned network connectivity by adjusting the BN statistics. Specifically, it resets the moving average statistics while freezing all network parameters, and then updates the moving average statistics using several iterations of forward propagation, but without backpropagation. Adaptive BN evaluation can provide more accurate predictions, thus helping to select a better pruning strategy.

[0018] However, the latest pruning scheme still has shortcomings. It uses a completely random number search method to obtain the subnet and assigns the same pruning priority to all convolutional layers. It does not take into account the different redundancy levels of each convolutional layer, and the performance of the subnet obtained by the random number search method has room for improvement.

[0019] Furthermore, Chinese patents have disclosed numerous neural network pruning schemes during long-term research, such as an efficient deep convolutional neural network pruning method [Publication No. CN113610227A]. This scheme primarily addresses the problem of high storage and computational resource consumption in existing deep convolutional neural networks. Its implementation involves optimizing the scaling factor using a sparse learning method based on the ADMM algorithm to train the deep convolutional neural network, thus sparsifying the network structure. A genetic algorithm is then used to search for suitable pruning rates for each layer of the trained deep convolutional neural network, and the optimal pruning rate is automatically searched under the guidance of the fitness function. However, this scheme cannot specify the pruning retention ratio; the compression values ​​for the number of neural network channels, parameters, and floating-point operations are fixed, making it impossible to arbitrarily specify the pruning retention ratio according to requirements. Summary of the Invention

[0020] The purpose of this invention is to address the above-mentioned problems by providing a neural network pruning subnetwork search method based on the relative information entropy of convolutional layers. This method measures the redundancy of each layer by the relative information entropy of the weights of each convolutional layer, and applies a larger pruning ratio to more redundant layers, thereby searching for subnetworks with better performance.

[0021] To achieve the above objectives, the present invention adopts the following technical solutions:

[0022] A neural network pruning subnet search method based on the relative information entropy of convolutional layers, the method comprising:

[0023] S1. Load the original model and the trained model weights, traverse the original model layer by layer, and calculate the information entropy of the weights of each convolutional layer; the information entropy calculated by the original model is uniformly saved to the information entropy dictionary;

[0024] S2. Normalize all information entropy in the information entropy dictionary and save it to the relative information entropy dictionary.

[0025] S3. Calculate the kernel retention ratio of each convolutional layer based on relative information entropy and random sampling;

[0026] S4. Determine whether the current pruning scheme obtained in step S3 meets the pruning requirements. If so, use the current pruning scheme as a new candidate scheme and proceed to S5. Otherwise, repeat steps S3 and S4.

[0027] S5. Evaluate the performance of the pruned subnetwork based on the current candidate scheme. If its performance is higher than the maximum value, update the current performance to the maximum value and update and save the pruned subnetwork, then proceed to S6; otherwise, proceed directly to S6.

[0028] S6. Determine whether the number of iterations in S5 has reached the maximum number of iterations. If so, use the latest saved pruned subnetwork as the pruning result; otherwise, repeat steps S3-S6.

[0029] In the above-mentioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, in step S1, during the process of traversing the original model layer by layer, it is determined whether each layer is a convolutional layer. If there is an ignore list, the information entropy is calculated for the convolutional layers not in the ignore list, and no information entropy is calculated for the convolutional layers in the ignore list.

[0030] In the above-mentioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, this method uses the information entropy method to characterize the redundancy of each convolutional layer, so as to determine the retention ratio of each convolutional kernel by the magnitude of the information entropy.

[0031] In the aforementioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, the calculation of the information entropy of each convolutional layer weight specifically includes:

[0032] S11. If the convolutional layer is not in the ignore list, obtain its weights and flatten them into a one-dimensional array;

[0033] S12. Calculate the frequency distribution of the weights;

[0034] S13. Calculate the information entropy of the corresponding convolutional layer weights based on the frequency distribution;

[0035] S14. Save the information entropy of each convolutional layer to the information entropy dictionary, using the name of the corresponding convolutional layer as the key and the information entropy as the value.

[0036] In the aforementioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, step S2 involves normalizing all information entropies in the information entropy dictionary to obtain a normalized relative information entropy dictionary. The relative information entropy values ​​range from [0,1]. The closer the relative information entropy is to 0, the lower the information content and the higher the redundancy of the layer; the closer the relative information entropy is to 1, the higher the information content and the lower the redundancy of the layer.

[0037] In the above-mentioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, the calculation method for the kernel retention ratio of each convolutional layer in step S3 is as follows:

[0038] ratio=Max(0.2,(0.5×random(0,0.8)+0.5×normalized_entropy_dict[name]))

[0039] The Max(x,y) function takes the maximum value of x and y, random(0,0.8) generates a random number between 0 and 0.8, and normalized_entropy_dict[name] represents the relative entropy of the convolutional layer named name.

[0040] In the above-described neural network pruning subnet search method based on the relative information entropy of convolutional layers, step S4, satisfying the pruning requirement means that the number of floating-point operations of the model after pruning meets the set pruning retention ratio requirement, denoted as R. The method for determining whether the current pruning scheme meets the requirements is as follows:

[0041] The number of floating-point operations in the pruned model and the original model are calculated and denoted as current_flops and origin_flops, respectively. The allowable error for the percentage of the original model retained after pruning is denoted as E, which is usually set to a small number, such as 0.02.

[0042] R–E<=current_flops / origin_flops<=R+E,

[0043] The current pruning scheme meets the requirements.

[0044] In the above-mentioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, before step S1, the following are initialized: the number of floating-point operations of the original model, the ignore list, the maximum retention ratio, the maximum performance value, the pruning mask of each layer, and the model evaluator ABE.

[0045] In the above-mentioned neural network pruning subnet search method based on the relative information entropy of convolutional layers, the pruning process under each pruning scheme specifically includes:

[0046] A. Save the calculated kernel retention ratio of each convolutional layer to the convolutional mask dictionary, and set the mask to 1 for convolutional layers in the ignore list;

[0047] B. Traverse all convolutional layers. For each convolutional layer, obtain its pruning retention ratio P from the convolutional mask dictionary.

[0048] C. During the traversal of B, if P is 1, skip the layer; if P is not 1, for each convolutional kernel in the layer, calculate the average absolute value of the kernel weights. Sort all convolutional kernels from largest to smallest according to the magnitude of the average absolute value; based on the sorting result, retain convolutional kernels with a weight ratio of P from front to back, and remove other convolutional kernels with a weight ratio other than P.

[0049] The advantages of this invention are:

[0050] 1. Innovatively, the information entropy of model weights is used to compare the redundancy of different convolutional layers. Each convolutional layer is treated differently based on its redundancy level. Layers with low redundancy are pruned with a relatively large proportion of redundancy, while layers with high redundancy are pruned with a relatively small proportion of redundancy, so that the sub-network obtained by this scheme has better network performance.

[0051] 2. This invention is applicable to neural network pruning requirements within any ratio range (between 0 and 1). It can search a large number of subnetworks that meet the requirements according to the set pruning retention ratio, and select the optimal subnetwork from them using an effective adaptive batch normalization evaluator. Attached Figure Description

[0052] Figure 1 This is a flowchart of the neural network pruning subnet search method based on the relative information entropy of convolutional layers according to the present invention;

[0053] Figure 2 The diagram shows a comparison of the subnetwork performance obtained by the method of the present invention and existing methods, demonstrating that the subnetwork searched by the method of the present invention has higher performance. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0055] This invention proposes a neural network pruning subnet search method based on the relative information entropy of convolutional layers. Based on the trained model weights, the relative information entropy of the convolutional layers is analyzed to identify which layers have higher redundancy. Higher redundancy indicates that the proportion of convolutional kernels retained in that layer can be smaller. The core idea of ​​this invention is to retain a larger proportion of layers with low redundancy during pruning, and to retain a smaller proportion of layers with high redundancy during pruning. (See attached diagram) Figure 1 As shown, the process of this invention is as follows:

[0056] 1. Train the baseline network to obtain the original model to be pruned;

[0057] 2. Load the pre-trained original model, traverse the original model layer by layer. If it is a convolutional layer, obtain the convolutional kernel weights, calculate the information entropy, and save it to the information entropy array.

[0058] 3. Normalize the information entropy to a relative information entropy in the range of 0 to 1;

[0059] 4. The kernel retention ratio of each convolutional layer is calculated using a combination of relative information entropy and random sampling.

[0060] 5. Calculate the number of floating-point operations of the sub-network under this pruning scheme and compare it with the number of floating-point operations of the original model. If the set pruning retention ratio requirement is met, then this pruning scheme is taken as a candidate scheme.

[0061] 6. Prune branches in each layer according to the pruning retention ratio;

[0062] 7. Use the subnet evaluation method to evaluate the performance of candidate subnets, and always retain the candidate subnet with the highest performance;

[0063] 8. Repeat steps 4 to 7 until the set upper limit for the number of candidate subnetworks to be searched is reached.

[0064] Candidate subnetworks are obtained by pruning the original model based on candidate schemes. Therefore, the number of candidate schemes is the same as the number of candidate subnetworks. In addition, each candidate subnetwork is compared with the maximum performance value once after it is generated, that is, it is iterated once. Therefore, it can also be said that the iteration count of the candidate subnetworks reaches the maximum number of iterations or the number of candidate schemes reaches the set upper limit threshold.

[0065] The following is a specific example:

[0066] The implementation steps of this invention are illustrated using the pruning of the widely used YOLOv5x network as an example. The YOLOv5x network performs excellently in tasks such as object detection, but its large network size and high hardware resource requirements limit its application scenarios. YOLOv5x has a total of 567 layers, containing over 86 million model parameters, and the computational cost of running one network inference is 206.3 Gflops. Therefore, pruning the network to reduce model size and computational cost is of significant value.

[0067] The COCO dataset is one of the most widely used large-scale object detection datasets, containing over 200,000 images labeled with more than 80 different object categories, such as people, animals, vehicles, food, and electronic devices. The images in the COCO dataset are all from real-world scenes, featuring rich imagery and complex backgrounds. The COCO2017 object detection dataset used in this example has 118,287 images in the training set and 5,000 images in the validation set.

[0068] First, train the Yolov5x network using the COCO dataset and save the trained model weight file. Yolov5x is the original model that needs to be pruned in this embodiment.

[0069] First, initialize some variables and objects, including the number of floating-point operations (FLOPS) of the original model, an ignore list containing the indices of the layers to be ignored, the maximum retention ratio, the pruning mask for each layer (i.e., the retention ratio of each convolutional kernel in the model, stored in maskconvdict), and the adaptive BN evaluator (ABE), etc.

[0070] Then, iterate through all layers of the original model. For each convolutional layer, if it is not in the ignore list, obtain the weights of each convolutional layer and calculate its information entropy. Save the information entropy of each convolutional layer into the information entropy dictionary. Normalize the information entropy in the information entropy dictionary (min-max normalization) to obtain the normalized relative information entropy dictionary.

[0071] Specifically, the execution process is as follows:

[0072] For each module in the model, perform the following operations:

[0073] a. Determine if the module is a convolutional layer and if it is in the ignore list;

[0074] b. If it is a convolutional layer and not in the ignore list, then obtain its weights and flatten them into a one-dimensional array;

[0075] c. Use the numpy.histogram() function to calculate the frequency distribution of the weights;

[0076] d. Calculate the information entropy of the corresponding convolutional layer weights based on the frequency distribution;

[0077] e. Save the information entropy of each convolutional layer into the information entropy dictionary entropy_dict, using the name of the corresponding convolutional layer as the key.

[0078] Normalize all the entropy values ​​in entropy_dict to obtain the normalized entropy dictionary, which is the relative entropy dictionary normalized_entropy_dict. The range of relative entropy values ​​is [0,1].

[0079] The first loop iterates through all layers of the model to calculate information entropy and information entropy normalization, and to obtain the relative information entropy of each convolutional layer.

[0080] Next, the model is traversed a second time to perform pruning. For convolutional layers, if they are in the ignore list, the masks of all their common layers are set to 1; otherwise, the mask of the convolutional layer is calculated by combining relative information entropy and random sampling, i.e., the proportion of the convolutional kernel to be retained, and the calculation formula is as follows:

[0081] ratio=Max(0.2,(0.5×random(0,0.8)+0.5×normalized_entropy_dict[name]))

[0082] In the formula, Max(x,y) represents taking the maximum value of x and y, random(0,0.8) represents generating a random number of type float between 0 and 0.8, and normalized_entropy_dict[name] represents the normalized information entropy of the convolutional layer named name, i.e., the relative information entropy.

[0083] Record the mask corresponding to each convolutional layer. For each convolutional layer that needs pruning, save its corresponding pruning mask to the `maskconvdict` dictionary. The key of `maskconvdict` is the name of the convolutional layer, and the value is the pruning mask for that convolutional layer. Prune the model based on the `maskconvdict` dictionary.

[0084] Call the update_yaml() function to update certain parameters in the original YAML file (such as the number of input and output channels of the convolutional layer, the coefficients of the BN layer, etc.) with pruned values, and generate a new YAML file pruned_yaml.

[0085] Create a new model `compact_model` based on `pruned_yaml` and deploy it to the specified device. This model is a pruned model, meaning that some relatively unimportant convolutional kernels have been removed.

[0086] After pruning based on the current pruning scheme, the number of floating-point operations in the pruned model and the original model are calculated and denoted as current_flops and origin_flops, respectively. These are then compared with the number of floating-point operations in the original model. The allowable error for the pruning retention ratio is denoted as E, which is usually set to a small number, such as 0.02.

[0087] RE<=current_flops / origin_flops<=R+E,

[0088] If the current pruning scheme meets the requirements, then the evaluation of this sub-network is abandoned, and other possible sub-networks are tried: by re-sampling randomly and combining it with the relative information entropy to jointly calculate the new mask for each convolutional layer, a new pruning scheme is formed. Otherwise, this sub-network is taken as a candidate sub-network, that is, this pruning scheme is taken as a candidate scheme, and the next step of evaluation is carried out.

[0089] As an object detection model, the performance metric for evaluating the model is mAP (mean Average Precision), which is the average accuracy across all classes (also known as the average precision across all classes). It is obtained by weighted averaging of the average accuracy (AP) of all classes. The ABE evaluator is used to calculate the performance metric mAP of the pruned model. If the current mAP value is greater than the maximum value, the maximum value is updated, the configuration of the pruned model is saved to a file, and the weights of the pruned model are saved to a file. If the current mAP value is not greater than the maximum value, the current maximum value, model configuration, weights, and other parameters are retained (this process is the process of selecting and recording the optimal candidate subnet). Then the iteration count is updated. If the number of iterations reaches the specified maximum number of iterations, the loop exits; otherwise, new pruning candidate schemes are obtained, and the above steps are repeated. Finally, the optimal subnet can be obtained, thus obtaining a new model that inherits the weights of the original model and the optimal subnet structure.

[0090] After pruning, fine-tuning the pruned model on the COCO dataset yields a new model with accuracy close to the original model, but significantly reduced size and faster execution speed. It should be noted that the paper [Li, Bailin, et al. "Eagleeye: Fast sub-net evaluation for efficient neural network pruning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.] has demonstrated that for a large number of sub-networks obtained through the search, the evaluation method provided in the paper (i.e., the ABE evaluator used in this invention) shows a positive correlation between the performance before and after fine-tuning (correlation coefficient approximately 0.8). Therefore, improving the sub-network search algorithm and enhancing sub-network performance can improve the performance of the final fine-tuned model.

[0091] like Figure 2 As shown in the figure, the horizontal axis represents the index of the subnetwork obtained by the search (from 1 to 700, representing 700 subnetworks that meet the pruning retention ratio requirements), and the vertical axis represents the evaluation result of the subnetwork using the ABE evaluator. The two horizontal dashed lines indicate the maximum values ​​of the search results of the existing methods and the method of the present invention, respectively. As can be seen from the figure, the performance index of the subnetwork obtained by the present invention is significantly better than that of the existing methods. The neural network pruning subnetwork search method proposed in this scheme can obtain higher quality pruned subnetworks more quickly and efficiently.

[0092] To further illustrate the advantages of the method of the present invention, the performance of the pruned models of the present invention and existing methods, after fine-tuning and training, is compared as follows:

[0093] mAP_0.5 mAP_0.5:0.95 Existing methods 0.66234 0.47023 Method of the present invention 0.66701 0.47614

[0094] Here, mAP_0.5 and mAP_0.5:0.95 are two indicators of mAP, both of which are better the higher the value, with mAP_0.5 usually being the primary indicator. Unless otherwise specified in this specification, mAP refers to mAP_0.5. As can be seen from the table, the method of this invention is superior to existing methods in both indicators.

[0095] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for neural network pruning subnetwork search based on relative information entropy of convolutional layers, characterized in that, The method includes: S1. Load the original model and the trained model weights, traverse the original model layer by layer, calculate the information entropy of the weights of each convolutional layer and save it to the information entropy dictionary; S2. Normalize all information entropies in the information entropy dictionary and save them to the relative information entropy dictionary; S3. Determine the kernel retention ratio of each convolutional layer based on relative information entropy and random sampling; S4. Determine whether the current pruning scheme obtained in step S3 meets the pruning requirements. If so, use the current pruning scheme as a new candidate scheme and then proceed to S5. Otherwise, repeat steps S3 and S4. Meeting the pruning requirements means that the number of floating-point operations of the model after pruning meets the requirements of the set pruning retention ratio, which is denoted as R. In step S3, the kernel retention ratio of each convolutional layer is calculated as follows: ratio=Max(0.2,(0.5×random(0,0.8)+0.5×normalized_entropy_dict[name])) The Max(x,y) function represents the maximum value of x and y, random(0,0.8) represents the random number generated between 0 and 0.8, and normalized_entropy_dict[name] represents the relative information entropy of the convolutional layer named name; In step S4, it is determined whether the current pruning plan meets the requirements in the following way: Calculate the number of floating-point operations of the pruned model and the number of floating-point operations of the original model, denoted as current_flops and origin_flops respectively. The allowable error of the pruned retention ratio of the original model is denoted as E. If R–E<=current_flops / origin_flops<=R+E; The current pruning plan meets the requirements; S5. Evaluate the performance of the pruned subnetwork based on the current candidate scheme. If its performance is higher than the maximum value, update the current performance to the maximum value and update and save the pruned subnetwork, then proceed to S6; otherwise, proceed directly to S6. S6. Determine whether the number of iterations in S5 has reached the maximum number of iterations. If so, use the latest saved pruned subnetwork as the pruning result; otherwise, repeat steps S3-S6.

2. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 1, characterized in that, In step S1, during the process of traversing the original model layer by layer, it is determined whether each layer is a convolutional layer. If there is an ignore list, the information entropy is calculated for the convolutional layers that are not in the ignore list, and no information entropy is calculated for the convolutional layers that are in the ignore list.

3. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 2, characterized in that, This method uses the information entropy method to characterize the redundancy of each convolutional layer, so as to determine the retention ratio of the corresponding convolutional kernel by the magnitude of the information entropy.

4. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 3, characterized in that, The calculation of the information entropy of the weights of each convolutional layer specifically includes: S11. If the convolutional layer is not in the ignore list, obtain its weights and flatten them into a one-dimensional array; S12. Calculate the frequency distribution of the weights; S13. Calculate the information entropy of the corresponding convolutional layer weights based on the frequency distribution; S14. Save the information entropy of each convolutional layer to the information entropy dictionary, using the name of the corresponding convolutional layer as the key and the information entropy as the value.

5. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 4, characterized in that, In step S2, all information entropies in the information entropy dictionary are normalized to obtain a normalized relative information entropy dictionary, with the relative information entropy values ​​ranging from [0,1].

6. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 1, characterized in that, Before step S1, initialize the following: the number of floating-point operations of the original model, the ignore list, the maximum retention ratio, the maximum performance, the pruning mask for each layer, and the model evaluator ABE.

7. The neural network pruning subnet search method based on relative information entropy of convolutional layers according to claim 6, characterized in that, The pruning process under each pruning scheme specifically includes: A. Save the calculated kernel retention ratio of each convolutional layer to the convolutional mask dictionary, and set the mask to 1 for convolutional layers in the ignore list; B. Traverse all convolutional layers. For each convolutional layer, obtain its pruning retention ratio P from the convolutional mask dictionary. C. During the traversal of B, if P is 1, skip the layer; if P is not 1, for each convolutional kernel in the layer, calculate the average absolute value of the weights of the convolutional kernel, and sort all convolutional kernels from largest to smallest according to the size of the average absolute value; according to the sorting result, retain the convolutional kernels with a ratio of P from front to back, and remove other convolutional kernels with a ratio other than P.

Images