Pruning method and device of neural network, electronic equipment and readable storage medium
By evaluating the importance of filters in neural networks and pruning based on the target pruning ratio, the problem of large differences in networks after pruning is solved, the reduction of parameter quantity and calculation complexity and the retention of reasoning ability is achieved, and the The efficiency and generalization capabilities of the model.
Patent Information
- Application Number
- CN202411020971.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
AI Technical Summary
The existing structural neural network pruning method leads to a large difference between the network and the original network after pruning, resulting in overall information loss.
By obtaining the data to be reasoned, the feature map information of each layer of sub-inference network is calculated, the importance of the filter is evaluated, and the target pruning ratio is determined based on the importance and preset pruning rate set, and pruning is performed to retain the overall structure of the network. Rationality and efficiency.
The parameter quantity and calculation complexity are reduced, while the reasoning ability of the original neural network is retained to the greatest extent, and the model's reasoning speed and generalization ability are improved.
Smart Images

Figure CN119026662A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a neural network pruning method, device, electronic device and readable storage medium. Background Art
[0002] As an efficient network optimization technology, neural network pruning aims to reduce the parameter size and computational complexity in the neural network, thereby optimizing the model's operating efficiency and reducing memory usage. According to the different characteristics of the pruned units, neural network pruning can be divided into two categories: structural pruning and non-structural pruning. In structural pruning, the core issue is how to accurately evaluate the importance of the filter. However, since the noise that may exist in the input data and the useless information that may be generated by multi-layer convolution operations will affect the importance assessment, determining the compression ratio of each layer according to the model compression requirements is an essential step in the pruning process. Although the existing structural pruning method searches for the best network structure from a global perspective, its candidate structure is not considered from a global perspective during the search process, resulting in a large difference between the pruned network and the original network, which in turn causes the loss of overall network information. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a neural network pruning method, device, electronic device and readable storage medium to solve the problem in the prior art that the difference between the pruned network and the original network is large, resulting in loss of network integrity information.
[0004] A first aspect of an embodiment of the present application provides a neural network pruning method, the method comprising:
[0005] Obtain the data to be inferred, and input the data to be inferred into a neural network, wherein the neural network includes multiple layers of sub-inference networks, and each sub-inference network is used to infer the data to be inferred, so as to obtain the feature map information corresponding to each sub-inference network; determine the important weight of each filter in each sub-inference network according to the feature map information corresponding to each sub-inference network; determine the pruning rate set corresponding to each sub-inference network, and determine the target pruning ratio of each sub-inference network based on the important weight of each filter in each sub-inference network and the pruning rate set corresponding to each sub-inference network; prune each sub-inference network based on the target pruning ratio and the important weight of each filter in each sub-inference network.
[0006] A second aspect of an embodiment of the present application provides a neural network pruning device, the method comprising:
[0007] The acquisition module is configured to acquire the data to be inferred and input the data to be inferred into the neural network, wherein the neural network includes multiple layers of sub-inference networks, each sub-inference network is used to infer the data to be inferred, and obtain the feature map information corresponding to each sub-inference network; the first determination module is configured to determine the important weights of each filter in each sub-inference network according to the feature map information corresponding to each sub-inference network; the second determination module is configured to determine the pruning rate set corresponding to each sub-inference network, and determine the target pruning ratio of each sub-inference network based on the important weights of each filter in each sub-inference network and the pruning rate set corresponding to each sub-inference network; the pruning module is configured to prune each sub-inference network based on the target pruning ratio and the important weights of each filter in each sub-inference network.
[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0011] Obtain the data to be inferred, and input the data to be inferred into the neural network. The neural network contains multiple layers of sub-inference networks, and each sub-inference network is used to infer the data to be inferred, and obtain the feature graph information corresponding to each sub-inference network; determine the important weights of each filter in each sub-inference network according to the feature graph information corresponding to each sub-inference network; determine the pruning rate set corresponding to each sub-inference network, and determine the target pruning ratio of each sub-inference network based on the important weights of each filter in each sub-inference network and the pruning rate set corresponding to each sub-inference network; prune each sub-inference network based on the target pruning ratio and the important weights of each filter in each sub-inference network. The present application determines the importance of the filter in each sub-inference network through the feature graph information corresponding to each sub-inference network, and uses this as the basis for pruning each sub-inference network. At the same time, the pruning rate set of each sub-inference network is used to guide the target pruning ratio of each layer of sub-inference network, so that the entire neural network can focus on the rationality and efficiency of its overall structure during the pruning process, and retain the overall information of the network. The pruned neural network reduces the number of parameters and computational complexity while retaining the reasoning ability of the original neural network to the greatest extent, thereby improving the reasoning speed and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 It is a flowchart of a neural network pruning method provided in an embodiment of the present application;
[0014] Figure 2 It is a structural schematic diagram of a neural network pruning device provided in an embodiment of the present application;
[0015] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0017] A neural network pruning method, device, electronic device, and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0018] Figure 1 Schematic diagram of a neural network pruning method provided in an embodiment of the present application. Figure 1 As shown, the method includes:
[0019] S101, obtaining the data to be inferred, and inputting the data to be inferred into a neural network, wherein the neural network includes multiple layers of sub-inference networks, each of which is used to infer the data to be inferred, and obtaining feature map information corresponding to each sub-inference network.
[0020] Specifically, the data to be inferred can be obtained from a certain source (such as a database, a file, a user input, etc.) and can be images, text, audio, or other types of data, which are not specifically limited here.
[0021] The data to be inferred is used as input to a trained neural network. In this neural network, there are multiple sub-inference networks, and different sub-inference networks are responsible for processing different parts of the data to be inferred or different levels of features. Each sub-inference network may contain multiple convolutional layers and other types of layers (such as pooling layers, fully connected layers, loop layers, attention mechanism layers, etc.). In each convolutional layer (containing multiple filters), the filter slides on the input data (or the feature map output by the previous layer) and performs a convolution operation to generate new feature map information. These feature map information describes different aspects or attributes of the data to be inferred.
[0022] S102, determining the important weight of each filter in each sub-inference network according to the feature graph information corresponding to each sub-inference network.
[0023] Specifically, the importance weight of a filter determines its role and importance in the convolution operation. The importance weight of each filter can be determined by evaluating the contribution of the feature map information produced by the filter to the final task (such as classification, regression, etc.).
[0024] S103, determining a pruning rate set corresponding to each sub-inference network, and determining a target pruning ratio of each sub-inference network based on an important weight of each filter in each sub-inference network and a pruning rate set corresponding to each sub-inference network;
[0025] Specifically, the pruning rate set can be a predefined set of values that indicate the proportion of filters that need to be pruned (i.e., removed) in each sub-inference network. The selection of the pruning rate can be based on the need for model compression, the requirement for inference speed, and the trade-off of maintaining model accuracy. When determining the pruning rate set, various factors can be considered, such as the complexity of the model, the characteristics of the data, and hardware resources, etc. The specific pruning rate set values are not specifically limited here.
[0026] Based on the important weights of the filters in each sub-inference network and the corresponding pruning rate set, the influence of the filters in each layer of the sub-inference network on the neural network after pruning at different pruning ratios can be obtained, and the target pruning ratio of each sub-inference network can be determined based on this.
[0027] S104, pruning each sub-inference network based on the target pruning ratio and the important weight of each filter in each sub-inference network;
[0028] According to the technical solution provided in the embodiment of the present application, the data to be inferred is obtained and input into the neural network, the neural network includes multiple layers of sub-inference networks, each sub-inference network is used to infer the data to be inferred, and the feature graph information corresponding to each sub-inference network is obtained; according to the feature graph information corresponding to each sub-inference network, the important weight of each filter in each sub-inference network is determined; the pruning rate set corresponding to each sub-inference network is determined, and the target pruning ratio of each sub-inference network is determined based on the important weight of the filter in each sub-inference network and the pruning rate set corresponding to each sub-inference network; each sub-inference network is pruned based on the target pruning ratio and the important weight of each filter in each sub-inference network. The importance of the filter in each sub-inference network is determined by the feature graph information, which is used as the basis for pruning each sub-inference network. At the same time, the target pruning ratio of each layer of the sub-inference network is guided by the pruning rate set of each sub-inference network, so that the entire neural network can focus on the rationality and efficiency of its overall structure during the pruning process, and retain the overall information of the network. As a result, the pruned neural network reduces the number of parameters and computational complexity while retaining the reasoning ability of the original neural network to the greatest extent, thereby improving the reasoning speed and generalization ability of the model.
[0029] In some embodiments, each filter in the sub-inference network corresponds to a feature graph information; based on the feature graph information corresponding to each sub-inference network, the important weight of each filter in each sub-inference network is determined, including: calculating the cosine similarity between the feature graph information corresponding to different filters in each sub-inference network, and based on the cosine similarity between the feature graph information corresponding to different filters, determining the mean similarity corresponding to each filter; sorting the similarity means to obtain an importance list of the filters in each sub-inference network, and obtaining the important weight of each filter based on the importance list.
[0030] Specifically, when determining the important weight of each filter in the sub-reasoning network, feature graph information is introduced as the basis for reasoning. Feature graph information represents the actual effect of the filter in the model. By calculating the cosine similarity between feature graph information corresponding to different filters, the similarity and redundancy between filters can be evaluated. Cosine similarity, as a measure of the cosine value of the angle between two vectors, can reflect the degree of proximity of the two vectors in direction. In this embodiment, feature graph information is regarded as a vector, and the cosine similarity between filters can be obtained by calculating the cosine similarity between feature graph information corresponding to different filters. Based on the cosine similarity, the mean similarity corresponding to each filter is further calculated. The mean similarity represents the average similarity between each filter and other filters, which can reflect the redundancy of the filter in the model. The higher the mean similarity, the higher the similarity between the filter and other filters, the greater the redundancy, that is, the lower the importance of the corresponding filter; conversely, the lower the mean similarity, the higher the uniqueness of the filter in the model and the greater the importance.
[0031] Furthermore, the similarity means are sorted to obtain the importance list of filters in each sub-inference network. Through sorting, we can clearly see the importance ranking of each filter in the model, thus providing guidance for subsequent pruning operations.
[0032] According to the technical solution provided in the embodiment of the present application, the cosine similarity between the feature graph information in each sub-inference network is calculated respectively, and the mean cosine similarity between each feature graph information and the other feature graph information in each sub-inference network is obtained, and the mean cosine similarity is sorted to obtain the importance list of the filters in each sub-inference network; based on the importance list, the important weight of each filter in each sub-inference network is determined, which not only considers the role of the filter in the convolution operation, but also considers its uniqueness and importance in the entire sub-inference network. Therefore, this method can more accurately evaluate the importance of the filter and provide more valuable guidance for subsequent pruning operations.
[0033] In addition, in some embodiments, determining a pruning rate set corresponding to each sub-inference network includes: obtaining a pruning rate extreme value corresponding to each sub-inference network, the pruning rate extreme value including a first preset pruning rate and a second preset pruning rate, the second preset pruning rate being greater than the first preset pruning rate; determining a pruning rate interval for each sub-inference network based on the first preset pruning rate and the second preset pruning rate; and dividing the pruning rate interval into equal parts according to preset equal division intervals to obtain a pruning rate set for each sub-inference network.
[0034] Specifically, in the process of neural network model optimization, the choice of pruning rate can determine the degree of model pruning, thereby directly affecting the complexity and performance of the model. Therefore, when setting the pruning rate, it is necessary to comprehensively consider the balance between the accuracy and computational efficiency of the model. In an embodiment of the present application, a method for determining a pruning rate set based on a pruning rate extreme value is proposed.
[0035] Get the pruning rate extreme value corresponding to each sub-inference network, which is set based on the actual application requirements and the goal of model optimization. The pruning rate extreme value includes a first preset pruning rate and a second preset pruning rate. The first preset pruning rate can be a relatively low pruning rate to ensure that the pruned model still maintains a high accuracy; while the second preset pruning rate is a relatively high pruning rate, which aims to further reduce the complexity of the model and improve computational efficiency. By setting these two pruning rate extreme values, a reasonable pruning rate range can be provided for subsequent pruning operations.
[0036] Furthermore, based on the first preset pruning rate and the second preset pruning rate of each sub-inference network, the pruning rate interval of each sub-inference network is determined. This interval contains all possible pruning rate values and can be adjusted as needed. By setting a suitable pruning rate interval, it can be ensured that different pruning degrees can be explored during the pruning process, so as to find the optimal pruning rate. The pruning rate interval is divided into equal parts according to the preset equal intervals to obtain a pruning rate set for each sub-inference network. This set contains multiple specific pruning rate values for subsequent pruning operations. By setting the equal intervals, the granularity of the pruning rate set can be controlled, so that more detailed adjustments can be made between different pruning degrees.
[0037] It should be noted that the preset equal division interval can be set to different equal division intervals according to the complexity and accuracy requirements of the model. For models with high complexity and strict accuracy requirements, a smaller equal division interval can be used to more finely adjust the pruning rate and find the optimal pruning balance point; while for models with low complexity and relatively loose accuracy requirements, a larger equal division interval can be used to improve the efficiency of the pruning operation.
[0038] According to the technical solution provided in the embodiments of the present application, the pruning rate extreme value corresponding to each sub-inference network is obtained, and the pruning rate extreme value includes a first preset pruning rate and a second preset pruning rate, and the second preset pruning rate is greater than the first preset pruning rate; based on the first preset pruning rate and the second preset pruning rate, the pruning rate interval of each sub-inference network is determined; the pruning rate interval is equally divided according to the preset equal division interval to obtain the pruning rate set of each sub-inference network, and the pruning rate set of each layer of sub-inference network can be obtained by setting appropriate pruning rate extreme values and equal division intervals for each layer of sub-inference network, so that in the actual pruning process, the appropriate pruning rate can be selected for pruning operation according to the specific situation and needs of the model.
[0039] In addition, in some embodiments, based on the important weight of each filter in each sub-inference network and the pruning rate set corresponding to each sub-inference network, the target pruning ratio of each sub-inference network is determined, including: using each pruning rate in the pruning rate set and the important weight of each filter in each sub-inference network to perform initial pruning on the filters in each sub-inference network in turn; performing precision loss calculation on each sub-inference network after the initial pruning to obtain the precision loss of each sub-inference network at each pruning rate, and based on the precision loss, obtaining a precision loss set of each sub-inference network at different pruning rates; performing a global importance test based on the precision loss set, and determining the target pruning ratio of the filters in each sub-inference network based on the test results.
[0040] Specifically, each pruning rate in the pruning rate set corresponding to each sub-inference network and the important weights of the filters in each sub-inference network are used to perform initial pruning on the filters with smaller important weights in the sub-inference network of this layer. In this process, the importance list is sorted from small to large according to the important weights of the filters, and the filters with smaller important weights are pruned in turn until the number of prunings corresponding to the current pruning rate is reached. Such initial pruning operations are intended to simulate the actual pruning effect and preliminarily evaluate the impact of pruning on model accuracy. The accuracy loss is then calculated for each sub-inference network after the initial pruning, and a set of accuracy losses for each sub-inference network at different pruning rates is obtained. The calculation of accuracy loss can be achieved by comparing the performance differences of the model before and after pruning on the validation set or test set. This set contains the changes in model accuracy at different pruning rates,
[0041] Furthermore, a global importance test is performed based on the accuracy loss set. This test aims to evaluate the global changes in model accuracy under different pruning rates and find a suitable pruning balance point. Specifically, a search algorithm (such as grid search, random search, or Bayesian optimization) can be used to search for the pruning rate that minimizes the model accuracy loss in the pruning rate set. This pruning rate is the target pruning ratio of the current sub-inference network.
[0042] It should be noted that in the process of global importance testing, the mutual influence between different sub-inference networks needs to be considered. Since neural network models are usually composed of multiple sub-inference networks, the pruning of one sub-inference network may affect the performance of other sub-inference networks. Therefore, when determining the target pruning ratio of each sub-inference network, it is necessary to comprehensively consider the performance and complexity requirements of the entire model and perform global optimization;
[0043] As an example, in order to achieve global optimization, an iterative method can be used. Based on the initial pruning rate set, prune and evaluate the accuracy of each sub-inference network to obtain a preliminary pruning ratio. Then, according to the performance of the model after preliminary pruning, adjust the pruning rate set or search strategy, and prune and evaluate again. This process can be repeated many times until a set of pruning ratio combinations of each layer of sub-inference networks that optimizes the performance of the entire model is found. For example, the pruning rate of the first layer of sub-inference network is 20%, the pruning rate of the second layer of sub-inference network is 15%, and the pruning rate of the third layer of sub-inference network is 10%. At this time, the pruning of the neural network is the best, that is, the 20%, 15%, and 10% obtained in this way are the optimal pruning ratio combinations of each layer of sub-inference networks.
[0044] According to the technical solution provided in the embodiment of the present application, by combining the important weights of the filter and the pruning rate set to determine the target pruning ratio of each sub-reasoning network, the impact of pruning on model performance can be more accurately evaluated, and the optimal target pruning ratio can be found. Not only the role and importance of the filter in the convolution operation are considered, but also the complexity and performance requirements of the entire model, so that the computational complexity can be reduced while maintaining the accuracy of the model, and the operating efficiency of the model can be improved.
[0045] In some embodiments, the precision loss of each sub-inference network after initial pruning is calculated to obtain the precision loss of each sub-inference network at each pruning rate, and based on the precision loss, a set of precision losses of each sub-inference network at different pruning rates is obtained, including: using a test data set to test the sub-inference network after initial pruning to obtain the test accuracy; calculating the difference between the original test accuracy of each sub-inference network and the test accuracy of each sub-inference network after pruning to obtain the precision loss of each sub-inference network at the corresponding pruning rate; traversing all pruning rates and each sub-inference network in the pruning rate set to obtain a set of precision losses of each sub-inference network at different pruning rates.
[0046] Specifically, in order to accurately calculate the accuracy loss after pruning, you first need to prepare an independent test dataset. This test dataset is representative and extensive, and can fully reflect the generalization ability and performance of the model. Then, use this test dataset to test the neural network before pruning, and record its test accuracy as the initial test accuracy. This initial test accuracy represents the performance level of the model before pruning.
[0047] Furthermore, for each sub-inference network, pruning operations are performed according to the determined pruning rate set. After pruning is completed, the pruned sub-inference network is tested again using the same test data set, and its test accuracy is recorded. Then, the difference between the initial test accuracy of the sub-inference network and the test accuracy after pruning is calculated, and the difference indicates the accuracy loss of the sub-inference network at the corresponding pruning rate. This process needs to traverse all pruning rates in the pruning rate set, and perform the above operations on each sub-inference network. Through such operations, a set of accuracy losses for each sub-inference network at different pruning rates can be obtained. This set reflects the specific impact of pruning operations at different pruning rates on model performance.
[0048] According to the technical solution provided in this embodiment, the neural network after initial pruning is tested using a test data set to obtain the test accuracy; the difference between the initial test accuracy of each sub-inference network and the test accuracy of each sub-inference network after pruning is calculated to obtain the accuracy loss of the sub-inference network at the corresponding pruning rate; all pruning rates and each sub-inference network in the pruning rate set are traversed to obtain the accuracy loss set of each sub-inference network at different pruning rates, and the accuracy loss set of each sub-inference network is determined by combining the important weights of each sub-inference network filter and the pruning rate set, and then the target pruning ratio is determined, so that the pruning process can be controlled more finely, the pruning effect can be optimized, the performance and generalization ability of the model can be improved, and the consumption of computing resources can be reduced.
[0049] In addition, in some embodiments, each sub-inference network is pruned based on a target pruning ratio and the importance weight of each filter in each sub-inference network, including: determining the number of filters that need to be pruned in each sub-inference network based on the target pruning ratio; selecting target filters that need to be pruned in order of importance from low to high based on the importance weights of the filters in each sub-inference network; and performing pruning operations on the target filters to complete the pruning of each sub-inference network.
[0050] Specifically, according to the target pruning ratio, the number of filters that need to be pruned in each sub-inference network is calculated. This number is a value proportional to the pruning ratio, which can determine the size and complexity of each sub-inference network after pruning. Then, according to the importance weights of the filters in each sub-inference network, the target filters that need to be pruned are selected in order from low to high importance. This selection process is based on an assumption that filters with lower importance weights contribute less to the model, so pruning them has less impact on model performance. In this way, the impact of pruning on model performance can be minimized while achieving lightweight and accelerated models.
[0051] Furthermore, after the target filter is selected, it is removed from each sub-inference network to complete the pruning operation of the sub-inference network. This process can be achieved by directly modifying the parameters of the network structure or by skipping the calculation of these filters during the inference process. Regardless of which method is used, it is necessary to ensure that the pruned network can correctly perform the inference task and maintain performance similar to that before pruning.
[0052] It should be noted that during the pruning process, it is necessary to ensure that the pruning operation is reversible. In other words, if the performance of the model after pruning decreases too much, it is necessary to be able to restore it to the state before pruning and try to use different pruning strategies or parameters for pruning.
[0053] According to the technical solution provided in the embodiment of the present application, based on the target pruning ratio, the number of filters that need to be pruned in each sub-inference network is determined; based on the important weights of the filters in each sub-inference network, the target filters that need to be pruned are selected in order of importance from low to high; the target filters are removed from each sub-inference network, thereby achieving lightweight model, improving model operation efficiency, and reducing computing resource consumption.
[0054] In addition, after pruning each sub-inference network based on the target pruning ratio and the important weight of each filter in each sub-inference network, the method includes: selecting the neural network before pruning as the teacher network and the pruned neural network as the student network; using the same target data for inference on the teacher network and the student network, and obtaining the outputs of the teacher network and the student network respectively; calculating the relative entropy loss and the cross entropy loss between the outputs of the teacher network and the student network, and training the student network by combining the relative entropy loss and the cross entropy loss to measure the difference in the output distributions of the teacher network and the student network, so that the output distribution of the student network is close to that of the teacher network and conforms to the true label at the same time.
[0055] Specifically, after pruning is completed, in order to further improve the performance of the pruned model, the knowledge distillation technology can be used. This technology uses a well-trained model with good performance (we call it the teacher network) to guide the training of the pruned model (we call it the student network). In this way, the student network can learn the knowledge and experience of the teacher network and optimize its own performance.
[0056] It can be understood that the neural network before pruning can be selected as the teacher network, representing the best performance of the model before pruning. Then, the pruned neural network is used as the student network for subsequent training and optimization.
[0057] During training, the same target data is used to perform inference on the teacher network and the student network, and their outputs are obtained respectively. Then, we calculate the relative entropy loss (also known as KL divergence loss) and cross entropy loss between the outputs of the teacher network and the student network. The relative entropy loss is used to measure the difference between the two output distributions, while the cross entropy loss is used to measure the difference between the output of the student network and the true label.
[0058] By combining relative entropy loss and cross entropy loss, the student network can be trained. During the training process, the parameters of the student network are constantly adjusted to make its output distribution closer to the output distribution of the teacher network and at the same time conform to the true label. In this way, the student network can learn the knowledge and experience of the teacher network and improve its own performance.
[0059] In addition, the training process can be further optimized according to specific task requirements and data characteristics. For example, different loss functions, learning rate adjustment strategies, optimization algorithms, etc. can be used to better adapt to different tasks and data. Through continuous trial and optimization, the training strategy that best suits the current task and data can be found, thereby further improving the performance of the pruned model.
[0060] According to the technical solution provided in the embodiment of the present application, by combining knowledge distillation technology and pruning technology, the model can be lightweight and performance optimized. Pruning technology can help reduce the complexity of the model and the consumption of computing resources, while knowledge distillation technology can improve the performance of the pruned model. This combination can achieve lightweight and acceleration of the model while maintaining model performance.
[0061] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.
[0062] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.
[0063] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0064] Figure 2 Schematic diagram of the structure of a neural network pruning device provided in an embodiment of the present application. Figure 2 As shown, the device comprises:
[0065] An acquisition module 201 is configured to acquire data to be inferred, and input the data to be inferred into a neural network, wherein the neural network includes multiple layers of sub-inference networks, each of which is used to infer the data to be inferred, and obtain feature graph information corresponding to each sub-inference network;
[0066] A first determination module 202 is configured to determine the important weight of each filter in each sub-inference network according to the feature map information corresponding to each sub-inference network;
[0067] The second determination module 203 is configured to determine a pruning rate set corresponding to each sub-inference network, and determine a target pruning ratio of each sub-inference network based on an important weight of each filter in each sub-inference network and a pruning rate set corresponding to each sub-inference network;
[0068] The pruning module 204 is configured to prune each sub-inference network based on a target pruning ratio and an important weight of each filter in each sub-inference network.
[0069] In some embodiments, the first determination module 202 is also configured to respectively calculate the cosine similarities between the feature graph information corresponding to different filters in each sub-inference network, and determine the mean similarity corresponding to each filter based on the cosine similarities between the feature graph information corresponding to different filters; sort the similarity means to obtain an importance list of the filters in each sub-inference network, and obtain the importance weight of each filter based on the importance list.
[0070] In some embodiments, the second determination module 203 is further configured to obtain a pruning rate extreme value corresponding to each sub-inference network, the pruning rate extreme value including a first preset pruning rate and a second preset pruning rate, the second preset pruning rate being greater than the first preset pruning rate; based on the first preset pruning rate and the second preset pruning rate, determine the pruning rate interval of each sub-inference network; divide the pruning rate interval into equal parts according to a preset equal interval to obtain a pruning rate set for each sub-inference network.
[0071] In some embodiments, the second determination module 203 is further configured to use each pruning rate in the pruning rate set and the important weight of each filter in each sub-inference network to perform initial pruning on the filters in each sub-inference network in turn; perform precision loss calculation on each sub-inference network after the initial pruning to obtain the precision loss of each sub-inference network at each pruning rate, and based on the precision loss, obtain a precision loss set for each sub-inference network at different pruning rates; perform a global importance test based on the precision loss set, and determine the target pruning ratio of the filter in each sub-inference network based on the test results.
[0072] In some embodiments, the second determination module 203 is further configured to use a test data set to test the sub-inference network after initial pruning to obtain a test accuracy; calculate the difference between the original test accuracy of each sub-inference network and the test accuracy of each sub-inference network after pruning to obtain the accuracy loss of each sub-inference network at the corresponding pruning rate; traverse all pruning rates and each sub-inference network in the pruning rate set to obtain a set of accuracy losses of each sub-inference network at different pruning rates.
[0073] In some embodiments, the pruning module 204 is further configured to determine the number of filters that need to be pruned in each sub-inference network based on the target pruning ratio; select the target filters that need to be pruned in order of importance from low to high based on the importance weights of the filters in each sub-inference network; and perform pruning operations on the target filters to complete the pruning of each sub-inference network.
[0074] In some embodiments, the pruning module 204 is further configured to select the neural network before pruning as the teacher network and the neural network after pruning as the student network; use the same target data for inference on the teacher network and the student network, and obtain the outputs of the teacher network and the student network respectively; calculate the relative entropy loss and cross entropy loss between the outputs of the teacher network and the student network, and train the student network by combining the relative entropy loss and the cross entropy loss to measure the difference in output distribution between the teacher network and the student network, so that the output distribution of the student network is close to that of the teacher network and at the same time conforms to the true label.
[0075] Figure 3 Schematic diagram of an electronic device 3 provided in an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0076] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 The electronic device 3 is merely an example and does not limit the electronic device 3 , and may include more or less components than those shown in the figure, or different components.
[0077] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0078] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. The memory 302 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.
[0079] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0080] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0081] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A neural network pruning method, characterized in that: include: Acquire data to be inferred, and input the data to be inferred into the neural network, wherein the neural network includes multiple layers of sub-inference networks, each of which is used to infer the data to be inferred, and obtain feature graph information corresponding to each of the sub-inference networks; Determine the important weight of each filter in each of the sub-inference networks according to the feature map information corresponding to each of the sub-inference networks; Determine a pruning rate set corresponding to each of the sub-inference networks, and determine a target pruning ratio of each of the sub-inference networks based on the important weight of each filter in each of the sub-inference networks and the pruning rate set corresponding to each of the sub-inference networks; Each of the sub-inference networks is pruned based on the target pruning ratio and the important weight of each filter in each of the sub-inference networks.
2. The method according to claim 1, characterized in that: Each filter in the sub-inference network corresponds to a feature graph information; and determining the important weight of each filter in each sub-inference network according to the feature graph information corresponding to each sub-inference network includes: Calculating the cosine similarity between the feature graph information corresponding to different filters in each of the sub-inference networks, and determining the mean similarity corresponding to each filter based on the cosine similarity between the feature graph information corresponding to different filters; The similarity means are sorted to obtain an importance list of filters in each of the sub-inference networks, and an important weight of each filter is obtained based on the importance list.
3. The method according to claim 1, characterized in that The determining of a set of pruning rates corresponding to each of the sub-inference networks comprises: Obtaining a pruning rate extreme value corresponding to each of the sub-inference networks, wherein the pruning rate extreme value includes a first preset pruning rate and a second preset pruning rate, and the second preset pruning rate is greater than the first preset pruning rate; Determining a pruning rate interval of each of the sub-inference networks based on the first preset pruning rate and the second preset pruning rate; The pruning rate interval is equally divided according to a preset equal division interval to obtain a pruning rate set for each of the sub-inference networks.
4. The method according to claim 3, characterized in that The determining of a target pruning ratio of each sub-inference network based on the important weight of each filter in each sub-inference network and a set of pruning rates corresponding to each sub-inference network includes: Using each pruning rate in the pruning rate set and the important weight of each filter in each sub-inference network, initially pruning the filter in each sub-inference network in turn; Calculating the precision loss of each of the sub-inference networks after initial pruning to obtain the precision loss of each sub-inference network at each pruning rate, and obtaining a precision loss set of each of the sub-inference networks at different pruning rates based on the precision loss; A global importance test is performed based on the precision loss set, and the target pruning ratio of the filter in each sub-inference network is determined based on the test result.
5. The method according to claim 4, characterized in that: The step of calculating the precision loss of each sub-inference network after the initial pruning to obtain the precision loss of each sub-inference network at each pruning rate, and obtaining a precision loss set of each sub-inference network at different pruning rates based on the precision loss includes: Using a test data set to test the sub-inference network after initial pruning to obtain a test accuracy; Calculate the difference between the original test accuracy of each of the sub-inference networks and the test accuracy of each of the sub-inference networks after pruning, and obtain the accuracy loss of each sub-inference network at the corresponding pruning rate; All pruning rates in the pruning rate set and each of the sub-inference networks are traversed to obtain a set of precision losses of each of the sub-inference networks at different pruning rates.
6. The method according to claim 1, characterized in that: The pruning of each of the sub-inference networks based on the target pruning ratio and the important weight of each filter in each of the sub-inference networks comprises: Based on the target pruning ratio, determining the number of filters that need to be pruned in each of the sub-inference networks; Based on the important weights of the filters in each sub-inference network, selecting the target filters to be pruned in order of importance from low to high; A pruning operation is performed on the target filter to complete the pruning of each of the sub-inference networks.
7. The method according to claim 1, characterized in that After pruning each of the sub-inference networks based on the target pruning ratio and the important weight of each filter in each of the sub-inference networks, the method further comprises: Select the neural network before pruning as the teacher network and the neural network after pruning as the student network; Using the same target data to perform reasoning on the teacher network and the student network, and obtaining outputs of the teacher network and the student network respectively; The relative entropy loss and the cross entropy loss between the outputs of the teacher network and the student network are calculated, and the student network is trained by combining the relative entropy loss and the cross entropy loss to measure the difference in output distributions between the teacher network and the student network, so that the output distribution of the student network is close to that of the teacher network and conforms to the true label at the same time.
8. A neural network pruning device, characterized in that: include: An acquisition module is configured to acquire data to be inferred, and input the data to be inferred into the neural network, wherein the neural network includes multiple layers of sub-inference networks, each of which is used to infer the data to be inferred, and obtain feature graph information corresponding to each sub-inference network; A first determination module is configured to determine an important weight of each filter in each of the sub-inference networks according to the feature map information corresponding to each of the sub-inference networks; A second determination module is configured to determine a pruning rate set corresponding to each of the sub-inference networks, and determine a target pruning ratio of each of the sub-inference networks based on the important weights of the filters in each of the sub-inference networks and the pruning rate set corresponding to each of the sub-inference networks; A pruning module is configured to prune each of the sub-inference networks based on the target pruning ratio and the important weight of each filter in each of the sub-inference networks.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
