An image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention

Abstract

The application discloses an image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention, and belongs to the fields of deep learning, computer vision and neural network pruning. Through reasonable setting of a parameter threshold and a parameter cycle compensation scheme, under the premise that no additional training overhead is introduced, the layer collapse phenomenon in initialization pruning is effectively alleviated, and the performance of the neural network initialization pruning method is improved. Meanwhile, the application strengthens the constraint on the network layer with net parameter outflow through the design of a parameter retention module, and further improves the robustness of the method.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning, computer vision and neural network pruning, and designs a high-precision neural network model parameter optimization method based on parameter threshold compensation and retention. BACKGROUND

[0002] Neural network model pruning is a basic topic and research hotspot in the field of computer vision, and its main task is to compress the parameter quantity of the neural network model and reduce the inference delay of the neural network model as much as possible without compromising the performance of the neural network model. In recent years, affected by the growing size of data sets and neural network models, neural network model pruning has developed rapidly and has shown important application value in the fields of autonomous driving, security monitoring, medical analysis, etc. Among the many branches of neural network model pruning, initialization pruning can avoid training on the complete neural network model, which has attracted widespread attention.

[0003] At present, the existing neural network model initialization pruning method uses randomness, parameter amplitude, connection sensitivity, network gradient flow and other standards to evaluate the importance of parameters. However, since the above pruning standards are all evaluated on randomly initialized neural network models that have not been trained at all, the evaluation results have considerable randomness and instability, which leads to the situation that the initialized pruned neural network model is prone to complete pruning of some network layer parameters, i.e. layer collapse phenomenon. The layer collapse phenomenon can seriously damage the performance of the neural network model, and for densely connected neural network models, it can even lead to failure to train and deploy normally. The existing methods for alleviating the layer collapse phenomenon mainly include two types: neural network model initialization pruning method based on iterative pruning and neural network model initialization pruning method based on training facts. The neural network model initialization pruning method based on iterative pruning introduces the idea of iterative pruning into the traditional initialization pruning method, and through the gradual iterative pruning strategy, it can more fully explore the potential information in the randomly initialized neural network model, thereby avoiding the layer collapse phenomenon to some extent. The neural network model initialization pruning method based on training facts takes into account the fact that the neural network model is trained immediately after pruning, and proposes to add a certain number of training steps in the initialization pruning process to avoid the layer collapse phenomenon. However, the above two methods ignore the additional training overhead caused by avoiding the layer collapse phenomenon in initialization pruning, which deviates from the design goal of initialization pruning to avoid training on the complete neural network model.

[0004] To solve the above problems, the application provides a novel neural network model initialization pruning method based on parameter threshold compensation and retention. SUMMARY

[0005] The application provides a neural network model initialization pruning method based on parameter threshold compensation and retention, aiming to solve the problem that the existing initialization pruning method uses iterative pruning and other operations to introduce additional training overhead in the pruning process in order to avoid layer collapse and improve the performance of the initialization pruning method, which deviates from the design goal of the initialization pruning method.

[0006] To achieve the above purpose, the application adopts the following specific technical solutions: an image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention, which comprises the following steps:

[0007] Step 1: pre-process the input image;

[0008] The input image is normalized, scaled, horizontally and vertically flipped, and randomly cropped to obtain the pre-processed image;

[0009] Step 2: calculate the parameter importance of each network layer parameter in the trained neural network model;

[0010] Step 3: set the parameter threshold Th of the pruning network according to the following constraints, wherein the parameter threshold represents the lower limit of the number of parameters retained by each network layer in the neural network model;

[0011] Constraint 1: the set parameter threshold is not greater than the parameter quantity of any network layer in the trained neural network model;

[0012] Constraint 2: the number of parameters retained by the neural network model meets the overall pruning rate requirement of the model;

[0013] Constraint 3: the number of parameters retained by each network layer in the neural network model is not completely the same;

[0014] Let the expected parameter retention ratio of the neural network model after pruning be u, the number of network layers be N, and the parameter quantity of each network layer be c l, wherein l ∈ [1, N], and the parameter threshold coefficient is κ; according to constraint 1 and constraint 2, the parameter quantity min of the network layer with the least parameter quantity in the trained neural network model is selected l c l As a reference for setting the parameter threshold, the relationship of the parameter threshold Th is as follows:

[0015]

[0016] According to the constraints of the above formula, a set of constraint relationships about the parameter threshold Th is obtained as follows, wherein the first inequality corresponds to constraint 1, and the second inequality corresponds to constraint 2;

[0017]

[0018] According to the above formula, the constraint relationship about the parameter threshold coefficient κ is as follows:

[0019]

[0020] Define the operation Floor(·), which means rounding down according to the highest significant bit of the current number. According to the second term corresponding to constraint 2 in the above formula, the complete constraint relationship about the parameter threshold coefficient κ is as follows:

[0021]

[0022] Finally, the parameter threshold coefficient κ is selected as the upper bound of the complete constraint relationship, and the parameter threshold Th is substituted into the relationship formula to obtain the parameter threshold setting method as follows:

[0023]

[0024] Step 4: According to the parameter importance obtained in step 2, the trained neural network model is pre-pruned to obtain the parameter quantity of each network layer after pre-pruning c l ′ , and the average value of the parameter quantity of each network layer after pre-pruning c ′ avg ;

[0025] Step 5: The average value of the parameter quantity of each network layer after pre-pruning c ′ avg is taken as the lower limit of the parameter quantity of the network layer that allows the parameter net flow to be out, and the network layer with a parameter quantity less than c ′ avg will not be further pruned, and the calculation method of c ′ avg is as follows:

[0026]

[0027] According to the flow of parameters when the neural network model is pruned, the network layer is divided into a parameter net inflow layer, a no-operation layer and a parameter net outflow layer; the corresponding network layer is defined as:

[0028]

[0029] Step 6: According to the definition of each layer network in step 5, the pre-pruned network is adjusted;

[0030] Step 6.1: The definition of the parameter net inflow layer is to increase the current network parameters of the layer network, and the definition of the parameter net outflow layer is to reduce the current network parameters of the layer network; and the corresponding processing is performed according to the definition of the layer neural network;

[0031] Step 6.2: According to the constraint in step 3, the processed layer neural network is checked, and if the constraint is met, it is considered that the matching is completed and enters step 6.3, and if the constraint is not met, forced matching is performed; the forced matching method is to adjust the layer neural network according to the parameter threshold set by the parameter threshold setting module and the parameter threshold forced to be retained by the parameter importance of the network layer;

[0032] Step 6.3: Steps 6.1 and 6.2 are cycled until all network layers are matched.

[0033] Step 7: The neural network obtained in step 6 is trained using the image preprocessed in step 1.

[0034] Further, the pre-pruning method in step 4 is any one of an initialization pruning method based on parameter amplitude, an SNIP initialization pruning method based on connection sensitivity and a GraSP initialization pruning method based on network gradient flow.

[0035] Further, the specific method for adjusting the layer neural network according to the parameter threshold set by the parameter threshold setting module and the parameter threshold forced to be retained by the parameter importance of the network layer in step 6.2 is to retain the parameters equal to the parameter threshold according to the parameter importance from large to small for the network layer that needs to be forced to match, and consider that the network layer is matched.

[0036] The neural network model initialization pruning method based on parameter threshold compensation and retention proposed in the application effectively alleviates the layer collapse phenomenon in initialization pruning and improves the performance of the neural network initialization pruning method by reasonably setting the parameter threshold and the parameter cycle compensation scheme without introducing any additional training overhead. At the same time, the application further improves the robustness of the method by designing a parameter retention module to strengthen the constraint on the parameter net outflow layer. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The application is based on the overall flow framework diagram of the neural network initialization pruning method of parameter threshold compensation and retention.

[0038] Figure 2 The application is based on the overall flow framework diagram of the neural network initialization pruning method of parameter threshold compensation and retention. DETAILED DESCRIPTION

[0039] The application is implemented in the image classification task of the Cifar10, Cifar100, Tiny-ImageNet or ImageNet image database based on the Pytorch deep learning platform, and the implementation manner specifically includes the following steps:

[0040] Step 1: according to the scale of the image classification task to be processed, a suitable neural network model is selected, such as the densely connected VGG series neural network model and the ResNet series neural network model with residual connection.

[0041] Step 2: select a suitable neural network model initialization pruning method, such as the initialization pruning method based on parameter amplitude, the SNIP initialization pruning method based on connection sensitivity and the GraSP initialization pruning method based on network gradient flow.

[0042] Step 3: input the images in the image database into the image preprocessing module to obtain the preprocessed images with the long side of 224 and the short side of 224. In the neural network model training process, the images are randomly horizontally flipped with a probability of 0.5 for data augmentation.

[0043] Step 4: use the parameter importance evaluation module to evaluate the parameter importance of the neural network model according to the selected initialization pruning method, obtain the parameter importance evaluation result, and use it for subsequent parameter threshold setting and the like. The parameter importance evaluation method completely follows the requirements of the initialization pruning method selected in step 2 and does not do any additional processing.

[0044] Step 5: Set a reasonable parameter threshold for the neural network model according to the specific rules, which is an important basis for subsequent parameter cycle compensation. First, introduce three basic constraints for parameter threshold setting: (1) The set parameter threshold is not greater than the parameter quantity of any network layer in the neural network model; (2) The parameter quantity reserved by the neural network model meets the overall pruning rate requirement of the model; (3) The parameter quantity reserved by each network layer in the neural network model should not be completely the same. Constraint 1 and constraint 2 establish the minimum requirement for setting the parameter threshold from a mathematical point of view, and the design idea of constraint 3 comes from the observation of the existing neural network model initialization pruning method and the analysis of the logical design of the neural network model structure. That is, different network layers in the neural network model have different sizes and different average parameter importance in pruning, which reflects the different importance of different network layers, so it is not desirable to reserve the same number of parameters for each network layer.

[0045] Then set a suitable parameter threshold according to the three basic constraints.

[0046] Assume that the expected parameter reservation ratio of the neural network model after pruning is u, the number of network layers is N, and the parameter quantity of each network layer is c l , where l ∈ [1, N], and the parameter threshold coefficient is κ. Then according to constraints 1 and 2, select the parameter quantity of the network layer with the least parameter quantity in the neural network model as min l c l As the basis for setting the parameter threshold, the relationship of the parameter threshold Th is as follows.

[0047]

[0048] Considering the constraints of constraints 1 and 2 on the above formula, a set of constraint relationships about the parameter threshold Th is obtained as follows, where the first inequality corresponds to constraint 1 and the second inequality corresponds to constraint 2.

[0049]

[0050] Simplify the above two inequality constraints about the parameter threshold Th to obtain the constraint relationship about the parameter threshold coefficient κ as follows, where the first term corresponds to constraint 1 and the second term corresponds to constraint 2.

[0051]

[0052] Further, the influence of constraint 3 on the parameter threshold coefficient κ is considered. Since the degree of constraint of constraint 1 depends on the parameter retention ratio u of the neural network model, which is not controllable in the pruning process, constraint 3 is used to more strongly limit the constraint relationship corresponding to constraint 2. A new operation Floor(·) is defined, which means rounding down to the highest significant bit of the current number, such as Floor(543) = 500. The second term corresponding to constraint 2 in the above formula is limited in this way, and the complete constraint relationship for the parameter threshold coefficient κ is shown in the following formula.

[0053]

[0054] Finally, the parameter threshold coefficient κ is selected as the upper bound of the complete constraint relationship, and the parameter threshold Th is substituted into the relationship formula to obtain the parameter threshold setting method as shown in the following formula, thereby completing the setting of the parameter threshold of the neural network model.

[0055] Step 6: According to the flow of parameters in the neural network model, the network layers are divided into parameter net inflow layers, no-operation layers, and parameter net outflow layers. Since the actual importance of a parameter is not only related to the parameter importance evaluation result in the neural network model, but also related to the current parameter retention situation of the network layer, such as the negative correlation between the importance degree of a specific parameter in the network layer and the actual parameter retention ratio of the network layer; therefore, in this module, the parameter net outflow layer is additionally limited to avoid further pruning of layers with fewer retained parameters. Specifically, the parameter quantity of each network layer after pre-pruning of the neural network model according to the parameter importance evaluation result obtained from the parameter importance evaluation module is c l ′ The average value c ′ avg of the parameter quantity of each network layer after pre-pruning is used as the lower limit of the parameter quantity of the network layer that allows parameter net outflow. The network layer with a parameter quantity less than c ′ will not be further pruned, and the network layer with a parameter quantity greater than c ′ avg The calculation method of c is shown in the following formula.

[0056]

[0057] In summary, the parameter threshold setting module and the parameter retention module limit each network layer in the neural network model as shown in the following formula.

[0058]

[0059] Step 7: According to the definitions of parameter importance of the neural network model, parameter threshold, and parameter net inflow layer and parameter net outflow layer, the parameter transfer is performed on the neural network model using the parameter cycle compensation module, the re-distribution of the reserved parameters in the neural network model is realized, and the pruned neural network model is obtained.

[0060] According to the parameter threshold determined by the parameter threshold setting module and the concept of parameter net inflow layer and parameter net outflow layer divided by the parameter retention module, the number of reserved parameters in each network layer of the neural network model is adjusted to achieve the effect of parameter compensation. The main operation of this module includes three steps of pre-pruning, checking and matching. In the pre-pruning step, the network layers that have not yet completed matching are pre-pruned according to the weight importance obtained in the parameter importance evaluation module. In this stage, the weight mask of the neural network model is not adjusted, and only the potential impact of pruning on the parameter distribution structure of each network layer is evaluated. In the checking step, according to the parameter distribution structure obtained by pre-pruning, it is checked whether each network layer meets the requirements of the parameter threshold set in the parameter threshold setting module. In the matching step, the network layers that do not meet the parameter threshold requirements are forced to match, that is, the number of parameters that meet the parameter threshold requirements is forced to be reserved according to the parameter threshold determined by the parameter threshold setting module and the weight importance obtained by the parameter importance evaluation module, and is marked as matching completed. All network layers that have met the parameter threshold requirements in the current round are regarded as a new neural network model, and the next round of parameter cycle compensation process continues to be processed. The above three steps are executed in a loop until all network layers in the neural network model are matched. Since the above matching process does not require any additional training of the neural network model, the method does not require additional training overhead.

[0061] Step 8: The pruned neural network model is trained using the neural network model training module to restore its performance. The training process optimizes the network model using the stochastic gradient descent method, a total of 160 epochs are trained, the initial learning rate is set to 0.01, and the learning rate is decayed at the 80th and 120th epochs during the training process, and the decay coefficient is 0.1.

[0062] The application initiates and prunes the neural network model through parameter threshold compensation and retention, proposes a parameter threshold setting module and a parameter cycle compensation module, effectively alleviates the layer collapse phenomenon in the initialization pruning without introducing any additional training overhead, improves the performance of the neural network model initialization pruning method, and at the same time proposes a parameter retention module to make stronger constraints on the network layer with net outflow, further improves the robustness of the method proposed in the application. On the commonly used image classification task database Cifar100, the VGG series neural network model with 16 network layers is experimented, and compared with the GraSP initialization pruning method based on network gradient flow, the average performance is improved by 1.13% when the pruning rate is 0.99, and the average performance is improved by 31.45% when the pruning rate is 0.999.

Claims

1. An image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention, the method comprising: Step 1: preprocessing the input image; normalizing, scaling, horizontal and vertical flipping, and random cropping the input image to obtain the preprocessed image; Step 2: calculating the parameter importance of each network layer parameter in the trained neural network model; Step 3: setting the parameter threshold of the pruning network according to the following constraints The parameter threshold represents the lower limit of the number of parameters retained in each network layer in the neural network model. Constraint 1: the set parameter threshold is not greater than the parameter quantity of any network layer in the trained neural network model; Constraint 2: the number of parameters retained by the neural network model meets the overall pruning rate requirement of the model; Constraint 3: the number of parameters retained by each network layer in the neural network model is not completely the same; The expected parameter retention ratio of the neural network model after pruning is , the number of network layers is , the parameter quantity of each network layer is , wherein , the parameter threshold coefficient is ; according to constraint 1 and constraint 2, the parameter quantity of the network layer with the least parameter quantity in the neural network model that has been trained is selected as a reference for setting the parameter threshold, and the parameter threshold is obtained, and the relationship formula is as follows: ； According to the constraints of the above equation by constraint 1 and constraint 2, a set of constraint relationships about the parameter threshold as shown below is obtained, wherein the first inequality corresponds to constraint 1, and the second inequality corresponds to constraint 2. ​ ； According to the above equation, the constraint relationship about the parameter threshold coefficient is shown in the following equation; ； Definition of operation The meaning of this operation is to round down to the highest significant bit of the current number, resulting in a complete constraint relationship with respect to the parameter threshold coefficient is shown in the following equation; ； Finally, the parameter threshold coefficient is selected is substituted into the relationship of the parameter threshold The parameter threshold setting mode is shown in the following formula ； Step 4: The trained neural network model is pre-pruned according to the parameter importance obtained in step 2, and the number of parameters of each network layer after pre-pruning is , and the average number of parameters of each network layer after pre-pruning is . Step 5: Average the number of parameters of each network layer before pre-pruning As the lower limit of the number of parameters for a network layer that allows parameter net outflow, the network layer with the number of parameters less than will not be further pruned, The calculation method is as follows: ； According to the flow of parameters when pruning the neural network model, the network layers are divided into parameter net inflow layer, no operation layer, and parameter net outflow layer; the corresponding network layers are defined as: ； Step 6: adjusting the pre-pruned network according to the definition of each layer network in step 5; Step 6.1: defining the parameter net inflow layer as increasing the current network parameters of the layer network, and defining the parameter net outflow layer as reducing the current network parameters of the layer network; according to the definition of the layer neural network, the corresponding processing is performed; Step 6.2: according to the constraints of step 3, the processed neural network layers are checked, and if the constraints are met, it is considered that the matching is completed and enters step 6.3, otherwise, forced matching is performed; the forced matching method is to adjust the neural network layers according to the determined parameter threshold and parameter importance forced parameter threshold retained by the network layer; Step 6.3: repeating step 6.1 and step 6.2 until all network layers are matched; Step 7: training the neural network obtained in step 6 using the preprocessed image in step 1.

2. The image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention according to claim 1, characterized in that, The pre-pruning method in step 4 is any one of the initialization pruning method based on parameter amplitude, the SNIP initialization pruning method based on connection sensitivity, and the GraSP initialization pruning method based on network gradient flow.

3. The image recognition neural network parameter quantity optimization method based on parameter threshold compensation and retention of claim 1, wherein, In step 6.2, the specific method for adjusting the neural network layers according to the determined parameter threshold and parameter importance forced parameter threshold retained by the network layer is to retain the parameters with the number equal to the parameter threshold in the network layer that needs to be forced to match according to the parameter importance from large to small, and consider that the network layer is matched.

Images