A High-Precision Large-Scale Structured Network Compression Method Based on Minimum Entropy

Abstract

This specification discloses a high-precision large-scale structured network compression method based on minimum entropy, belonging to the technical field of deep model compression, including building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the network to be pruned, and the outputs of the base network and the network to be pruned are respectively input into the MSE loss function module and the discriminator; based on the minimum entropy pruning framework, an optimization objective function is constructed; the optimization objective function is iteratively optimized to obtain the soft mask of the network to be pruned; based on the soft mask and the pruning selection of the model structure, model pruning is performed to obtain the pruned model, so as to solve the problems of low efficiency, poor flexibility and single use scenario existing in the current structured pruning methods.

Description

Technical Field

[0001] The present invention relates to the field of deep model compression technology, and in particular to a high-precision, large-scale structured network compression method based on minimum entropy. Background Art

[0002] Deep model compression technology aims to compress deep neural network models without significantly reducing their accuracy, thereby accelerating model inference and reducing model storage usage. Currently, mainstream deep model compression technologies include model pruning, model quantization, low-rank decomposition, knowledge distillation, network structure search, and compact structure design. The most closely related publicly available prior art to this application is structured model pruning. Structured model pruning involves using the complete structure of a neural network model as the granularity (such as filter convolution kernels in a convolutional neural network, residual blocks in a residual network, and branches in a multi-branch network), removing redundant network structures and retaining those that are effective for the output. Among these structured model pruning methods, a common approach is to add constraints to the loss function during model training or to use the model pruning decision as an optimization objective, thereby learning the model's pruning decisions simultaneously during training. For example, a pruning method proposed by Lebedev et al. adds a structured sparsity term to the loss function. This allows the sparsity term to be optimized during model training. Finally, when the model training is completed, the filter weights of the corresponding sparse items below the threshold are set to 0, thus achieving the pruning of the convolutional network model. Another structured model pruning method is to judge the significance of the model structure by the norm size. For example, for a convolutional network model, the convolution filter can be measured. Norm, for a specified layer of convolution operation, a certain number of layers with the minimum The filter is filtered by the norm and the corresponding feature map is also deleted. After the filter is pruned, the model is retrained to restore or improve its recognition accuracy. Some structured model pruning methods also use the characteristics of the deep model calculation process to judge the significance of the filter. For example, the output of a deep model using the ReLU function as the activation function is sparse. In a model pruning method proposed by Hu et al., they used the proportion of non-zero values in the feature map output by the convolutional network model as the basis for judging the importance of the filter. In addition, the geometric median of the filter and the average rank of the feature map have also been proposed as criteria for judging the importance of the filter.

[0003] However, current structured pruning methods also face many challenges. There are three potential problems with structured pruning methods:

[0004] (1) Low efficiency caused by offline pruning. Many structured model pruning techniques adopt offline pruning, which can only be carried out after the model training is completed, and the pruned model needs to be retrained after pruning. For example, in the model pruning method proposed by Luo, He and others, pruning needs to be carried out layer by layer, and redundant filters in the current layer are identified according to the channels in the next layer. In the method of pruning based on filter norms, filters with lower importance need to be gradually removed, and then the pruned network is fine-tuned layer by layer. Therefore, the time consumed for model pruning is relatively long and the efficiency is low.

[0005] (2) Low flexibility caused by hard filter pruning. The mask adopted by hard filter pruning is a 0-1 binary global mask, so it lacks flexibility. To solve the optimization problem of the binary mask, a greedy alternating update strategy needs to be used, and the process is relatively complex. Moreover, hard filter pruning is highly dependent on prior knowledge of filters.

[0006] (3) Dependence on class labels. Most of the existing online pruning methods need to perform model pruning through a supervised learning process. This makes the model pruning method need to be applied to a dataset with class labels, and it is difficult to be applied in the case where the data lacks class labels or needs to be migrated to other datasets with different class labels. Summary of the Invention

[0007] The purpose of the present invention is to provide a high-precision large-scale structured network compression method based on minimum entropy to solve the problems of low efficiency, poor flexibility and single use scenario existing in the current structured pruning methods.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] On the one hand, this specification provides a high-precision large-scale structured network compression method based on minimum entropy, including:

[0010] Step 102, build a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator;

[0011] Step 104, construct an optimization objective function based on the minimum entropy pruning framework;

[0012] Step 106, perform iterative optimization on the optimization objective function to obtain the soft mask of the pruned network;

[0013] Step 108, perform model pruning based on the soft mask and the pruning selection of the model structure to obtain the pruned model.

[0014] On the other hand, this specification provides a high-precision large-scale structured network compression device based on minimum entropy, including:

[0015] A pruning framework construction module for constructing a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator;

[0016] An optimization function construction module for constructing an optimization objective function based on the minimum entropy pruning framework;

[0017] A function iterative optimization module for iteratively optimizing the optimization objective function to obtain a soft mask of the pruned network;

[0018] A model pruning selection module for pruning the model based on the soft mask and pruning selection of the model structure to obtain the pruned model.

[0019] Based on the above technical solutions, this specification can achieve the following technical effects:

[0020] (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network as close as possible to the output feature distribution of the base network, and to make the model pruning process independent of labels. That is, in the model pruning framework, a learnable discriminator is introduced to distinguish the outputs of the base network and the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters (soft mask and weights) of the pruned network and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the gap between the output of the pruned network and the output of the base network; while when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discrimination difference of the discriminator for the outputs of the base network and the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made as close as possible to the output feature distribution of the base network. And the label-independent characteristic in the optimization method enables the pruning method to be used in unlabeled scenarios.

[0021] (2) Soft filtering pruning is implemented using a soft mask based on score norm constraint. Different from existing hard filtering, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the complexity of mask optimization. In order to obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative threshold shrinkage algorithm based on score norm.

[0022] (3) Model pruning based on channel selection, block selection, and branch selection in an end-to-end optimization manner. Unlike the layer-by-layer multi-stage iterative pruning method, the model pruning method in this application adopts an end-to-end optimization method. By setting a loss function with the soft mask, weights, and discriminator weights of the pruned model as independent variables, and optimizing the loss function in a staged optimization method, the soft mask of the entire pruned model is updated end-to-end, and channel selection, block selection, and branch selection are determined based on the soft mask, the efficiency of model pruning is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The figure is a flowchart of a high-precision, large-scale structured network compression method based on minimum entropy in one embodiment of the present invention.

[0024] Figure 2 Schematic diagram of a minimum entropy pruning framework in one embodiment of the present invention.

[0025] Figure 3 Schematic diagram of the structure of a high-precision, large-scale structured network compression device based on minimum entropy in one embodiment of the present invention.

[0026] Figure 4 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in a very simplified form and are not to exact scale. They are only used for the purpose of conveniently and clearly illustrating the embodiments of the present invention.

[0028] It should be noted that, in order to clearly illustrate the contents of the present invention, the present invention specifically provides multiple embodiments to further illustrate different implementations of the present invention. These multiple embodiments are provided in an enumerated manner rather than an exhaustive manner. Furthermore, for the sake of brevity, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, for the contents not mentioned in the subsequent embodiments, reference may be made to the previous embodiments accordingly.

[0029] Example 1

[0030] Please refer to Figure 1 , Figure 1 The figure shows a high-precision, large-scale structured network compression method based on minimum entropy provided by this embodiment. In this embodiment, the method includes:

[0031] Step 102: Building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are input into the MSE loss function module and the discriminator respectively;

[0032] Step 104: Construct an optimization objective function based on the minimum entropy pruning framework;

[0033] In this embodiment, the optimization objective function includes an adversarial objective function between the discriminator and the pruned network, an output loss function between the pruned network and the base network, and regularization constraint terms for the parameters of the pruned network and the discriminator.

[0034] In this embodiment, one implementation of Step 104 is as follows:

[0035] Step 202: Characterize the adversarial relationship between the discriminator and the pruned network using an adversarial loss function with minimum entropy to obtain the adversarial objective function between the discriminator and the pruned network;

[0036] Step 204: Characterize the loss value of the output of the pruned network relative to the output of the base network using the MSE loss function to obtain the output loss function between the pruned network and the base network;

[0037] Step 206: Obtain regularization constraint terms for the parameters of the pruned network and the discriminator based on the weight parameters of the pruned network and the discriminator.

[0038] Step 106: Iteratively optimize the optimization objective function to obtain the soft mask of the pruned network;

[0039] In this embodiment, one implementation of Step 106 is as follows:

[0040] Step 302: Obtain base network sample features and pruned network sample features based on the training sample dataset;

[0041] In this embodiment, one implementation of Step 302 is as follows:

[0042] Step 3021: Divide the training sample dataset into several batches to obtain several sub-datasets of batches;

[0043] Step 3022: Propagate each batch of sub-datasets forward through the base network and the pruned network respectively to obtain several base network sample features and several pruned network sample features.

[0044] Step 304: Obtain a first loss function and a second loss function based on the base network sample features, the pruned network sample features, and the optimization objective function;

[0045] In this embodiment, one implementation of Step 304 is as follows:

[0046] Step 3041: Obtain the first loss function based on the base network sample features, the pruned network sample features, the adversarial objective function of the optimization objective function, and the regularization constraint term of the discriminator;

[0047] Step 3042: Obtain a second loss function based on the base network sample features, the pruned network sample features, the output loss function of the optimization objective function, and the regularization constraint term of the pruned network parameters.

[0048] Step 306: Fix the parameters of the pruned network, and optimize the parameters of the discriminator using the first loss function to obtain the updated weights of the discriminator.

[0049] In this embodiment, one implementation of Step 306 is as follows:

[0050] Fix the parameters of the pruned network, and update the weights of the discriminator based on the first loss function according to the gradient descent method to obtain the updated weights of the discriminator.

[0051] Step 308: Fix the weights of the discriminator, and optimize the parameters of the pruned network using the second loss function to obtain the updated parameters of the pruned network; the parameters of the pruned network include the weights and soft masks of the pruned network.

[0052] In this embodiment, one implementation of Step 308 is as follows:

[0053] Step 3081: Fix the weights of the discriminator, and alternately update the weights and soft masks of the pruned network based on the second loss function to obtain the updated weights and soft masks of the pruned network.

[0054] In this embodiment, one implementation of Step 3081 is as follows:

[0055] Step 402: Fix the soft mask of the pruned network, and update the weights of the pruned network using the stochastic gradient descent method with momentum to obtain the updated weights of the pruned network.

[0056] Step 404: Fix the weights of the pruned network, and update the soft mask of the pruned network using the fast iterative soft threshold algorithm to obtain the updated soft mask of the pruned network.

[0057] Step 108: Perform model pruning based on the soft mask and the pruning selection of the model structure to obtain the pruned model.

[0058] In this embodiment, one implementation of Step 108 is as follows:

[0059] Step 502: If the pruning selection of the model structure is channel selection, add the scaling factor of the soft mask to each input channel of the model, and remove the input channels with a scaling factor of 0 and the corresponding output channels from the model after model optimization.

[0060] Step 504: If the pruning of the model structure is block selection, add a scaling factor of the soft mask to each residual block of the model, and after optimizing the model, remove the residual blocks with a scaling factor of 0 from the model to obtain the pruned model;

[0061] Step 506: If the pruning of the model structure is branch selection, add a scaling factor of the soft mask to each branch of the model, and after optimizing the model, remove the branches with a scaling factor of 0 from the model to obtain the pruned model.

[0062] Specifically, the specific steps of a high-precision large-scale structured network compression method based on minimum entropy include:

[0063] The first step: Build a model pruning framework

[0064] Build a model pruning framework, connect the base network and the pruned network in parallel, and use the mean square loss function and a learnable discriminator to compare the outputs of the base network and the pruned network.

[0065] As Figure 2 , connect the original pre-trained base network and the pruned network in parallel, where and represent the input image, the weights of the base network, and the weights of the pruned network respectively, represents the soft mask of the fractional norm constraint of each network structure in the pruned network. Denote the th element in as . The output features of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator (with weights

[0066] The second step: Define the optimization objective function

[0067] Define the optimization objective function, which includes the adversarial objective function between the discriminator and the pruned network, the output loss function between the pruned network and the base network, and the regularization constraint terms of the parameters of the pruned network and the discriminator.

[0068] We transform the model pruning process into the following optimization problem

[0069]

[0070] where represents the adversarial loss based on minimum entropy, which is used for the confrontation between the pruned model and the discriminator, and is defined as

[0071] [[ID=5�]]

[0072] where represents the distribution of the input data, and 0 <= a <= 1 is set according to the actual situation.

[0073] is the loss value between the outputs of the base network and the pruned network, which is used to align the output distributions of the two networks. We use the MSE loss to characterize the difference between the outputs of the two networks, and its formula is:

[0074]

[0075] represents the regularization constraint acting on and and can be decomposed into three parts:

[0076]

[0077] where is the weight decay regularization term in the pruned network and can be defined as is the sparsity regularization constraint for the soft mask and is a hyperparameter set by humans and can be specifically defined as is the regularization constraint for the discriminator weights, which prevents the optimization of the discriminator from dominating the model pruning process during the optimization process and can be defined as [[ID=३८]]

[0078]

[0079] Step 3 Iteratively optimize and solve for parameters

[0080] Iteratively optimize and solve for the optimal parameters of the pruned network and the discriminator. The optimization process is divided into two stages, namely the adversarial optimization stage of the discriminator and the pruned network, and the alignment optimization stage of the pruned network fitting the output distribution of the base network.

[0081] Given training sample datasets and the base network model parameters set the sparsity factor of score regularization the learning rate of the stochastic gradient descent method and the maximum number of iterations . First, initialize the weights of the pruned network and set them to be the same as the weights of the pre-trained base network, that is , and then randomly initialize the scaling factor of the mask according to the normal distribution .

[0082] The pruning process starts from the iteration round Start iterating until is reached. Each iteration round can be divided into two phases.

[0083] Phase 1: Fix the parameters of the pruned network , and optimize the discriminator parameters to counter the pruned network. Specifically, divide the dataset into several batches of data. For each batch of data, forward propagate it through the base network to obtain sample features , and forward propagate it through the pruned network to obtain sample features . Using the function

[0084]

[0085] as the loss function, update the weights of the discriminator according to the gradient descent method. This loss function is composed of the adversarial objective function and the discriminator regularization constraint terms.

[0086] Phase 2: Freeze the weights of the discriminator , and update the parameters and of the pruned network to align the outputs of the pruned network and the base network. Similar to Phase 1, divide the dataset into several batches. For each batch of data, forward propagate it through the base network to obtain sample features , and forward propagate it through the pruned network to obtain sample features . Since is not updated in this phase, let

[0087]

[0088] Then add the regularization term to it, and the overall loss function for this phase is

[0089]

[0090] For the optimization of formula (8), it can also be achieved by alternately updating and :

[0091] 1. Fix , and use the stochastic gradient descent method with momentum to update ;

[0092] 2. Fix , and we use FISTA to optimize and update , specifically, first initialize , and then update according to the following formula (9) :

[0093]

[0094] where is the learning rate at the -th iteration, and

[0095]

[0096] Step 4 Model Structure Pruning Selection

[0097] Use the model pruning soft mask obtained by optimization to perform pruning selection on the model structure. According to the different model types and pruning granularities, it is divided into channel selection, branch selection, and block selection.

[0098] After optimizing the solution parameters, we obtain a mask that regularizes the scores representing the model pruning decisions , and then we use the mask to prune the redundant model structure. The structure selection of the model can be divided into three granularities: channel selection, block selection, and branch selection.

[0099] Channel Selection (channel selection): Channel selection refers to using the pruning mask to select the effective output / input channels from the convolutional network and removing the redundant channels. In our pruning method, a soft mask is added to the input channels of each layer of the convolutional network. We assume that after optimization, the parameter matrix corresponding to the -th layer of the pruned network and the -th convolutional kernel corresponding to the input -th channel is , and the mask scaling factor of the -th layer and the -th input channel is . Then the -th feature map output by this layer should be

[0100]

[0101] where * represents the convolution operation, represents a non-linear transformation such as ReLU. For the filter with a scaling factor of and its corresponding output feature channel and the input channel of the next layer, we regard them as channels with lower importance and delete them from the model.

[0102] Block selection: For the pruning of the ResNet network, there is a pruning granularity of block selection. By setting the residual mapping value of the residual block to 0 while retaining the identity mapping of the input, the residual block structure can be removed without cutting off the information flow of the network. The calculation formula of the residual block after adding the soft mask is as follows:

[0103]

[0104] where and represent the input and output of the th residual block respectively. represents the convolutional network branch in the residual block, is all the convolutional filter weights in the th residual block, and represents the skip connection in the residual block. For the optimized residual block with the scaling factor in the soft mask, we can remove its convolutional network and only retain the skip connection after training, without causing a significant change in the network output accuracy.

[0105] Branch selection: For multi-branch networks (such as GoogLeNet and ResNeXts, etc.), the model structure can be selected according to the granularity of the branches. Similar to the residual block mask of ResNet, a scaling factor of the soft mask is added to each branch of the multi-branch network. After the model training and pruning are completed, we remove the branches with the corresponding scale factor of 0. Taking GoogLeNet as an example, the calculation formula of the initial module (inception) with a soft mask is:

[0106]

[0107] where represents the concatenation operation. represents the output of the th branch in the module, and its weight is . is the total number of branches. If among them, then we consider that the th branch can be removed without causing a significant change in the model accuracy.

[0108] The above is the whole process of structured network pruning based on minimum entropy.

[0109] In summary, the present method can obtain the following technical effects:

[0110] (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network approximate the output feature distribution of the base network as much as possible, and to make the model pruning process independent of labels. That is, in the framework of model pruning, a learnable discriminator is introduced to discriminate the output of the base network and the output of the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters (soft mask and weights) of the pruned network and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the gap between the output of the pruned network and the output of the base network; while when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discrimination difference of the discriminator for the output of the base network and the output of the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made to approximate the output feature distribution of the base network as much as possible. And the label-independent feature in the optimization method enables the pruning method to be used in unlabeled scenarios.

[0111] (2) Soft filter pruning is implemented using a soft mask based on the fractional norm constraint. Different from existing hard filters, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the complexity of mask optimization. To obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative fractional norm-based threshold shrinkage algorithm.

[0112] (3) Model pruning is performed in an end-to-end optimization manner based on channel selection, block selection, and branch selection. Different from the method of layer-by-layer multi-stage iterative pruning, the model pruning method in this application adopts an end-to-end optimization manner. By setting a loss function with the soft mask, weights, and discriminator weights of the pruned model as independent variables, and optimizing the loss function in a staged optimization manner, the soft mask of the entire pruned model is updated end-to-end, and channel selection, block selection, and branch selection are determined according to the soft mask, greatly improving the efficiency of model pruning.

[0113] Embodiment 2

[0114] Please refer to Figure 3 , Figure 3 shown in the figure is a high-precision large-scale structured network compression device based on minimum entropy provided in this embodiment. In this embodiment, the device includes:

[0115] A pruning framework building module for building a minimum entropy pruning framework; the base network and the pruned network of the minimum entropy pruning framework are connected in parallel, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator;

[0116] An optimization function construction module for constructing an optimization objective function based on the minimum entropy pruning framework;

[0117] A function iteration optimization module for iteratively optimizing an optimization objective function to obtain a soft mask of the pruned network;

[0118] A model pruning selection module for performing model pruning based on the soft mask and pruning selection of the model structure to obtain a pruned model.

[0119] Optionally, the optimization objective function includes an adversarial objective function between a discriminator and the pruned network, an output loss function between the pruned network and a base network, and a regularization constraint term for the parameters of the pruned network and the discriminator.

[0120] Optionally, the optimization function construction module includes:

[0121] An adversarial objective function construction unit for characterizing the adversarial relationship between the discriminator and the pruned network using a minimum entropy adversarial loss function to obtain an adversarial objective function between the discriminator and the pruned network;

[0122] An output loss function construction unit for characterizing the loss value of the output of the pruned network relative to the output of the base network using an MSE loss function to obtain an output loss function between the pruned network and the base network;

[0123] A regularization constraint term determination unit for obtaining a regularization constraint term for the parameters of the pruned network and the discriminator based on the weight parameters of the pruned network and the discriminator.

[0124] Optionally, the function iteration optimization module includes:

[0125] A sample feature acquisition sub-module for obtaining base network sample features and pruned network sample features based on a training sample data set;

[0126] A loss function determination sub-module for obtaining a first loss function and a second loss function based on the base network sample features, the pruned network sample features, and the optimization objective function;

[0127] A discriminator optimization sub-module for fixing the parameters of the pruned network and optimizing the parameters of the discriminator using the first loss function to obtain the updated weights of the discriminator;

[0128] A pruned network optimization sub-module for fixing the weights of the discriminator and optimizing the parameters of the pruned network using the second loss function to obtain the updated parameters of the pruned network; the parameters of the pruned network include the weights and the soft mask of the pruned network.

[0129] Optionally, the sample feature acquisition sub-module includes:

[0130] A sample data division unit for dividing a training sample data set into several batches to obtain several sub-data sets of batches;

[0131] A sample feature generation unit for respectively propagating each batch of sub-data sets forward through a base network and a pruned network to obtain several base network sample features and several pruned network sample features.

[0132] Optionally, the loss function determination sub-module includes:

[0133] A first loss function construction unit for obtaining a first loss function based on the base network sample features, the pruned network sample features, the adversarial objective function of the optimization objective function, and the regularization constraint term of the discriminator;

[0134] A second loss function construction unit for obtaining a second loss function based on the base network sample features, the pruned network sample features, the output loss function of the optimization objective function, and the regularization constraint term of the pruned network parameters.

[0135] Optionally, a discriminator optimization sub-module for fixing the parameters of the pruned network and updating the weights of the discriminator based on the first loss function according to the gradient descent method to obtain the updated weights of the discriminator.

[0136] Optionally, a pruned network optimization sub-module for fixing the weights of the discriminator and alternately updating the weights and soft masks of the pruned network based on the second loss function to obtain the updated weights and soft masks of the pruned network.

[0137] Optionally, the pruned network optimization sub-module includes:

[0138] A weight optimization unit for fixing the soft mask of the pruned network and updating the weights of the pruned network using the stochastic gradient descent method with momentum to obtain the updated weights of the pruned network;

[0139] A soft mask optimization unit for fixing the weights of the pruned network and updating the soft mask of the pruned network using the fast iterative soft threshold algorithm to obtain the updated soft mask of the pruned network.

[0140] Optionally, the model pruning selection module includes:

[0141] A channel selection unit for adding a scaling factor of the soft mask to the input channels of each layer of the model if the pruning selection of the model structure is channel selection, and removing the input channels with a scaling factor of 0 and the corresponding output channels from the model after model optimization;

[0142] A block selection unit, which is used to add a scaling factor of the soft mask to each residual block of the model if the pruning selection of the model structure is block selection, and remove the residual blocks with a scaling factor of 0 from the model after model optimization to obtain the pruned model;

[0143] A branch selection unit, which is used to add a scaling factor of the soft mask to each branch of the model if the pruning selection of the model structure is branch selection, and remove the branches with a scaling factor of 0 from the model after model optimization to obtain the pruned model.

[0144] Based on this, the present device can obtain the following technical effects:

[0145] (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network as close as possible to the output feature distribution of the base network, and make the model pruning process independent of labels. That is, in the framework of model pruning, a learnable discriminator is introduced to distinguish the output of the base network and the output of the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters (soft mask and weights) of the pruned network and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the gap between the output of the pruned network and the output of the base network; while when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discrimination difference of the discriminator for the output of the base network and the output of the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made as close as possible to the output feature distribution of the base network. And the label-independent characteristic in the optimization method enables the pruning method to be used in unlabeled scenarios.

[0146] (2) Soft filter pruning is implemented by using a soft mask based on the score norm constraint. Different from the existing hard filter, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the optimization complexity of the mask. In order to obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative threshold shrinkage algorithm based on the score norm.

[0147] (3) Model pruning is performed in an end-to-end optimization manner based on channel selection, block selection, and branch selection. Different from the method of layer-by-layer multi-stage iterative pruning, the model pruning method in this application adopts an end-to-end optimization manner. By setting a loss function with the soft mask, weights of the pruned model, and weights of the discriminator as independent variables, and optimizing the loss function in a staged optimization manner, the soft mask of the entire pruned model is updated end-to-end, and channel selection, block selection, and branch selection are determined according to the soft mask, greatly improving the efficiency of model pruning.

[0148] Embodiment 3

[0149] Please refer to Figure 4 Figure 4 , this embodiment provides an electronic device, which includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a high-precision large-scale structured network compression method based on minimum entropy at the logical level. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logical devices or the combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0150] The network interface, the processor, and the memory can be interconnected through a bus system. The above bus can be divided into an address bus, a data bus, a control bus, etc.

[0151] The memory is used to store programs. Specifically, the program can include program code, and the above program code includes computer operation instructions. The memory can include a read-only memory and a random access memory, and provide instructions and data to the processor.

[0152] The processor is used to execute the program stored in the above memory, and specifically execute:

[0153] Step 102, build a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator;

[0154] Step 104, based on the minimum entropy pruning framework, construct an optimization objective function;

[0155] Step 106, perform iterative optimization on the optimization objective function to obtain the soft mask of the pruned network;

[0156] Step 108, perform model pruning based on the soft mask and the pruning selection of the model structure to obtain the pruned model.

[0157] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the processor's hardware or instructions in software form.

[0158] Based on the same inventive concept, this embodiment of the specification also provides a computer-readable storage medium. The above computer-readable storage medium stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the above electronic device is caused to execute Figure 1 - Figure 2The corresponding embodiment provides a high-precision large-scale structured network compression method based on minimum entropy.

[0159] Those skilled in the art should understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code.

[0160] In addition, for the specific implementation of the above system, since it is basically similar to the method implementation, the description is relatively simple. For related parts, refer to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of this application, the components are logically divided according to the functions to be realized. However, this application is not limited to this, and the components can be re-divided or combined as needed.

[0161] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

[0162] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the embodiments and still achieve the desired results. Additionally, a specific order or a continuous order shown in the drawings is not necessarily required to achieve the desired results during the process depicted. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] The above are only the embodiments of this application and are not used to limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims

1. A high-precision, large-scale structured network compression method based on minimum entropy, characterized in that: include: Building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator, and the inputs of the base network and the pruned network are images; Based on the minimum entropy pruning framework, the optimization objective function is constructed; The optimization objective function constructed based on the minimum entropy pruning framework includes: The minimum entropy adversarial loss function is used to characterize the adversarial relationship between the discriminator and the pruned network, and the adversarial objective function between the discriminator and the pruned network is obtained. The MSE loss function is used to characterize the loss value of the output of the pruned network relative to the output of the base network, and the output loss function between the pruned network and the base network is obtained; Based on the weight parameters of the pruned network and the discriminator, the regularization constraints of the pruned network parameters and the discriminator parameters are obtained; the optimization objective function is iteratively optimized to obtain the soft mask of the pruned network; The iterative optimization of the optimization objective function to obtain the soft mask of the pruned network includes: Based on the training image sample data set, the base network sample features and the pruned network sample features are obtained; Obtaining a first loss function and a second loss function based on the base network sample features, the pruned network sample features, and the optimization objective function; Fix the parameters of the pruned network, use the first loss function to optimize the parameters of the discriminator, and obtain the updated discriminator weights; Fixing the weight of the discriminator, optimizing the parameters of the pruned network using the second loss function, and obtaining updated parameters of the pruned network; the parameters of the pruned network include the weight and soft mask of the pruned network; Perform model pruning based on soft mask and model structure pruning selection to obtain the pruned model; The pruning selection based on the soft mask and the model structure performs model pruning, and the pruned model obtained includes: If the pruning selection of the model structure is channel selection, a soft mask scaling factor is added to the input channel of each layer of the model, and the input channels with a scaling factor of 0 and the corresponding output channels are removed from the model after the model is optimized; If the pruning selection of the model structure is block selection, a soft mask scaling factor is added to each residual block of the model, and after the model optimization, the residual blocks with a scaling factor of 0 are removed from the model to obtain the pruned model; If the pruning selection of the model structure is branch selection, a soft mask scaling factor is added to each branch of the model, and after the model is optimized, branches with a scaling factor of 0 are removed from the model to obtain the pruned model.

2. The method according to claim 1, characterized in that The optimization objective function includes an adversarial objective function between the discriminator and the pruned network, an output loss function between the pruned network and the base network, and regular constraints on the pruned network parameters and the discriminator parameters.

3. The method according to claim 1, characterized in that The obtaining of base network sample features and pruned network sample features based on the training image sample dataset includes: Divide the training image sample dataset into several batches to obtain several batches of sub-datasets; Each batch of sub-datasets is forward propagated through the base network and the pruned network respectively to obtain several base network sample features and several pruned network sample features.

4. The method according to claim 3, characterized in that The obtaining of the first loss function and the second loss function based on the base network sample features, the pruned network sample features and the optimization objective function includes: Based on the sample features of the base network and the pruned network, as well as the adversarial objective function of the optimization objective function and the regularization constraint term of the discriminator, a first loss function is obtained; A second loss function is obtained based on the sample features of the base network and the sample features of the pruned network, the output loss function of the optimization objective function, and the regularization constraint terms of the pruned network parameters.

5. The method according to claim 4, characterized in that The parameters of the pruned network are fixed, and the parameters of the discriminator are optimized using the first loss function to obtain the updated weights of the discriminator. The parameters of the pruned network are fixed, and based on the first loss function, the weights of the discriminator are updated according to the gradient descent method to obtain the updated weights of the discriminator.

6. The method according to claim 5, characterized in that The weight of the fixed discriminator is optimized using the second loss function to obtain updated parameters of the pruned network, including: The weights of the discriminator are fixed, and the weights and soft masks of the pruned network are alternately updated based on the second loss function to obtain the updated weights and soft masks of the pruned network.

7. The method according to claim 6, characterized in that The weight of the fixed discriminator is alternately updated based on the second loss function on the weight and soft mask of the pruned network to obtain the updated weight and soft mask of the pruned network, comprising: Fix the soft mask of the pruned network, and use the stochastic gradient descent method with momentum to update the weights of the pruned network to obtain the updated weights of the pruned network; The weights of the pruned network are fixed, and the soft mask of the pruned network is updated using a fast iterative soft threshold algorithm to obtain an updated soft mask of the pruned network.

Images