High-precision large-proportion structured network compression method based on minimum entropy

Abstract

The invention discloses a high-precision large-proportion structured network compression method based on minimum entropy, and belongs to the technical field of depth model compression, and the method comprises the steps: building a minimum entropy pruning framework; a base network and a pruned network of the minimum entropy pruning framework are connected in parallel, and outputs of the base network and the pruned network are respectively input into an MSE loss function module and a discriminator; constructing an optimization objective function based on a minimum entropy pruning framework; performing iterative optimization on the optimization objective function to obtain a soft mask of the pruned network; and performing model pruning based on pruning selection of the soft mask and the model structure to obtain a pruned model so as to solve the problems of low efficiency, poor flexibility and use scene simplification of the existing structured pruning method.

Description

Technical Field

[0001] The present invention relates to the technical field of deep model compression, and in particular to a high-precision large-scale structured network compression method based on minimum entropy. Background Art

[0002] Deep model compression technology aims to compress the network model without significantly reducing the accuracy of the deep neural network, thereby accelerating model reasoning and reducing the storage occupancy of the model. The current mainstream deep model compression technology mainly includes model pruning, model quantization, low-rank decomposition, knowledge distillation, network structure search, and compact structure design. The closest public prior art to this application is structured model pruning technology. Structured model pruning technology refers to the use of the complete structure in the neural network model as the granularity (such as the filter convolution kernel in the convolutional neural network, the residual block in the residual network, and the branch in the multi-branch network), eliminating redundant network structures, and retaining the network structure that is effective for the output result. Among these structured model pruning methods, a common method is to add restrictions to the loss function in the model training stage, or to use the decision of model pruning as an optimization target, so as to learn the pruning decision of the model synchronously during the training process. For example, in a pruning method proposed by Lebedev et al., they added structured sparse terms to the loss function. In this way, when the model is trained, the sparse terms will be optimized at the same time. Finally, when the model training is completed, the filter weights of the corresponding sparse items below the threshold are set to 0, and the convolutional network model is pruned. Another structured model pruning method is to judge the significance of the model structure by the norm size. For example, for a convolutional network model, its convolution filter can be measured. norm, for a specified layer of convolution operation, a certain number of minimum The filter is filtered by the norm and the corresponding feature map is deleted at the same time. The model after filter pruning is retrained to restore or improve its recognition accuracy. There are also some structured model pruning methods that use the characteristics of the deep model calculation process to judge the significance of the filter. For example, the output of the deep model with ReLU function as the activation function has sparse characteristics. In a model pruning method proposed by Hu et al., they used the proportion of non-zero values ​​in the feature map output by the convolutional network model as the basis for judging the importance of the filter. In addition, the geometric median of the filter and the average rank of the feature map have also been proposed as criteria for judging the importance of the filter.

[0003] However, current structured pruning methods also face many challenges. There are three potential problems with structured pruning methods: (1) Inefficiency caused by offline pruning. Many structured model pruning techniques use offline pruning, which requires pruning after model training is completed, and the pruned model needs to be retrained after pruning is completed. For example, in the model pruning method proposed by Luo and He et al., pruning needs to be performed layer by layer, and the redundant filters of the current layer are identified based on the channels of the next layer. In the pruning method based on the filter norm, it is necessary to gradually remove filters with lower importance, and then fine-tune the pruned network layer by layer. Therefore, model pruning takes a long time and is inefficient.

[0004] (2) Low flexibility caused by hard filter pruning. The mask used in hard filter pruning is a global mask of 0-1 binary value, so it lacks flexibility. To solve the optimization problem of binary mask, it is necessary to use a greedy alternating update strategy, which is a relatively complicated process. Moreover, hard filter pruning is highly dependent on prior knowledge of filtering.

[0005] (3) Dependence on category labels. Most existing online pruning methods require supervised learning to perform model pruning. This means that model pruning methods need to be applied to datasets with category labels, and are difficult to apply when the data lacks category labels or needs to be migrated to other datasets with different category labels. Summary of the invention

[0006] The purpose of the present invention is to provide a high-precision large-scale structured network compression method based on minimum entropy to solve the problems of low efficiency, poor flexibility and single usage scenario of current structured pruning methods.

[0007] To achieve the above object, the present invention adopts the following technical solution: On the one hand, the present specification provides a high-precision large-scale structured network compression method based on minimum entropy, comprising: Step 102, building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; Step 104, constructing an optimization objective function based on the minimum entropy pruning framework; Step 106, iteratively optimizing the optimization objective function to obtain a soft mask of the pruned network; Step 108, perform model pruning based on the pruning selection of the soft mask and the model structure to obtain a pruned model.

[0008] On the other hand, the present specification provides a high-precision large-scale structured network compression device based on minimum entropy, comprising: A pruning framework building module is used to build a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; The optimization function building module is used to build the optimization objective function based on the minimum entropy pruning framework; The function iterative optimization module is used to iteratively optimize the optimization objective function to obtain the soft mask of the pruned network; The model pruning selection module is used to perform model pruning based on pruning selection of soft masks and model structures to obtain a pruned model.

[0009] Based on the above technical solution, this specification can achieve the following technical effects: (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network as close to the output feature distribution of the base network as possible, and to make the model pruning process label-independent. That is, in the framework of model pruning, a learnable discriminator is introduced to discriminate the output of the base network and the output of the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters of the pruned network (soft mask and weight) and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the difference between the output of the pruned network and the output of the base network; and when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discriminant difference between the output of the base network and the output of the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made as close to the output feature distribution of the base network as possible. The label-independent characteristics of the optimization method enable the pruning method to be used in unlabeled scenarios.

[0010] (2) A soft mask based on fractional normal constraints is used to implement soft filter pruning. Unlike existing hard filters, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the complexity of mask optimization. In order to obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative threshold shrinkage algorithm based on the fractional normal.

[0011] (3) Model pruning based on channel selection, block selection, and branch selection in an end-to-end optimization manner. Unlike the layer-by-layer multi-stage iterative pruning method, the model pruning method in this application adopts an end-to-end optimization method. By setting a loss function with the soft mask, weights, and discriminator weights of the pruned model as independent variables, and optimizing the loss function in a staged optimization manner, the soft mask of the entire pruned model is updated end-to-end, and the channel selection, block selection, and branch selection are determined based on the soft mask, which greatly improves the efficiency of model pruning. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The figure is a flowchart of a high-precision large-scale structured network compression method based on minimum entropy in one embodiment of the present invention.

[0013] Figure 2 Schematic diagram of a minimum entropy pruning framework in one embodiment of the present invention.

[0014] Figure 3 The figure is a schematic diagram of the structure of a high-precision large-scale structured network compression device based on minimum entropy in one embodiment of the present invention.

[0015] Figure 4 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0017] It should be noted that, in order to clearly explain the content of the present invention, the present invention specifically cites multiple embodiments to further illustrate different implementations of the present invention, wherein the multiple embodiments are enumerated rather than exhaustive. In addition, for the sake of brevity of explanation, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, the contents not mentioned in the subsequent embodiments can refer to the previous embodiments accordingly.

[0018] Example 1 Please refer to Figure 1 , Figure 1 The figure shows a high-precision large-scale structured network compression method based on minimum entropy provided by this embodiment. In this embodiment, the method includes: Step 102, building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; Step 104, constructing an optimization objective function based on the minimum entropy pruning framework; In this embodiment, the optimization objective function includes an adversarial objective function between the discriminator and the pruned network, an output loss function between the pruned network and the base network, and regular constraints on the pruned network parameters and the discriminator parameters.

[0019] In this embodiment, one implementation of step 104 is: Step 202, using a minimum entropy adversarial loss function to characterize the adversarial relationship between the discriminator and the pruned network, and obtaining an adversarial objective function between the discriminator and the pruned network; Step 204, using the MSE loss function to characterize the loss value of the output of the pruned network relative to the output of the base network, and obtaining the output loss function between the pruned network and the base network; Step 206, based on the weight parameters of the pruned network and the discriminator, obtain regularization constraints on the pruned network parameters and the discriminator parameters.

[0020] Step 106, iteratively optimizing the optimization objective function to obtain a soft mask of the pruned network; In this embodiment, one implementation of step 106 is: Step 302, based on the training sample data set, obtaining base network sample features and pruned network sample features; In this embodiment, one implementation of step 302 is: Step 3021, dividing the training sample data set into several batches to obtain several batches of sub-data sets; Step 3022, forward propagate each batch of sub-datasets through the base network and the pruned network respectively to obtain a number of base network sample features and a number of pruned network sample features.

[0021] Step 304, obtaining a first loss function and a second loss function based on the base network sample features, the pruned network sample features and the optimization objective function; In this embodiment, one implementation of step 304 is: Step 3041, obtaining a first loss function based on the sample features of the base network and the sample features of the pruned network, the adversarial objective function of the optimization objective function, and the regularization constraint term of the discriminator; Step 3042: Obtain a second loss function based on the base network sample features and the pruned network sample features, the output loss function of the optimization objective function, and the regularization constraint terms of the pruned network parameters.

[0022] Step 306, fixing the parameters of the pruned network, optimizing the parameters of the discriminator using the first loss function, and obtaining the weights of the updated discriminator; In this embodiment, one implementation of step 306 is: The parameters of the pruned network are fixed, and based on the first loss function, the weights of the discriminator are updated according to the gradient descent method to obtain the updated weights of the discriminator.

[0023] Step 308, fixing the weight of the discriminator, optimizing the parameters of the pruned network using the second loss function, and obtaining updated parameters of the pruned network; the parameters of the pruned network include the weight and soft mask of the pruned network.

[0024] In this embodiment, one implementation of step 308 is: Step 3081, fix the weight of the discriminator, and alternately update the weight and soft mask of the pruned network based on the second loss function to obtain the updated weight and soft mask of the pruned network.

[0025] In this embodiment, one implementation of step 3081 is: Step 402, fixing the soft mask of the pruned network, and updating the weights of the pruned network using the stochastic gradient descent method with momentum to obtain the updated weights of the pruned network; Step 404, fix the weight of the pruned network, and use the fast iterative soft threshold algorithm to update the soft mask of the pruned network to obtain an updated soft mask of the pruned network.

[0026] Step 108, perform model pruning based on the pruning selection of the soft mask and the model structure to obtain a pruned model.

[0027] In this embodiment, one implementation of step 108 is: Step 502: if the pruning selection of the model structure is channel selection, a scaling factor of a soft mask is added to each layer input channel of the model, and after the model is optimized, input channels with a scaling factor of 0 and corresponding output channels are removed from the model; Step 504: if the pruning selection of the model structure is block selection, a scaling factor of the soft mask is added to each residual block of the model, and residual blocks with a scaling factor of 0 are removed from the model after the model is optimized to obtain a pruned model; Step 506: If the pruning selection of the model structure is branch selection, a scaling factor of a soft mask is added to each branch of the model, and branches with a scaling factor of 0 are removed from the model after the model is optimized to obtain a pruned model.

[0028] Specifically, a high-precision large-scale structured network compression method based on minimum entropy includes the following specific steps: The first step is to build a model pruning framework Build a model pruning framework, connect the base network and the pruned network in parallel, and use the mean square loss function and a learnable discriminator to compare the outputs of the base network and the pruned network.

[0029] like Figure 2 , the original pre-trained base network With the pruned network In parallel, and Represent the input image, the weight of the base network and the weight of the pruned network respectively, Represents the soft mask of the fractional norm constraint of each network structure in the pruned network. Middle The elements are The output features of the base network and the pruned network are input into the MSE loss function module and the discriminator respectively. (The weight is ).

[0030] The second step is to define the optimization objective function Define the optimization objective function, which includes the adversarial objective function between the discriminator and the pruned network, the output loss function between the pruned network and the base network, and the regular constraints on the pruned network parameters and the discriminator parameters.

[0031] We transform the model pruning process into the following optimization problem

[0032] in It represents the adversarial loss based on minimum entropy, which is used for the confrontation between the pruned model and the discriminator, and is defined as

[0033] in Represents the distribution of input data, 0<= a<=1 is set according to actual conditions.

[0034] is the loss value between the output of the base network and the pruned network, which is used to align the output distribution of the two networks. We used the MSE loss to characterize the difference between the outputs of the two networks, and its formula is:

[0035] Indicates the effect on and The regularization constraint on can be decomposed into three parts:

[0036] in is the weight decay in the pruned network The regularization term can be defined as For soft mask The sparsification regularization constraint of It is a hyperparameter set by humans, which can be defined as is the regularization constraint of the discriminator weights to prevent the optimization of the discriminator from dominating the model pruning process during the optimization process, which can be defined as

[0037] The third step is iterative optimization to solve the parameters The optimal parameters of the pruned network and the discriminator are optimized in an iterative manner. The optimization process is divided into two stages: the adversarial optimization stage between the discriminator and the pruned network and the alignment optimization stage of the pruned network fitting the output distribution of the base network.

[0038] Given Training sample dataset , and the base network model parameters , set the sparsity factor of the fractional regularization , the learning rate of stochastic gradient descent and the maximum number of iterations First, initialize the weights of the pruned network and set them to be the same as the pre-trained base network weights, that is, , and then the scaling factor of the mask is randomly initialized according to the normal distribution .

[0039] The pruning process starts from the iteration round Start, iterate to Each iteration round can be divided into two stages.

[0040] Stage 1: Fixing the parameters of the pruned network , optimize the discriminator parameters to counter the pruned network. Specifically, the dataset Divide the data into several batches. For each batch of data, forward propagate it through the base network to obtain Sample characteristics , and the sample features obtained by the pruned network . With function

[0041] As the loss function, update the weights of the discriminator according to the gradient descent method . The loss function is derived from the adversarial objective function and the discriminator regularization constraint Items.

[0042] Stage 2: Freeze the Discriminator The weight of , update the pruned network parameters and , to align the output of the pruned network and the base network. Similar to stage 1, the data set is divided into several batches. For each batch of data, forward propagate it through the base network to obtain Sample characteristics , and the sample features obtained by the pruned network Since in this stage No update, so

[0043] If the regularization term is added, the overall loss function of this stage is

[0044] For the optimization of formula (8), we can also update and To achieve: 1. Fixation , update using stochastic gradient descent with momentum ; 2. Fixation , we use FISTA to optimize the update formula (8) Specifically, first initialize , and then update according to the following formula (9): :

[0045] in It is The learning rate at the iteration, and

[0046] Step 4: Model structure pruning selection The model pruning soft mask obtained by optimization is used to select the model structure for pruning. According to the model type and pruning granularity, it is divided into channel selection, branch selection and block selection.

[0047] After optimizing the parameters, we obtain the score regularization mask representing the model pruning decision ,Next, we use mask pruning to prune redundant model structures.,The model structure selection can be divided into three granularities: channel selection,,block selection, and branch selection.

[0048] Channel selection: Channel selection refers to using pruning masks to select valid output / input channels from the convolutional network and remove redundant channels. In our pruning method, soft masks are added to the input channels of each layer of the convolutional network. We assume that after optimization, the pruned network Layer The convolution kernel corresponds to the input The parameter matrix of the channels is , and the Layer The mask scaling factor for the input channels is Then the output of this layer is Feature Map Should be

[0049] Where * represents the convolution operation, Represents a nonlinear transformation such as ReLU. For the scaling factor The filtering, as well as its corresponding output feature channel and the input channel of the next layer, are regarded as channels of less importance and deleted from the model.

[0050] Block selection: For ResNet network pruning, there is a pruning granularity of block selection. By setting the residual map value of the residual block to 0 and retaining the input identity map, the residual block structure can be removed without cutting off the information flow of the network. The residual block calculation formula after adding soft mask is as follows:

[0051] in and Respectively represent The input and output of the residual block. represents the convolutional network branch in the residual block, For the All the convolution filter weights in the residual block, and It represents the skip connection in the residual block. After optimization, the scaling factor in the soft mask If we have residual blocks of , we can remove its convolutional network after training and keep only the skip connections without causing significant changes in the network output accuracy.

[0052] Branch selection: For multi-branch networks (such as GoogLeNet and ResNeXts), the model structure can be selected according to the granularity of the branch. Similar to the residual block mask of ResNet, each branch of the multi-branch network adds a scaling factor of the soft mask. After the model training and pruning are completed, we remove the branches with the corresponding scaling factor of 0. Taking GoogLeNet as an example, the calculation formula of the initial module (inception) with soft mask is:

[0053] in Represents a concatenation operation. Indicates the first The output of each branch has a weight of . is the total number of branches. If , then we think that The branches can be removed without causing a significant change in model accuracy.

[0054] The above is the whole process of structured network pruning based on minimum entropy.

[0055] In summary, this method can achieve the following technical effects: (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network as close to the output feature distribution of the base network as possible, and to make the model pruning process label-independent. That is, in the framework of model pruning, a learnable discriminator is introduced to discriminate the output of the base network and the output of the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters of the pruned network (soft mask and weight) and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the difference between the output of the pruned network and the output of the base network; and when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discriminant difference between the output of the base network and the output of the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made as close to the output feature distribution of the base network as possible. The label-independent characteristics of the optimization method enable the pruning method to be used in unlabeled scenarios.

[0056] (2) A soft mask based on fractional normal constraints is used to implement soft filter pruning. Unlike existing hard filters, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the complexity of mask optimization. In order to obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative threshold shrinkage algorithm based on the fractional normal.

[0057] (3) Model pruning based on channel selection, block selection, and branch selection in an end-to-end optimization manner. Unlike the layer-by-layer multi-stage iterative pruning method, the model pruning method in this application adopts an end-to-end optimization method. By setting a loss function with the soft mask, weights, and discriminator weights of the pruned model as independent variables, and optimizing the loss function in a staged optimization manner, the soft mask of the entire pruned model is updated end-to-end, and the channel selection, block selection, and branch selection are determined based on the soft mask, which greatly improves the efficiency of model pruning.

[0058] Example 2 Please refer to Figure 3 , Figure 3 The figure shows a high-precision large-scale structured network compression device based on minimum entropy provided by this embodiment. In this embodiment, the device includes: A pruning framework building module is used to build a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; The optimization function building module is used to build the optimization objective function based on the minimum entropy pruning framework; The function iterative optimization module is used to iteratively optimize the optimization objective function to obtain the soft mask of the pruned network; The model pruning selection module is used to perform model pruning based on pruning selection of soft masks and model structures to obtain a pruned model.

[0059] Optionally, the optimization objective function includes an adversarial objective function between the discriminator and the pruned network, an output loss function between the pruned network and the base network, and regular constraints on the pruned network parameters and the discriminator parameters.

[0060] Optionally, the optimization function building blocks include: An adversarial objective function construction unit is used to characterize the adversarial relationship between the discriminator and the pruned network using a minimum entropy adversarial loss function to obtain an adversarial objective function between the discriminator and the pruned network; An output loss function construction unit, used to use an MSE loss function to characterize the loss value of the output of the pruned network relative to the output of the base network, and obtain an output loss function between the pruned network and the base network; The regular constraint item determination unit is used to obtain the regular constraint items of the pruned network parameters and the discriminator parameters based on the weight parameters of the pruned network and the discriminator.

[0061] Optionally, the function iteration optimization module includes: A sample feature acquisition submodule is used to obtain base network sample features and pruned network sample features based on a training sample data set; A loss function determination submodule, used to obtain a first loss function and a second loss function based on the base network sample characteristics, the pruned network sample characteristics and the optimization objective function; The discriminator optimization submodule is used to fix the parameters of the pruned network, optimize the parameters of the discriminator using the first loss function, and obtain the updated weights of the discriminator; The pruned network optimization submodule is used to fix the weight of the discriminator, and optimize the parameters of the pruned network using the second loss function to obtain the updated parameters of the pruned network; the parameters of the pruned network include the weight and soft mask of the pruned network.

[0062] Optionally, the sample feature acquisition submodule includes: A sample data division unit is used to divide the training sample data set into several batches to obtain several batches of sub-data sets; The sample feature generation unit is used to forward propagate each batch of sub-data sets through the base network and the pruned network respectively to obtain a number of base network sample features and a number of pruned network sample features.

[0063] Optionally, the loss function determination submodule includes: A first loss function construction unit, used to obtain a first loss function based on the base network sample features and the pruned network sample features, the adversarial objective function of the optimization objective function, and the regularization constraint term of the discriminator; The second loss function construction unit is used to obtain the second loss function based on the sample characteristics of the base network and the sample characteristics of the pruned network, the output loss function of the optimization objective function, and the regularization constraint terms of the pruned network parameters.

[0064] Optionally, the discriminator optimization submodule is used to fix the parameters of the pruned network, and based on the first loss function, update the weight of the discriminator according to the gradient descent method to obtain the updated weight of the discriminator.

[0065] Optionally, the pruned network optimization submodule is used to fix the weights of the discriminator, and alternately update the weights and soft masks of the pruned network based on the second loss function to obtain updated weights and soft masks of the pruned network.

[0066] Optionally, the pruned network optimization submodules include: The weight optimization unit is used to fix the soft mask of the pruned network, and update the weight of the pruned network using the stochastic gradient descent method with momentum to obtain the updated weight of the pruned network; The soft mask optimization unit is used to fix the weight of the pruned network, and update the soft mask of the pruned network using a fast iterative soft threshold algorithm to obtain an updated soft mask of the pruned network.

[0067] Optionally, the model pruning selection module includes: A channel selection unit, for adding a soft-masked scaling factor to each layer input channel of the model if the pruning selection of the model structure is channel selection, and removing the input channels with a scaling factor of 0 and the corresponding output channels from the model after the model is optimized; A block selection unit, for adding a scaling factor of a soft mask to each residual block of the model if the pruning selection of the model structure is block selection, and removing the residual blocks with a scaling factor of 0 from the model after the model is optimized to obtain a pruned model; The branch selection unit is used to add a soft mask scaling factor to each branch of the model if the pruning selection of the model structure is branch selection, and remove the branches with a scaling factor of 0 from the model after the model is optimized to obtain a pruned model.

[0068] Based on this, the device can achieve the following technical effects: (1) A model pruning method based on minimum entropy loss is adopted to make the output feature distribution of the pruned network as close to the output feature distribution of the base network as possible, and to make the model pruning process label-independent. That is, in the framework of model pruning, a learnable discriminator is introduced to discriminate the output of the base network and the output of the pruned network. During the model pruning process, the optimization algorithm alternately optimizes the parameters of the pruned network (soft mask and weight) and the parameters of the discriminator. When optimizing the parameters of the pruned network, the optimization algorithm minimizes the difference between the output of the pruned network and the output of the base network; and when optimizing the parameters of the discriminator, the optimization algorithm maximizes the discriminant difference between the output of the base network and the output of the pruned network. Through the confrontation between the discriminator and the pruning parameters, the output feature distribution of the pruned network can be made as close to the output feature distribution of the base network as possible. The label-independent characteristics of the optimization method enable the pruning method to be used in unlabeled scenarios.

[0069] (2) A soft mask based on fractional normal constraints is used to implement soft filter pruning. Unlike existing hard filters, the model pruning mask in this application is a soft mask with continuous values, which can be optimized and updated using the gradient descent method, reducing the complexity of mask optimization. In order to obtain a scale factor that is strictly 0 during the optimization update process to determine which deep network structures should be pruned, we update the soft mask through a fast iterative threshold shrinkage algorithm based on the fractional normal.

[0070] (3) Model pruning based on channel selection, block selection, and branch selection in an end-to-end optimization manner. Unlike the layer-by-layer multi-stage iterative pruning method, the model pruning method in this application adopts an end-to-end optimization method. By setting a loss function with the soft mask, weights, and discriminator weights of the pruned model as independent variables, and optimizing the loss function in a staged optimization manner, the soft mask of the entire pruned model is updated end-to-end, and the channel selection, block selection, and branch selection are determined based on the soft mask, which greatly improves the efficiency of model pruning.

[0071] Example 3 Please refer to Figure 4 , this embodiment provides an electronic device, which includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a high-precision large-scale structured network compression method based on minimum entropy at the logical level. Of course, in addition to software implementation methods, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0072] The network interface, processor and memory can be connected to each other through a bus system. The above bus can be divided into an address bus, a data bus, a control bus, etc.

[0073] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor.

[0074] The processor is used to execute the program stored in the above memory, and specifically perform: Step 102, building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; Step 104, constructing an optimization objective function based on the minimum entropy pruning framework; Step 106, iteratively optimizing the optimization objective function to obtain a soft mask of the pruned network; Step 108, perform model pruning based on the pruning selection of the soft mask and the model structure to obtain a pruned model.

[0075] The processor may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of the processor hardware or by instructions in the form of software.

[0076] Based on the same invention, the embodiment of this specification also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1-Figure 2 The corresponding embodiment provides a high-precision large-scale structured network compression method based on minimum entropy.

[0077] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code.

[0078] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.

[0079] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between it and other embodiments.

[0080] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the specific order or sequential order shown in the process depicted in the drawings is not necessarily required to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0081] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A high-precision large-scale structured network compression method based on minimum entropy, characterized in that: include: Building a minimum entropy pruning framework; the base network of the minimum entropy pruning framework is connected in parallel with the pruned network, and the outputs of the base network and the pruned network are respectively input into the MSE loss function module and the discriminator; Based on the minimum entropy pruning framework, the optimization objective function is constructed; Iteratively optimize the optimization objective function to obtain the soft mask of the pruned network; The model is pruned based on the pruning selection of soft mask and model structure to obtain the pruned model.

2. The method according to claim 1, characterized in that The optimization objective function includes an adversarial objective function between the discriminator and the pruned network, an output loss function between the pruned network and the base network, and regular constraints on the pruned network parameters and the discriminator parameters.

3. The method according to claim 2, characterized in that The optimization objective function constructed based on the minimum entropy pruning framework includes: The minimum entropy adversarial loss function is used to characterize the adversarial relationship between the discriminator and the pruned network, and the adversarial objective function between the discriminator and the pruned network is obtained. The MSE loss function is used to characterize the loss value of the output of the pruned network relative to the output of the base network, and the output loss function between the pruned network and the base network is obtained; Based on the weight parameters of the pruned network and the discriminator, regular constraints on the pruned network parameters and the discriminator parameters are obtained.

4. The method according to claim 3, characterized in that The iterative optimization of the optimization objective function to obtain the soft mask of the pruned network includes: Based on the training sample data set, the base network sample features and the pruned network sample features are obtained; Based on the base network sample features, the pruned network sample features and the optimization objective function, a first loss function and a second loss function are obtained; The parameters of the pruned network are fixed, and the parameters of the discriminator are optimized using the first loss function to obtain the updated weights of the discriminator; The weight of the discriminator is fixed, and the parameters of the pruned network are optimized using the second loss function to obtain updated parameters of the pruned network; the parameters of the pruned network include the weight and soft mask of the pruned network.

5. The method according to claim 4, characterized in that The obtaining of base network sample features and pruned network sample features based on the training sample data set includes: Divide the training sample data set into several batches to obtain several batches of sub-data sets; Each batch of sub-datasets is forward propagated through the base network and the pruned network respectively to obtain several base network sample features and several pruned network sample features.

6. The method according to claim 5, characterized in that The obtaining of the first loss function and the second loss function based on the base network sample features, the pruned network sample features and the optimization objective function comprises: Based on the sample features of the base network and the sample features of the pruned network, as well as the adversarial objective function of the optimization objective function and the regularization constraint term of the discriminator, a first loss function is obtained; based on the sample features of the base network and the sample features of the pruned network, as well as the output loss function of the optimization objective function and the regularization constraint term of the pruned network parameters, a second loss function is obtained.

7. The method according to claim 6, characterized in that The parameters of the pruned network are fixed, and the parameters of the discriminator are optimized using the first loss function to obtain the updated weights of the discriminator. The parameters of the pruned network are fixed, and based on the first loss function, the weights of the discriminator are updated according to the gradient descent method to obtain the updated weights of the discriminator.

8. The method according to claim 7, characterized in that The weight of the fixed discriminator is optimized by using the second loss function to obtain the updated parameters of the pruned network, and the parameters of the pruned network are obtained. The weights of the discriminator are fixed, and the weights and soft masks of the pruned network are alternately updated based on the second loss function to obtain the updated weights and soft masks of the pruned network.

9. The method according to claim 8, characterized in that The weight of the fixed discriminator is alternately updated based on the second loss function on the weight and soft mask of the pruned network to obtain the updated weight and soft mask of the pruned network, including: Fix the soft mask of the pruned network, use the stochastic gradient descent method with momentum to update the weight of the pruned network, and obtain the updated weight of the pruned network; The weights of the pruned network are fixed, and the soft mask of the pruned network is updated using a fast iterative soft threshold algorithm to obtain an updated soft mask of the pruned network.

10. The method according to claim 1, characterized in that The pruning selection based on the soft mask and the model structure performs model pruning, and the pruned model obtained includes: If the pruning selection of the model structure is channel selection, a soft mask scaling factor is added to the input channel of each layer of the model, and the input channels with a scaling factor of 0 and the corresponding output channels are removed from the model after the model is optimized; If the pruning selection of the model structure is block selection, a scaling factor of the soft mask is added to each residual block of the model, and after the model is optimized, the residual blocks with a scaling factor of 0 are removed from the model to obtain a pruned model; If the pruning selection of the model structure is branch selection, a soft mask scaling factor is added to each branch of the model, and after the model is optimized, branches with a scaling factor of 0 are removed from the model to obtain a pruned model.

Images