An image representation model pruning method based on multi-granularity importance measurement

Abstract

The application relates to a multi-granularity importance metric-based image representation model pruning method, and belongs to the technical field of model compression. The method combines information loss and redundancy criteria to construct a multi-granularity importance metric-based model channel pruning method, compensates for the single importance metric and unstable performance of an existing algorithm, and improves the fine-tuning performance of a pruned model by keeping the balance and diversity of pruned channel distribution, thereby expanding the model pruning idea based on importance metric.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model compression, and particularly relates to an image representation model pruning method based on multi-granularity importance measurement. BACKGROUND

[0002] Image representation models, which are represented by object classification and target detection in images, have always been a hot spot in the industry, and their applications involve traffic security, shopping, food and travel, etc. The superior performance of such models often depends on a large parameter size and a large amount of computation, and their lightweight and acceleration are the key to landing applications. At present, various model optimization and acceleration methods for deep neural networks have been proposed, mainly including lightweight network structure design, distillation, quantization, pruning, low-rank decomposition, etc. Among them, lightweight network structure design and distillation methods mostly rely on artificial prior knowledge for model design, and it is difficult to respond to the rapid update needs of industrial scenarios. Low-rank decomposition methods can only compress layer by layer, and it is difficult to pay attention to global information, and the decomposition calculation consumes a lot. Pruning and quantization methods have good performance, strong generalization, and are widely used in industrial applications. Compared with quantization, the parameter size adjustment of pruning methods is more flexible, and it can also be used as a pre-step of quantization, so related research has always been committed to designing more effective pruning methods.

[0003] General model pruning methods can be roughly divided into two categories: non-structural pruning methods and structural pruning methods. Non-structural pruning methods only focus on the retention and deletion of a single weight or connection, and the removed weights do not have regularity in the distribution position, and the obtained sparse matrix cannot be used by most frameworks and hardware acceleration, so there are limitations in practical applications. In addition, the actual acceleration effect is limited. Structural pruning methods take larger granularity structures such as filters and convolution output channels as the smallest unit of pruning. This method effectively utilizes the regularity of the removed parameter position, and can directly delete the operation, thereby achieving a large amount of compression and speedup. In summary, the present application is based on the structural pruning method to improve the practical operability.

[0004] At present, most structural pruning techniques are based on importance criteria for model pruning, that is, different criteria are designed to measure the importance of units, and according to the assumption that the lower the importance of a unit, the lower the contribution of the model, the interval is set to delete low-importance network structure units. Although these importance criterion-based methods have achieved relatively significant compression effect, they are only based on a single dimension to design pruning criteria, such as task loss, redundancy or feature alignment degree. These single criteria have their own defects in different scenarios. For example, the feature alignment or redundancy criterion does not fully consider the impact on the overall task, and relies on manually designed compression rate for each layer. At the same time, the loss-based model does not meet the convergence assumption after pruning, and the loss cannot effectively represent the performance of the optimized model, limiting the performance recovery ability of the model in the pruning optimization stage, ultimately leading to problems such as large loss of model accuracy after compression or unstable model performance under different compression rates in the actual pruning process.

[0005] Therefore, there is still a lot of room for performance improvement in existing importance criterion-based model pruning methods. SUMMARY

[0006] In view of the above problems, the present application provides a multi-granularity importance measurement-based image representation model pruning method. The pruning method combines information loss and redundancy criteria to overcome the defects of single importance measurement and unstable method performance of existing algorithms, and expands the model pruning ideas based on importance measurement.

[0007] The present application provides a multi-granularity importance measurement-based image representation model pruning method, comprising the following specific steps:

[0008] S1, collect images to establish an image training data set;

[0009] Preferably, the specific steps for establishing an image training data set in step S1 are as follows:

[0010] Prepare images to build an image data set;

[0011] Preprocess the image data set to obtain a preprocessed image data set;

[0012] Divide the preprocessed image data set to obtain an image training data set.

[0013] Further, the image data set contains real images and corresponding object annotation information in the images; the annotation information includes object classification and target objects in the images;

[0014] The image data preprocessing includes image enhancement operation and image normalization operation;

[0015] The image enhancement operation includes adjusting image size, randomly flipping the image, and edge padding 0 values (black edges).

[0016] The image dataset division uses an officially disclosed divided dataset; for example, using the target detection dataset COCO, the application directly uses the official training set containing 118287 images and the verification set including 5000 images.

[0017] S2, obtaining a neural network model as a pre-training model, and copying a model copy as a to-be-pruned model; setting a training constraint in the to-be-pruned model;

[0018] Preferably, the neural network model in step S2 is a convolutional neural network (CNN) image processing model, including a residual network (ResNet) used for an image classification task and a RetinaNet used for an image detection task; as shown in Figure 2

[0019] The to-be-pruned model includes three parts:

[0020] A backbone network extracts texture and structural features of an image from input data;

[0021] A transition network combines and mixes the features extracted by the backbone network;

[0022] A network head deforms the extracted features according to a given task, so that the output meets the actual task requirements.

[0023] Preferably, the training constraint of the to-be-pruned model is set by using a joint loss; the joint loss includes a task loss of the to-be-pruned model and an information loss of an importance measurement module.

[0024] S3, defining an importance measurement module for calculating the importance of each channel of a network layer of the to-be-pruned model based on the training constraint;

[0025] Preferably, the importance measurement module includes a global importance measurement module based on the joint loss and an intra-layer importance measurement module based on redundancy.

[0026] The global importance measurement module based on the joint loss includes two units: an importance measurement unit using an original task loss and an importance measurement unit using an information loss.

[0027] Preferably, the importance measurement module comprehensively considers the importance of a channel from multiple angles, such as a network layer, a single channel, and part of channels of a network layer.

[0028] ​Preferably, the importance of each channel of the to-be-pruned model network layer comprises: a task loss importance of the network layer channel, an information loss importance of the network layer channel, and a group importance of the network layer channel;

[0029] The expression of the importance of the channel i of the to-be-pruned model network layer l is:

[0030]

[0031] Wherein, is the importance of the channel i of the to-be-pruned model network layer l; is the task loss importance of the channel i of the to-be-pruned model network layer l; is the information loss importance of the channel i of the to-be-pruned model network layer l; is the group importance of the channel i of the network layer l.

[0032] Further, the importance measurement unit of the original task loss records the input of the network layer in the network forward propagation process, and records the gradient of the network layer input with respect to the task loss in the network back propagation process;

[0033] The task loss importance of the channel i of the to-be-pruned model network layer l is obtained by presetting the task loss constraint of each channel of the network layer to the task of the to-be-pruned model The expression is:

[0034]

[0035] Wherein, L task is the preset task loss of the to-be-pruned model; m is the mask corresponding to the input of the to-be-pruned model network layer (0 in the mask represents that the channel is deleted), e i is a full 0 vector with only channel i being 1, is the transpose of the full 0 vector with only channel i being 1; m i is the corresponding mask of the deleted channel i; g is the Jacobian matrix of the task loss, g i is the Jacobian matrix of the task loss with respect to the channel i; G is the second-order Hessian matrix of the task loss, H ii is the Hessian matrix of the task loss with respect to the channel i, i=1,2,3…I, I represents the total number of channels of the to-be-pruned model network layer l, l=1,2,3…L, L represents the total number of network layers of the to-be-pruned model.

[0036] Further, considering that the current to-be-pruned model is continuously trained, it can be assumed that the current to-be-pruned model has converged, that is, the first-order gradient g i of any channel of the model with respect to the task loss is 0, and the preliminary task loss importance of the channel i of the to-be-pruned model network layer l Can be converted to:

[0037]

[0038] Wherein, is the expected value of the distribution, x is the input of the model to be pruned, y is the label of the model to be pruned; n=1,2,3…N, N is the total number of samples at each time of training; A is the input of the network layer of the model to be pruned; ⊙ is the dot product operation; is the gradient of the task loss of each layer of the network of the model to be pruned.

[0039] The task loss original importance of the network layer l channel i of the preliminary model to be pruned is calculated as Regularization, obtain the task loss importance of the network layer l channel i of the model to be pruned after regularization The expression is:

[0040]

[0041] Wherein, H is the feature map height of the network layer l channel i of the model to be pruned, W is the feature map width of the network layer l channel i of the model to be pruned.

[0042] Based on the task loss importance of the network layer l channel i of the model to be pruned after regularization The task loss importance of all coupled layers of the model to be pruned is summed up, and the task loss importance of the network layer l channel i of the model to be pruned is obtained The expression is:

[0043]

[0044] Wherein, b∈B,B is the coupling layer index set coupled with channel i, is the task loss importance of the channel i of the coupling layer b of the model to be pruned.

[0045] It can be understood that when the channels of the network layer of the model to be pruned exist coupling relationship, that is, for each channel, the network layer associated with it can be obtained, called coupling layer; for example, the skip connection layer in the residual network (ResNet), in which the channels need to be deleted at the same time to keep the input tensor matching; specifically, the calculation graph established by the unit when using forward propagation determines the source and destination of each network layer, and the network layers with the same source are coupled layers, which change the channels by sharing the mask.

[0046] Further, the importance measurement unit using information loss creates a projection layer at the output of a specific layer of the model to be pruned, and the projection layer is based on the information loss L info Constrain the network layer, obtain the information loss importance of the network layer l channel i of the model to be pruned The specific layer is a selected network layer for calculating information loss, for example, all network layers included in a transition network neck;

[0047] The information loss L info is expressed as:

[0048]

[0049] wherein, is an output feature of a pre-trained model network layer s (s∈S), S is a selected network layer index set to which information loss is applied, is an output feature of a current pruned model network layer s (s∈S); is a mean value of cosine similarity calculated according to spatial positions; f is a projection layer.

[0050] The information loss importance of the network layer l of the to-be-pruned model channel i is expressed as:

[0051]

[0052] wherein, b∈B, B is a coupling layer index set coupled with the channel i, is a gradient of each network layer input of the to-be-pruned model with respect to the task loss; H is a feature map height of the network layer l of the to-be-pruned model channel i, W is a feature map width of the network layer l of the to-be-pruned model channel i; n=1, 2, 3, …, N, N is a total number of samples at each time of training;

[0053] Further, the intra-layer importance measurement module based on redundancy divides the group importance of the network layer channel of the to-be-pruned model according to the network layer channel clustering situation, and the specific steps are as follows:

[0054] The weights of the network layer l channel i of the to-be-pruned model are collected to obtain the aggregated weights of the network layer l channel i;

[0055] The hierarchical clustering algorithm is used to cluster and group the aggregated weights of the network layer l channel i to obtain the channel grouping situation of the network layer l; the channel grouping situation of the network layer l includes the grouping where the network layer l channel i is located and the number of channels in the group;

[0056] Based on the channel grouping situation of the network layer l, the preliminary group importance of the network layer l channel i is obtained The expression is:

[0057]

[0058] wherein, m c,l is a mask value of the network layer l channel c of the to-be-pruned model, c=1, 2, 3, …, C, C is the total number of channels of the network layer l of the to-be-pruned model; mk,l is the mask value of the channel k of the network layer l to be pruned, k ∈ G j,l j,l is the channel index set of the jth group of the network layer l to be pruned.

[0059] scaling the preliminary group importance of the channel i of the network layer l to obtain the updated group importance of the channel i of the network layer l, the expression is:

[0060]

[0061]

[0062] wherein, is the updated group importance of the channel i of the network layer l; is the importance set of all channels of the network layer l; is the minimum value of the preliminary group importance of all channels of the network layer l, is the maximum value of the preliminary group importance of all channels of the network layer l; is the group importance of the channel i of the network layer l; exp(·) represents the natural exponential function, λ scale is the scaling parameter, the group importance value of each channel ranges from 0 to 1.

[0063] Further, the weight of each channel of the network layer to be pruned includes the convolution layer filter and the fully connected layer parameter of the source and destination;

[0064] The aggregated weight of the ath channel of the convolution layer m is obtained by the convolution layer filter of the source and destination, and the expression is:

[0065]

[0066] f(W :,:,: ) = concat(||W 0,:,: ||2, ||W 1,:,: ||2…||W C,:,: ||2);

[0067] wherein, concat(·) represents the channel concatenation operation, ||·||2 represents the 2-norm; f(W :,:,: ) represents the weight value splicing into a vector, is the ath filter of the network layer m.

[0068] Further, the fully connected layer operation step is completely consistent with the convolution layer operation step.

[0069] ​​Further, the grouped weights are grouped using a hierarchical clustering algorithm, which is a Euclidean Distance, and the Euclidean Distance is used to measure the distance between different samples.

[0070] S4, initializing an importance measurement module in the to-be-pruned model based on the training constraints to obtain an initialized to-be-pruned model;

[0071] Preferably, the specific steps of obtaining the initialized to-be-pruned model are as follows:

[0072] The remaining parameters are all parameters except the projection layer in the global importance measurement module of the to-be-pruned model;

[0073] The remaining parameters are all parameters except the projection layer in the global importance measurement module of the to-be-pruned model;

[0074] The projection layer in the information loss-based global importance measurement module is updated using information loss optimization until the information loss of the importance measurement module tends to converge, and an initialized information loss-based importance measurement module is obtained.

[0075] The intra-layer importance measurement module based on redundancy is used to traverse all network layers of the to-be-pruned model, and the group importance of all network layers is initialized using a hierarchical clustering algorithm.

[0076] S5, inputting the image training data set into the initialized to-be-pruned model, pruning the to-be-pruned model based on the importance of each channel of the model network layer and the training constraints, and obtaining a pruned model.

[0077] Preferably, the specific steps of obtaining the pruned model in step S5 are as follows:

[0078] The network layer of the to-be-pruned model receives the image training data set for data forward propagation and backward propagation, and obtains the importance of each channel of the network layer of the to-be-pruned model.

[0079] According to the importance of each channel of the network layer of the to-be-pruned model, the channels of the network layer are pruned.

[0080] After each backward propagation, the gradient of each network layer input of the to-be-pruned model with respect to the task loss is obtained, and the network layer is updated iteratively using the gradient. The iterative update is performed for the dth time, d is a preset number of times, and a pruning operation is performed once. After the (d-1)th iterative update, the importance of each channel of the network layer of the to-be-pruned model for all previous iterations before the dth iteration is summed by channel. After the dth iterative update, all unpruned channels of the network are sorted in ascending order according to the importance of each channel of the network layer of the to-be-pruned model, and the channel with the smallest importance and the network layer that has not been completely deleted is selected for pruning to obtain a pruned model.

[0081] The preset number of times is 25.

[0082] S6, fine-tuning the pruned model to obtain a compressed model;

[0083] Preferably, the specific steps of fine-tuning the pruned model to obtain a compressed model in step S6 are as follows:

[0084] The pruned model deletes the convolutional layer filter and the fully connected parameter corresponding to each channel according to the mask corresponding to each channel of the network layer during pruning, so as to realize the reduction of the pruned model.

[0085] The importance measurement module in the pruned model is removed to obtain a compressed model.

[0086] The compressed model is updated using the parameter configuration and loss consistent with the pre-trained model until the parameters converge, so as to obtain a compressed model.

[0087] S7, inputting the image to be detected into the compressed model to output the object category and specific position in the detected image.

[0088] Compared with the prior art, the present application has at least the following beneficial effects:

[0089] (1) The present application proposes an iterative pruning method combining information loss and redundancy criteria, which makes up for the defects of single importance measurement and unstable method performance of existing algorithms, and expands the model pruning ideas based on importance measurement.

[0090] (2) The present application proposes a global importance measurement module based on information loss supervision. The pre-trained model is used to guide the pruning of the model to be pruned, so as to reduce the intermediate layer expression information loss in the compression process.

[0091] (3) The present application proposes an intra-layer importance measurement module based on redundancy, which clusters and groups the internal channels of the network layer, maintains a certain proportion of channels in each channel group to keep the balance and diversity of the pruning channel distribution, and improves the fine-tuning performance of the pruned model. BRIEF DESCRIPTION OF DRAWINGS

[0092] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not considered as limiting the present application.

[0093] Figure 1 It is a flowchart of the iterative pruning method based on multi-granularity importance measurement of the present application.

[0094] Figure 2 It is a schematic diagram of the pruned model architecture of the present application. DETAILED DESCRIPTION

[0095] In order to enable the above-mentioned objects, features and advantages of the present application to be clearer, the present application will be further described below in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0096] One specific embodiment of the present application, as Figures 1-2 , discloses a model pruning method based on multi-granularity importance measurement. In order to illustrate the effectiveness of the method of the present application, the above technical solutions of the present application will be described in detail below through a specific embodiment, and the specific implementation steps are as follows:

[0097] S1, collect images to establish an image training data set;

[0098] Preferably, the specific steps of establishing an image training data set in step S1 are as follows:

[0099] Prepare images to build an image data set; select COCO2012 and Pascal VOC public data sets for target detection tasks; the image data set and the annotation organization method are consistent with the open source target detection framework MMDetection;

[0100] Preprocess the image data set to obtain a preprocessed image data set; the image enhancement operation includes adjusting the size, randomly flipping, and filling black pixels; on the COCO data set, the present example adjusts the image to 1333x800 pixel size in proportion, and on the Pascal VOC data set, the present example adjusts the image to 1000x600 pixel size in proportion. Subsequently, on all data sets, the present example randomly flips the image horizontally and vertically with a probability of 50%, and fills the image to 32 integer pixel, and finally performs image normalization operation;

[0101] Divide the preprocessed image data set to obtain an image training data set; the data set division is based on the standard given by each data set. In accordance with the general practice, for the COCO data set, 118287 images are used for training, and 5000 images are used for verification and testing. For the Pascal VOC data set, 16551 training set and validation set images of VOC2007 and VOC2012 are used for training, and 4952 images of the VOC2007 test set are used for verification and testing;

[0102] Further, the image data set contains real images and corresponding object annotation information in the images; the annotation information includes object classification and target object in the image;

[0103] The image data preprocessing includes an image enhancement operation and an image normalization operation.

[0104] The image enhancement operation includes resizing, random flipping and padding, the image dataset division uses an officially disclosed divided dataset, and the target detection dataset COCO is taken as an example, the official training set containing 118287 pictures and the verification set containing 5000 pictures are directly used.

[0105] S2, a neural network model is obtained as a pre-training model, and a model copy is copied as a to-be-pruned model, and a training constraint is set in the to-be-pruned model;

[0106] Preferably, the neural network model in step S2 is a convolutional neural network (CNN) image processing model, including a residual network (ResNet) used for an image classification task and a RetinaNet used for an image detection task. Figure 2 As shown in the figure.

[0107] The to-be-pruned model includes three parts:

[0108] A backbone network extracts texture and structural features of an image from input data;

[0109] A transition network combines and mixes the features extracted by the backbone network;

[0110] A network head deforms the extracted features according to a given task, so that the output meets the actual task requirements.

[0111] Preferably, the training constraint of the to-be-pruned model is set by using a joint loss, and the joint loss contains a task loss of the to-be-pruned model and an information loss of an importance measurement module.

[0112] S3, based on the training constraint, an importance measurement module for calculating the importance of each channel of a network layer of the to-be-pruned model is defined.

[0113] Preferably, the importance measurement module includes a global importance measurement module based on a joint loss and an intra-layer importance measurement module based on redundancy.

[0114] The global importance measurement module based on the joint loss includes two units: an importance measurement unit using an original task loss and an importance measurement unit using an information loss.

[0115] Preferably, the importance measurement module comprehensively considers the importance of the channel from multiple angles such as a network layer, a single channel and a part of channels of a network layer.

[0116] Preferably, the importance of each channel of the to-be-pruned model network layer comprises: a task loss importance of the network layer channel, an information loss importance of the network layer channel, and a group importance of the network layer channel;

[0117] The expression of the importance of the channel i of the to-be-pruned model network layer l is:

[0118]

[0119] Wherein, is the importance of the channel i of the to-be-pruned model network layer l; is the task loss importance of the channel i of the to-be-pruned model network layer l; is the information loss importance of the channel i of the to-be-pruned model network layer l; is the group importance of the channel i of the network layer l.

[0120] Further, the importance measurement unit of the original task loss records the input of the network layer in the network forward propagation process, and records the gradient of the network layer input with respect to the task loss in the network back propagation process;

[0121] The task loss importance of the channel i of the to-be-pruned model network layer l is obtained by presetting the task loss constraint of each channel of the network layer to the task of the to-be-pruned model The expression is:

[0122]

[0123] Wherein, L task is the preset task loss of the to-be-pruned model; m is the mask corresponding to the input of the to-be-pruned model network layer (0 in the mask represents that the channel is deleted), e i is a full 0 vector with only channel i being 1, is the transpose of the full 0 vector with only channel i being 1; m i is the corresponding mask of the deleted channel i; g is the Jacobian matrix of the task loss, g i is the Jacobian matrix of the task loss with respect to the channel i; H is the second-order Hessian matrix of the task loss, H ii is the Hessian matrix of the task loss with respect to the channel i, i=1,2,3…I, I represents the total number of channels of the to-be-pruned model network layer l, l=1,2,3…L, L represents the total number of network layers of the to-be-pruned model.

[0124] Further, considering that the current generation pruned model is continuously trained, it can be assumed that the current to-be-pruned model has converged, i.e. the first-order gradient g i of the task loss of any channel of the model is 0, and the preliminary task loss importance of the channel i of the to-be-pruned model network layer l Can be converted to:

[0125]

[0126] Wherein, is the expected value of the distribution, x is the input of the model to be pruned, y is the label of the model to be pruned; n=1,2,3…N, N is the total number of samples at each time of training; A is the input of the network layer of the model to be pruned; ⊙ is the dot product operation; is the gradient of the task loss of each layer of the network of the model to be pruned.

[0127] The task loss original importance of the network layer l channel i of the preliminary model to be pruned is obtained Regularization, obtain the task loss importance of the network layer l channel i of the model to be pruned after regularization The expression is:

[0128]

[0129] Wherein, H is the feature map height of the network layer l channel i of the model to be pruned, W is the feature map width of the network layer l channel i of the model to be pruned.

[0130] Based on the task loss importance of the network layer l channel i of the model to be pruned after regularization The task loss importance of all coupled layers of the model to be pruned is summed up, and the task loss importance of the network layer l channel i of the model to be pruned is obtained The expression is:

[0131]

[0132] Wherein, b∈B,B is the coupling layer index set coupled with channel i, is the task loss importance of the channel i of the coupling layer b of the model to be pruned.

[0133] It can be understood that when the channels of the network layer of the model to be pruned exist coupling relationship, that is, for each channel, the network layer associated with it can be obtained, called coupling layer; for example, the skip connection layer in residual network (ResNet), in which the channels need to be deleted at the same time to keep the input tensor matching; specifically, the calculation graph established by the unit when using forward propagation determines the source and destination of each network layer, and the network layers with the same source are coupled layers, which change the channels by sharing the mask.

[0134] Further, the importance measurement unit using information loss creates a projection layer at the output of a specific layer of the model to be pruned, and the projection layer is based on the information loss L info Constrain the network layer, obtain the information loss importance of the network layer l channel i of the model to be pruned The specific layer is a selected network layer for calculating information loss, for example, all layers included in a transition network neck;

[0135] The information loss L info is expressed as:

[0136]

[0137] wherein, is an output feature of a pre-trained model network layer s (s∈S), S is a selected network layer index set to which information loss is applied, is an output feature of a current pruned model network layer s (s∈S); is a mean value of cosine similarity calculated according to spatial positions; f is a projection layer.

[0138] The information loss importance of the network layer l of the to-be-pruned model channel i is expressed as:

[0139]

[0140] wherein, b∈B, B is a coupling layer index set coupled with the channel i, is a gradient of each network layer input of the to-be-pruned model with respect to the task loss; H is a feature map height of the network layer l of the to-be-pruned model channel i, W is a feature map width of the network layer l of the to-be-pruned model channel i; n=1, 2, 3, …, N, N is a total number of samples at each time of training;

[0141] Further, the intra-layer importance measurement module based on redundancy divides the group importance of the network layer channel of the to-be-pruned model according to the network layer channel clustering situation, and the specific steps are as follows:

[0142] The weights of the network layer l channel i of the to-be-pruned model are collected to obtain the aggregated weights of the network layer l channel i;

[0143] The hierarchical clustering algorithm is used to cluster and group the aggregated weights of the network layer l channel i to obtain the channel grouping situation of the network layer l. The channel grouping situation of the network layer l includes the grouping where the network layer l channel i is located and the number of channels in the group.

[0144] Based on the channel grouping situation of the network layer l, the preliminary group importance of the network layer l channel i is obtained The expression is:

[0145]

[0146] wherein, m c,l is a mask value of the network layer l channel c of the to-be-pruned model, c=1, 2, 3, …, C, C is the total number of channels of the network layer l of the to-be-pruned model; mk,l is the mask value of the channel k of the network layer l to be pruned, k G j,l j,l is the channel index set of the jth group of the network layer l to be pruned.

[0147] scaling the preliminary group importance of the channel i of the network layer l to obtain the updated group importance of the channel i of the network layer l, the expression is as follows:

[0148]

[0149]

[0150] wherein, is the updated group importance of the channel i of the network layer l; is the importance set of all channels of the network layer l; is the minimum value of the initial group importance of all channels of the network layer l, is the maximum value of the initial group importance of all channels of the network layer l; is the group importance of the channel i of the network layer l; exp(·) represents the natural exponential function, and λ scale is the scaling parameter, and the group importance value of each channel ranges from

[0151] Further, the weight of each channel of the network layer to be pruned includes the convolution layer filter and the fully connected layer parameter of the source and destination;

[0152] The aggregated weight of the ath channel of the convolution layer m is obtained through the convolution layer filter of the source and destination, and the expression is as follows:

[0153]

[0154] f(W :,:,: ) = concat (||W 0,:,: ||2, ||W 1,:,: ||2…||W C,:,: ||2) ;

[0155] wherein, concat(·) represents the channel concatenation operation, and ||·||2 represents the 2-norm; f(W :,:,: ) represents that the weight values are concatenated into a vector, is the ath filter of the network layer m.

[0156] Further, the fully connected layer operation step is completely consistent with the convolution layer operation step.

[0157] ​Further, the grouped weights are grouped using a hierarchical clustering algorithm, which is Euclidean Distance, which measures the distance between different samples.

[0158] S4, initializing an importance measurement module in the to-be-pruned model based on a training constraint to obtain an initialized to-be-pruned model;

[0159] In the importance measurement module based on information loss, the projection layer is applied to all layers included in the transition network, and is specifically set as a convolution with equal input and output channels, and the step and padding values are set as 1 and 0 respectively. The module only accounts for a very small part (2%) of the entire model parameter amount, and is removed before the compressed model is actually used. In the intra-layer importance measurement module based on redundancy, the maximum cluster number G max = 0.9 * C l is set, and only obviously redundant channels are grouped into the same group, and the group importance scaling parameter λ scale is set as 2.

[0160] Preferably, the specific steps of obtaining the initialized to-be-pruned model are as follows:

[0161] The remaining parameters of the to-be-pruned model are frozen, and the normalization layer of the model is set to a verification mode to prevent the historical statistical variables from being updated.

[0162] The remaining parameters are all parameters except the projection layer in the global importance measurement module of the to-be-pruned model.

[0163] The projection layer in the global importance measurement module based on information loss is updated using information loss optimization until the information loss of the importance measurement module tends to converge, and the initialized importance measurement module based on information loss is obtained.

[0164] The intra-layer importance measurement module based on redundancy is used to traverse all network layers of the to-be-pruned model, and the group importance of all network layers is initialized using a hierarchical clustering algorithm.

[0165] S5, inputting an image training data set into the initialized to-be-pruned model, pruning the to-be-pruned model based on the importance of each channel of the network layer of the model and a training constraint to obtain a pruned model.

[0166] Preferably, the specific steps of obtaining the pruned model in step S5 are as follows:

[0167] The network layer of the to-be-pruned model receives the image training data set for data forward propagation and backward propagation.

[0168] According to the importance of each channel of the network layer of the to-be-pruned model, the channels of the network layer are pruned.

[0169] After each back propagation, the gradient of each layer of the network layer input of the to-be-pruned model with respect to the task loss is obtained, and the network layer is updated iteratively using the gradient, and the iterative updating is performed for the dth time, d is a preset to-be-number, and one pruning operation is performed, wherein, after the (d-1)th iterative updating, the importance of each channel of the network layer of the to-be-pruned model of all iterations before the dth iteration is summed up by channel, and after the dth iterative updating, all unpruned channels of the network are sorted in ascending order according to the importance of each channel of the network layer of the to-be-pruned model, the channel with the smallest importance and the network layer is selected for pruning, and a pruned model is obtained; the preset number is 25.

[0170] S6, fine-tuning the pruned model to obtain a compressed model;

[0171] Preferably, the specific steps of fine-tuning the pruned model to obtain a compressed model in step S6 are:

[0172] The pruned model deletes the convolutional layer filter and the fully connected parameter corresponding to the channel according to the mask corresponding to each channel of the network layer during pruning, so as to realize the reduction of the pruned model;

[0173] The importance measurement module in the pruned model is removed to obtain a compressed model;

[0174] The compressed model is updated using the parameter configuration and loss consistent with the pre-trained model until the parameters converge, and a compressed model is obtained.

[0175] S7, inputting a to-be-detected image into the compressed model to output the object class and specific position in the detected image.

[0176] The importance measurement module is initialized. In the global importance measurement module based on information loss supervision, the batch size of the training data is set to 128. The SGD optimizer is used for optimization, the learning rate is 2×10 -1 , and the number of iterations is 8000.

[0177] In the intra-layer importance measurement module based on redundancy, the parameters of all network layers are processed on the CPU, which facilitates the hierarchical clustering algorithm to obtain the initial group importance of each channel.

[0178] The model pruning is performed; in this step, the task loss and the information loss are used for network gradient back propagation, the model channel is iteratively deleted according to the importance measurement module, and the network parameter weight is updated and optimized according to the gradient. In this embodiment, the RetianNet model is pruned to reduce the calculation amount (FLOPs) by 50% of the original. The batch size of the training data is set to 2. The entire model is optimized using the SGD optimizer, the learning rate is 5×10 -4 , the momentum update coefficient is 0.9, and the weight decay parameter is 1×10-4 At the same time, an iterative pruning parameter d = 10 is set, and one channel is pruned every 10 iterations until the FLOPs meets the condition.

[0179] Fine-tune the pruned model; this step first removes each importance measure module, then deletes the channels of the original model according to the channel mask, only retains the RetinaNet model of the pruned part of the channel, and then updates and optimizes the pruned model using the task loss. In this embodiment, the training data batch size of the pruned model is set to 2. The entire model is optimized using the SGD optimizer, the learning rate is 2.5x10 -3 , the momentum update coefficient is 0.9, and the weight decay parameter is 1x10 -4 .

[0180] On the COCO dataset, the model training epoch is set to 12. The learning rate is reduced by 0.1 times at the end of the 8th and 11th training.

[0181] On the Pascal VOC dataset, the model training epoch is set to 12. The learning rate is reduced by 0.1 times at the end of every three training.

[0182] The model training is completed, and the compressed model is used; the image to be detected is input into the compressed model, and the model output is the detection result. On the COCO dataset, the pruning model detection performance indicator mAP reaches 0.366 (the uncompressed model mAP is 0.365). On the Pascal VOC dataset, the pruning model detection performance indicator mAP reaches 0.791 (the uncompressed model mAP is 0.773). The present application has higher precision with significantly lower computational load than the original model. Using the present application, the parameter quantity and computational load of the model can be effectively reduced on the basis of ensuring the performance of the model, and the inference task can be efficiently completed.

[0183] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A pruning method for image representation models based on multi-granularity importance measurement, characterized in that, include: S1. Collect images to build an image training dataset; S2. Obtain the neural network model as a pre-trained model, and simultaneously copy the pre-trained model as a model to be pruned; use joint loss to set the training constraints of the model to be pruned; the joint loss includes the task loss inherent in the model to be pruned and the information loss of the importance measurement module; S3. Based on the training constraints, define an importance measurement module for calculating the importance of each channel in the network layer of the model to be pruned; the importance measurement module includes a global importance measurement module based on joint loss and an intra-layer importance measurement module based on redundancy; The global importance measurement module based on joint loss includes an importance measurement unit using the original task loss and an importance measurement unit using information loss. The importance of each channel in the network layer of the model to be pruned includes the task loss importance of the network layer channel, the information loss importance of the network layer channel, and the group importance of the network layer channel. The expression for the importance of each channel in the network layer of the model to be pruned is: ； in, For the network layers of the model to be pruned l passage i The importance of; For the network layers of the model to be pruned l passage i The importance of mission loss; For the network layers of the model to be pruned l passage i The importance of information loss; For the network layers of the model to be pruned l passage i The importance of the group; S4. Initialize the importance measurement module in the model to be pruned; S5. Input the image training dataset into the initialized model to be pruned, use the importance measurement module to calculate the importance of each channel, and prune the model based on the importance of each channel of the network layer of the model to be pruned and the training constraints to obtain the pruned model. S6. Fine-tune the pruning model to obtain the compressed model; S7. Input the image to be detected into the compression model, and output the object category and specific location in the detected image.

2. The image representation model pruning method according to claim 1, characterized in that, The specific steps for establishing the image training dataset in step S1 are as follows: Prepare images and construct an image dataset; The image dataset is preprocessed to obtain the preprocessed image dataset; The preprocessed image dataset is divided to obtain the image training dataset.

3. The image representation model pruning method according to claim 1, characterized in that, The pruning model described in step S2 includes a backbone network, a transition network, and a network head.

4. The image representation model pruning method according to any one of claims 1-3, characterized in that, The specific steps to obtain the initialized model to be pruned are as follows: Freeze the remaining parameters of the model to be pruned, and set the normalization layer of the model to validation mode; The projection layer in the global importance measurement module based on information loss is updated using information loss optimization until the information loss of the importance measurement module tends to converge, thus obtaining the initialized importance measurement module based on information loss. Using a redundancy-based intra-layer importance measurement module, all network layers of the model to be pruned are traversed, and the group importance of all network layers is initialized using a hierarchical clustering algorithm.

5. The image representation model pruning method according to any one of claims 1-3, characterized in that, The specific steps to obtain the pruning model are as follows: The network layer of the model to be pruned receives the image training dataset and performs forward and backward propagation to obtain the importance of each channel of the network layer of the model to be pruned. Based on the importance of each channel in the network layer of the model to be pruned, the channels with the least importance in the network layer are pruned. After each backpropagation, the gradient of the input of each network layer of the model to be pruned with respect to the task loss is obtained. This gradient is then used to iteratively update the network layers until the _____ is reached. d Second-rate, d Perform a pruning operation once for a preset number of iterations, where the first iteration... d After -1 iterations, for the ... d The importance of each channel in the network layer of the model to be pruned in all previous iterations is summed channel by channel, and the importance of each channel is summed in the nth iteration. d After each iteration, all unpruned channels in the network are sorted in ascending order according to the importance of each channel in the network layer of the model to be pruned. The channels with the lowest importance and whose network layers have not been completely deleted are selected for pruning to obtain the pruned model.

Images