Channel pruning method for visual understanding model of classification task and application

Abstract

The invention belongs to the technical field of visual understanding, and discloses a channel pruning method and application of a visual understanding model for classification tasks, and the method comprises the steps: employing a divergence-based channel importance function to calculate the importance value of each channel in a trained visual understanding model; wherein the divergence-based channel importance function is a product of a standard deviation of sample distribution corresponding to each channel output feature map in the visual understanding model and a variable coefficient, and the variable coefficient is equal to a ratio of the standard deviation of the sample distribution to a mathematical expectation of the sample distribution; the standard deviation of the sample distribution is used for reflecting the dispersion degree between the samples; and sorting the importance values of the channels, and pruning the corresponding channels from the channels with low importance values until the parameters and operand required by the visual understanding model are met, thereby obtaining the pruned lightweight visual understanding model. According to the method, the classification performance of the model can be improved under a relatively high pruning rate.

Description

Technical Field

[0001] The present invention belongs to the field of visual understanding technology, and more specifically, relates to a channel pruning method and application of a visual understanding model for classification tasks. Background Art

[0002] In recent years, deep learning technology has demonstrated powerful capabilities in the field of visual understanding. However, with the gradual increase in the number of model parameters and the continuous increase in network depth, the consumption of computing resources for reasoning in deep learning networks has also increased rapidly, placing higher requirements on hardware performance. In particular, as more models are deployed on smartphones and embedded devices, it is difficult for current deep learning technology to move towards larger-scale applications if deep neural networks that consume a lot of computing and storage costs are deployed on resource-constrained devices. To this end, model pruning methods have been proposed to perform lightweight processing on various visual understanding models. Among them, channel pruning (also known as filter pruning) directly removes all parameters of the entire redundant channel. Its sparse pattern has structured characteristics and does not require the support of specific hardware and algorithm libraries. It is the most widely used.

[0003] Model pruning aims to remove redundant parts of the model while minimizing the impact on the model's prediction accuracy. Therefore, how to identify which parts of the model are redundant has become the focus of pruning methods. The current mainstream method is to determine whether a channel is important based on the contribution of each channel. This type of method assumes that channels with larger parameter amplitudes, larger feature map values, or larger feature map ranks have a greater impact on the output, and are therefore more important and less redundant. In addition, there are also some methods that measure the importance of each channel based on the relationship between channels. For example, the more similar the parameters of a channel in the convolutional layer are to those of other channels, the higher its redundancy and the lower its importance.

[0004] Although the above methods have been proven to be able to achieve a large pruning rate and are relatively stable, they all ignore the distinguishability between samples in each channel during the pruning process, resulting in poor classification performance of the pruned visual understanding model. Specifically, if the feature map of a certain channel can fully and comprehensively capture and reflect the difference between samples on a certain attribute, the features of this channel are conducive to the network distinguishing samples of different categories. However, if different samples have only extremely subtle feature differences in the feature map of a certain channel, the features of this channel are not conducive to distinguishing samples of different categories, making it difficult to perform recognition tasks of different categories. In other words, sample distinguishability has a great impact on the classification results. Therefore, if the distinguishability between samples in each channel is ignored during pruning, the classification performance of the pruned visual understanding model will be affected. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a channel pruning method and application of a visual understanding model for classification tasks, which aims to improve the classification performance of the pruned visual understanding model.

[0006] To achieve the above object, according to a first aspect of the present invention, a channel pruning method for a visual understanding model for a classification task is provided, comprising:

[0007] The importance value of each channel in the trained visual understanding model is calculated using a divergence-based channel importance function; wherein the divergence-based channel importance function is the product of the standard deviation of the sample distribution corresponding to the output feature map of each channel in the visual understanding model and the coefficient of variation, and the coefficient of variation is equal to the ratio of the standard deviation of the sample distribution to the mathematical expectation of the sample distribution; the standard deviation of the sample distribution is used to reflect the degree of dispersion between samples;

[0008] The importance values ​​of each channel are sorted, and starting from the channels with low importance values, the corresponding channels are pruned until the number of parameters and the amount of computation required by the visual understanding model are met, thereby obtaining a pruned lightweight visual understanding model.

[0009] Furthermore, the visual understanding model includes a convolutional layer and / or a linear layer, and an activation function layer and a normalization layer adjacent to the convolutional layer and / or the linear layer;

[0010] The divergence-based channel importance function is:

[0011]

[0012]

[0013] In the formula, is the random variable of the j-th channel feature map of the activation function layer adjacent to the i-th layer of the convolutional layer and / or the linear layer; express The variance of the distribution followed by The distribution obeyed is used to characterize the sample distribution corresponding to the channel feature map; express The coefficient of variation of the distribution followed; express The mathematical expectation of the distribution it obeys.

[0014] Furthermore, using mathematical expectation and variance Estimated value of Corresponding to replace mathematical expectation and variance

[0015] Estimated value The calculation method is:

[0016]

[0017] Among them, f(x) is the activation function, x represents the random variable and Corresponding to the estimated value when the activation function is f(x) β j With γ j They represent the learnable offset coefficient and scale coefficient in the jth channel of the normalization layer, respectively.

[0018] Furthermore, the activation function f(x) is a ReLU activation function, a ReLU6 activation function or a tanh activation function.

[0019] Furthermore, the trained visual comprehension model is obtained by training an untrained visual comprehension model using a standard data set;

[0020] After obtaining the pruned lightweight visual understanding model, the method further includes:

[0021] The pruned lightweight visual comprehension model is trained again on the standard data set to obtain a final lightweight visual comprehension model.

[0022] According to a second aspect of the present invention, a visual understanding model compression method for a classification task is provided, comprising:

[0023] Prune the trained visual understanding model using the channel pruning method described in any one of the first aspects to obtain a pruned lightweight visual understanding model;

[0024] The pruned lightweight visual understanding model is used as the compressed visual understanding model.

[0025] According to the third aspect of the present invention, a visual understanding task classification method is provided, comprising: inputting an image to be classified into a trained lightweight visual understanding model to obtain a classification prediction result; wherein the lightweight visual understanding model is obtained by the channel pruning method described in any one of the first aspects.

[0026] According to a fourth aspect of the present invention, there is provided an electronic device comprising a computer-readable storage medium and a processor;

[0027] The computer-readable storage medium is used to store executable instructions;

[0028] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the channel pruning method described in any one of the first aspects, or to execute the visual understanding model compression method described in the second aspect, or to execute the visual understanding task classification method described in the third aspect.

[0029] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the channel pruning method as described in any one of the first aspects, or implements the visual understanding model compression method as described in the second aspect, or implements the visual understanding task classification method as described in the third aspect.

[0030] According to the sixth aspect of the present invention, a computer program product is provided. When the computer program product is run on a computer, the computer executes the channel pruning method described in any one of the first aspects, or implements the visual understanding model compression method described in the second aspect, or implements the visual understanding task classification method described in the third aspect.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0032] (1) In the present invention, in order to distinguish samples of different categories during the pruning process, the present invention designs a channel importance function based on divergence, which is calculated as follows: the product of the standard deviation of the sample distribution corresponding to each channel output feature map in the visual understanding model and the coefficient of variation, wherein the coefficient of variation of the sample distribution can control the information entropy of the entire feature map distribution. The larger the coefficient of variation, the lower the information entropy. Considering that the model classification is essentially a process of gradually decreasing information entropy, if the model feature map can maintain a low information entropy, it is beneficial to the classification of the model. At the same time, the standard deviation of the sample distribution is used to reflect the degree of dispersion between samples. The influence of the standard deviation of the sample distribution of each channel output feature map is considered in the channel importance function, which can encourage a greater degree of dispersion between samples, that is, the better the feature distinguishability of the channel, the more conducive it is to capture the subtle differences between samples, and the stronger the classification performance. Therefore, the importance values ​​of the channels calculated by the divergence-based channel importance function are sorted, and the channels with lower importance values ​​are pruned. The pruned visual understanding model obtained can improve the classification performance of the pruned visual understanding model.

[0033] (2) Preferably, the divergence-based channel importance function in the present invention sorts the importance values ​​of each channel, and prunes the corresponding channels starting from the channels with low importance values. This can retain lower information entropy in the features of the middle layers (convolutional layers, and / or linear layers) of the visual understanding model, which is beneficial to reducing the reduction of information entropy in subsequent layers of the visual understanding model and facilitating the improvement of classification performance.

[0034] (3) Furthermore, in order to reduce the computational cost, the mathematical expectation is used in the present invention and variance Estimated value of Corresponding to replace mathematical expectation and variance A corresponding parameter estimation method is provided, in which only the model parameters (β j and γ j ) is used to mine the feature map distribution information, avoiding the additional large computational cost of using a large amount of real data to collect sample distribution in the feature map, and accelerating the pruning process.

[0035] (4) Furthermore, after obtaining the pruned lightweight visual understanding model, the pruned lightweight visual understanding model is trained again on the standard dataset to recover the performance loss caused by channel pruning and further improve the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a channel pruning method for a visual understanding model for classification tasks in an embodiment of the present invention;

[0037] Figure 2 Flow chart of a channel pruning method for a visual understanding model for classification tasks in an embodiment of the present invention;

[0038] Figure 3a is an image to be tested in an embodiment of the present invention;

[0039] Figure 3b for Figure 3a The corresponding true value label;

[0040] Figure 3c for Figure 3a The corresponding unpruned baseline model that has been initially trained;

[0041] Figure 3d for Figure 3a The corresponding segmentation prediction results of the pruned lightweight visual understanding model obtained by the DCFP channel pruning method;

[0042] Figure 3e for Figure 3a Correspondingly, the segmentation prediction result of the lightweight visual understanding model obtained by the method in the embodiment of the present invention;

[0043] Figure 4a is another image to be tested in an embodiment of the present invention;

[0044] Figure 4b for Figure 4a The corresponding true value label;

[0045] Figure 4c for Figure 4a The corresponding unpruned baseline model that has been initially trained;

[0046] Figure 4d for Figure 4a The corresponding segmentation prediction results of the pruned lightweight visual understanding model obtained by the DCFP channel pruning method;

[0047] Figure 4e for Figure 4a Correspondingly, the segmentation prediction result of the lightweight visual understanding model is obtained by the method in the embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown, the channel pruning method of the visual understanding model for classification tasks in the embodiment of the present invention mainly includes:

[0051] A divergence-based channel importance function is used to calculate the importance value of each channel in the trained visual understanding model; wherein the divergence-based channel importance function is calculated as follows: the product of the standard deviation of the sample distribution corresponding to the output feature map of each channel in the visual understanding model and the coefficient of variation, the coefficient of variation is equal to the ratio of the standard deviation of the sample distribution to the mathematical expectation of the sample distribution; the standard deviation of the sample distribution is used to reflect the degree of dispersion between samples.

[0052] Sort the importance values ​​of each channel, start from the channel with the low importance value, and prune the corresponding channels until the number of parameters and the amount of computation required by the visual understanding model are met, so as to obtain a pruned lightweight visual understanding model and reduce the number of parameters and the amount of computation. In an embodiment of the present invention, the number of parameters required for the visual understanding model includes the number of parameters and sparsity of the channel; the amount of computation includes the number of floating-point operations per second of the visual understanding model. Pruning the corresponding channels includes: deleting channels with low importance values ​​and their corresponding network parameters. It is preferred to arrange the importance values ​​of each channel in descending order. When pruning, delete the channels and their corresponding network parameters in order from back to front according to the sorting order of the importance values ​​until the requirements are met.

[0053] The method also includes: fine-tuning the pruned lightweight visual understanding model to obtain a final lightweight visual understanding model.

[0054] Specifically, the trained visual understanding model is obtained by training an untrained visual understanding model using a standard data set, wherein each sample in the standard training set includes an image related to the target visual understanding task and its corresponding true label value, and the true label value is used to indicate the task result.

[0055] After obtaining the pruned visual understanding model, the pruned visual understanding model is fine-tuned, including: retraining the pruned visual understanding model on the above-mentioned standard data set to obtain the final lightweight visual understanding model. Through retraining, the pruned lightweight model can be adjusted to obtain parameters that adapt to its structure, and the excellent classification performance can be restored based on the large divergence channel and its feature map retained by the lightweight model.

[0056] In the present invention, in order to distinguish samples of different categories during the pruning process, the present invention designs a channel importance function based on divergence, which is calculated as follows: the product of the standard deviation of the sample distribution corresponding to each channel output feature map in the visual understanding model and the coefficient of variation, wherein the coefficient of variation of the sample distribution can control the information entropy of the entire feature map distribution, and the larger the coefficient of variation, the lower the information entropy, and considering that the model classification is essentially a process of gradually decreasing information entropy, if the model feature map can maintain a low information entropy, it is beneficial to the classification of the model. At the same time, the standard deviation of the sample distribution is used to reflect the degree of dispersion between samples. The influence of the standard deviation of the sample distribution of each channel output feature map is considered in the channel importance function, which can encourage a greater degree of dispersion between samples, that is, the better the feature distinguishability of the channel, the more conducive it is to capture the subtle differences between samples, and the classification performance is stronger. Therefore, the importance values ​​of the channels calculated by the channel importance function based on the divergence are sorted, and the channels with lower importance values ​​are pruned, and the pruned visual understanding model obtained can improve the classification performance of the pruned visual understanding model.

[0057] Fine-tune the pruned lightweight visual understanding model and train the pruned lightweight model on a standard dataset to optimize the learnable parameters of the remaining channels in the lightweight model, thereby restoring the performance of the model to recover the performance loss caused by channel pruning.

[0058] Preferably, the visual understanding model in the embodiment of the present invention includes a convolutional layer, and / or a linear layer, and an activation function layer and a normalization layer adjacent to the convolutional layer and / or the linear layer. Correspondingly, the divergence-based channel importance function in the embodiment of the present invention adopts an ideal divergence channel importance function, specifically:

[0059]

[0060] in, is the random variable of the j-th channel feature map of the activation function layer adjacent to the i-th layer of the convolutional layer and / or linear layer of the visual understanding model; is a random variable The distribution of the j-th channel feature map of the activation function layer adjacent to the i-th layer of the obedient convolutional layer and / or linear layer is used to characterize the sample distribution corresponding to the channel feature map; represents the divergence-based channel importance function, represents the ideal divergence channel importance function; express The distribution The true variance of express The true coefficient of variation is calculated as follows:

[0061]

[0062] in, represents the coefficient of variation, that is, express The distribution The true mathematical expectation of .

[0063] In an embodiment of the present invention, the importance value of each channel is sorted based on the divergence of the channel importance function, and the corresponding channels are pruned starting from the channels with low importance values. This can retain a lower information entropy in the features of the middle layers (convolutional layers, and / or linear layers) of the visual understanding model, which is beneficial to reducing the reduction of information entropy in subsequent layers of the visual understanding model and facilitating the improvement of classification performance.

[0064] As a further design of the present invention, considering the real distribution data ( and ) is inferred from the data of the trained unpruned visual understanding model on the standard dataset. It is necessary to count all the feature maps corresponding to the samples. When the dataset and the visual understanding model are large, or a small part of the data cannot accurately represent the characteristics of the entire dataset, it is necessary to obtain and The computational cost is very high. In this case, in order to reduce the computational cost, we can use and The estimated value replaces the corresponding true value. In the embodiment of the present invention, a preferred feature map sample distribution estimation method is provided. and Estimate the standard deviation, expectation and other parameters of the output feature map distribution at a low computational cost based on the parameters stored in the convolution layer in each channel, and / or the activation function layer and normalization layer after the linear layer, and use the estimated value instead of the true value. The specific calculation method is:

[0065]

[0066] in, express An estimated value of and Corresponding to and An estimated value of based on and mathematical expectation Estimated value of get.

[0067]

[0068] Among them, f(x) is the activation function, x represents the random variable β j With γ j They represent the learnable bias and scale factor in the jth channel of the normalization layer, respectively.

[0069] The estimation method constructed in the embodiment of the present invention only needs to convert the learnable offset coefficient β in the jth channel of the normalization layer j and the scale factor γ j Substitute the corresponding formula to achieve and There is no need to estimate the random variables in the feature map Sampling is performed, which greatly reduces the computational cost.

[0070] Preferably, the activation function f(x) can be a commonly used activation function such as a ReLU activation function, a ReLU6 activation function or a tanh activation function.

[0071] When the activation function f(x) is the ReLU activation function, and They are:

[0072]

[0073] in, and They represent the estimated expected value and estimated variance value of the convolution layer-batch normalization layer-ReLU activation function layer, respectively. ReLU(·) is the ReLU activation function, erf(·) and erfc(·) are the Gaussian error function and the complementary Gaussian error function, respectively.

[0074] When the activation function f(x) is the ReLU6 activation function, and They are:

[0075]

[0076] in, and They represent the estimated expected value and estimated variance value of the convolution layer-batch normalization layer-ReLU6 activation function layer, respectively. ReLU6(·) is the ReLU6 activation function.

[0077] When the activation function f(x) is the tanh activation function, and They are:

[0078]

[0079] in, and They represent the estimated expected value and estimated variance of the convolutional layer-batch normalization layer-tanh activation function layer, respectively. Tanh(·) is the hyperbolic tangent activation function.

[0080] Preferably, the visual understanding model used for classification tasks in the embodiments of the present invention may be an image classification recognition model, an image segmentation model, an image detection model, or other visual understanding models used for visual understanding classification prediction tasks.

[0081] Example 2

[0082] An embodiment of the present invention provides a visual understanding model compression method for classification tasks, including: pruning a trained visual understanding model using the channel pruning method in the above-mentioned embodiment 1 to obtain a pruned lightweight visual understanding model; and using the pruned lightweight visual understanding model as a compressed visual understanding model.

[0083] For related schemes, please refer to the corresponding description in Example 1, which will not be repeated here.

[0084] The beneficial effects that can be achieved by the present invention are further explained below in combination with some specific application scenarios:

[0085] Taking the semantic segmentation task as an example of the target visual understanding task, ResNet50 is used as the backbone network to perform the target visual understanding task, Deeplabv3 is used as the decoder to perform the target visual understanding task, and the ReLU activation function and its corresponding and The channel importance calculated by the estimation method is used to perform channel pruning on the ResNet50-Deeplabv3 model, achieving the goal of reducing the number of floating-point operations per second (FLOPs) by 60%, so as to lightweight the visual understanding model, improve the model's operating efficiency and computing resource consumption, and try to obtain semantic segmentation performance similar to that of the original unpruned model.

[0086] The ADE20K dataset is used to train the initial visual understanding model. The ADE20K dataset is an image segmentation dataset for complex scenes, which contains 20,000 images as training sets, 2,000 images as validation sets, and 3,000 images as test sets. Each pixel is labeled with a predefined category, and the dataset has 150 predefined semantic labels. The training set is randomly divided into training subsets of the same size. Specifically, the size of each training subset is 16. The data of each training subset is augmented to improve the accuracy. The data augmentation method specifically includes the following operations: (1) Calculate the mean of each channel of the image in the training set. (2) Subtract the image mean from each image in the training subset. (3) Randomly flip horizontally and randomly scale, with the scaling factor being any one of {0.5, 0.75, 1.0, 1.5, 1.7}. During the training process, one training subset is trained each time, and the training of all training subsets is considered an iteration. Repeat the training until the number of iterations reaches the upper limit to obtain a trained visual understanding model. In actual training, the upper limit of the number of iterations is preferably 100,000. In the above iterative training, the training process in one iteration is as follows: the network parameters of the visual understanding model are trained using the forward propagation algorithm and the back propagation algorithm. Forward propagation calculates the loss function corresponding to each training subset, and back propagation obtains the gradient corresponding to this training subset. The learning rate is set to 0.0001. The loss calculation uses the cross entropy loss.

[0087] The ResNet50-Deeplabv3 model is pruned using the divergence-based visual understanding model channel pruning method proposed in the present invention, where the maximum channel sparsity rate of each layer is set to 90%, that is, at most 90% of the channels in the same layer are deleted. Channel pruning adopts a global importance setting, that is, each channel is sorted according to its importance with all channels in all layers to be pruned in the entire network as the order of subsequent channel pruning. During the pruning process, the layer that reaches the maximum channel sparsity rate is immediately excluded from the layer to be pruned, and all remaining channels of the layer are retained, and the layer no longer participates in channel pruning.

[0088] Similarly, the ADE20K dataset is used to fine-tune the pruned visual understanding model. The upper limit of the number of iterations is preferably 50,000, and the settings such as training subset size, data augmentation, and learning rate are kept consistent with the corresponding settings of the initial model training.

[0089] The pruned and compressed lightweight visual understanding model is used to perform visual understanding tasks in different scenarios. Taking the semantic segmentation task, which is easier to demonstrate the prediction quality, as an example, the segmentation prediction results of the lightweight model with a 60% reduction in floating point calculations per second under the geographic information system are compared. Figure 3a to Figure 3e , Figure 4a to Figure 4e As shown. Among them, Figure 3a and Figure 4a are two different test images input. Figure 3b and Figure 4b is the corresponding true value label, and different colors represent different category true values. Figure 3c to Figure 3e and Figure 4c to Figure 4e In the figure, different shades of colors represent different category predictions. Figure 3c and Figure 4c Predict results for the unpruned, initially trained baseline model (unpruned, trained visual understanding model), Figure 3d and Figure 4d The segmentation prediction results of the pruned lightweight visual understanding model obtained using the mainstream DCFP channel pruning method. Figure 3e and Figure 4e It is the segmentation prediction result of the lightweight visual understanding model obtained in the embodiment of the present invention.

[0090] It can be seen that Figure 3a to Figure 3e and Figure 4a to Figure 4e In contrast Figure 3c and Figure 3e ,as well as Figure 4c and Figure 4e , the prediction results of trees and building walls in the lightweight visual understanding model obtained by using the divergence-based channel pruning method in the embodiment of the present invention are basically consistent with the prediction results of the baseline model. And, obviously, compared Figure 3d and 3e ,as well as Figure 4d and 4e ,Compared with the prediction results of DCFP, the model in the ,embodiment of the present invention has obvious performance ,advantages.

[0091] Example 3

[0092] An embodiment of the present invention provides a visual understanding task classification method, including:

[0093] The image of the visual understanding task to be classified is input into the trained lightweight visual understanding model to obtain a classification prediction result; wherein the lightweight visual understanding model is obtained by the channel pruning method in the above-mentioned embodiment 1.

[0094] For related schemes, please refer to the corresponding description in Example 1, which will not be repeated here.

[0095] Example 4

[0096] An embodiment of the present invention provides an electronic device, including a computer-readable storage medium and a processor;

[0097] Computer-readable storage media for storing executable instructions;

[0098] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the channel pruning method in Example 1, or to execute the visual understanding model compression method in Example 2, or to execute the visual understanding task classification method in Example 3.

[0099] For related schemes, please refer to the corresponding descriptions in Examples 1-3, which will not be repeated here.

[0100] Example 5

[0101] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the channel pruning method as in Example 1 is implemented, or the visual understanding model compression method as in Example 2 is implemented, or the visual understanding task classification method as in Example 3 is implemented.

[0102] For related schemes, please refer to the corresponding descriptions in Examples 1-3, which will not be repeated here.

[0103] Example 6

[0104] A computer program product is provided in an embodiment of the present invention. When the computer program product runs on a computer, the computer executes the channel pruning method in Example 1, or implements the visual understanding model compression method in Example 2, or implements the visual understanding task classification method in Example 3.

[0105] For related schemes, please refer to the corresponding descriptions in Examples 1-3, which will not be repeated here.

[0106] The channel pruning method of the divergence-based visual understanding model provided by the present invention overcomes the disadvantage of insufficient discriminative features caused by previous methods ignoring the important characteristic of sample distinctiveness by taking the degree of dispersion of sample distribution as the pruning criterion when performing channel pruning. It can retain more discriminative channels and their corresponding feature maps in the lightweight visual understanding model after pruning, thereby having stronger classification performance than the current mainstream pruning methods.

[0107] The divergence-based channel importance function provided by the present invention provides an effective quantitative indicator of sample discrimination, and at the same time opens up a new idea for measuring channel importance, adding a new perspective to the existing methods for measuring channel importance, and has better classification performance than the existing methods at a higher pruning ratio.

[0108] The feature map sample distribution estimation method provided by the present invention can mine the feature map distribution information using only model parameters, avoiding the additional large amount of computational cost brought by using a large amount of real data to collect the sample distribution in the feature map, and realizing the distribution information estimation of the intermediate layer features of the visual understanding model with low computational complexity, while achieving high-precision sample discrimination degree estimation and accelerating the pruning process.

[0109] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A channel pruning method for a visual understanding model for classification tasks, characterized in that: include: The importance value of each channel in the trained visual understanding model is calculated using a divergence-based channel importance function; wherein the divergence-based channel importance function is the product of the standard deviation of the sample distribution corresponding to the output feature map of each channel in the visual understanding model and the coefficient of variation, and the coefficient of variation is equal to the ratio of the standard deviation of the sample distribution to the mathematical expectation of the sample distribution; the standard deviation of the sample distribution is used to reflect the degree of dispersion between samples; The importance values ​​of each channel are sorted, and starting from the channels with low importance values, the corresponding channels are pruned until the number of parameters and the amount of computation required by the visual understanding model are met, thereby obtaining a pruned lightweight visual understanding model.

2. The channel pruning method according to claim 1, characterized in that: The visual understanding model includes a convolutional layer and / or a linear layer, and an activation function layer and a normalization layer adjacent to the convolutional layer and / or the linear layer; The divergence-based channel importance function is: In the formula, is the random variable of the j-th channel feature map of the activation function layer adjacent to the i-th layer of the convolutional layer and / or the linear layer; express The variance of the distribution followed by The distribution obeyed is used to characterize the sample distribution corresponding to the channel feature map; express The coefficient of variation of the distribution followed; express The mathematical expectation of the distribution it obeys.

3. The channel pruning method according to claim 2, characterized in that: Using mathematical expectation and variance Estimated value of Corresponding to replace mathematical expectation and variance Estimated value The calculation method is: Among them, f(x) is the activation function, x represents the random variable and Corresponding to the estimated value when the activation function is f(x) β j With γ j They represent the learnable offset coefficient and scale coefficient in the jth channel of the normalization layer, respectively.

4. The channel pruning method according to claim 3, characterized in that: The activation function f(x) is a ReLU activation function, a ReLU6 activation function or a tanh activation function.

5. The channel pruning method according to any one of claims 1 to 4, characterized in that: The trained visual understanding model is obtained by training an untrained visual understanding model using a standard data set; After obtaining the pruned lightweight visual understanding model, the method further includes: The pruned lightweight visual comprehension model is trained again on the standard data set to obtain a final lightweight visual comprehension model.

6. A visual understanding model compression method for classification tasks, characterized in that: include: Pruning the trained visual understanding model using the channel pruning method described in any one of claims 1 to 5 to obtain a pruned lightweight visual understanding model; The pruned lightweight visual understanding model is used as the compressed visual understanding model.

7. A visual understanding task classification method, characterized in that: include: The image to be classified is input into a trained lightweight visual understanding model to obtain a classification prediction result; wherein the lightweight visual understanding model is obtained by the channel pruning method described in any one of claims 1-5.

8. An electronic device, characterized in that: comprising a computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the channel pruning method described in any one of claims 1-5, or to execute the visual understanding model compression method described in claim 6, or to execute the visual understanding task classification method described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the channel pruning method as described in any one of claims 1 to 5, or implements the visual understanding model compression method as described in claim 6, or implements the visual understanding task classification method as described in claim 7.

10. A computer program product, characterized in that When the computer program product runs on a computer, the computer executes the channel pruning method described in any one of claims 1 to 5, or implements the visual understanding model compression method described in claim 6, or implements the visual understanding task classification method described in claim 7.

Images