Image processing system and method based on dynamic pruning

Abstract

The present application discloses an image processing system and method based on dynamic pruning, which relates to the field of computer vision. The image processing system based on dynamic pruning includes: a feature extraction module, which is used to extract the feature map of each image to be processed by using a feature extraction network; an adaptive convolution acquisition module, which is used to obtain a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel; a channel learning module, which is used to learn the importance of each channel in the feature map to obtain the channel importance; a dynamic pruning module, which is used to dynamically prune the initial image processing network to obtain a sparse network; an image processing module, which is used to process each image to be processed by using a sparse network to obtain a corresponding processed image. It can not only reduce the computing power required for image processing, but also complete image processing tasks more targeted.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image processing system and method based on dynamic pruning. Background Art

[0002] Image processing is to apply certain operations and processing to images to extract various information from them. Deep learning applications have gained great attention in the field of image processing due to their extremely high prediction accuracy in recognition applications, and deep neural networks such as convolutional neural networks are gradually replacing traditional image processing work.

[0003] At present, with the development of deep learning technology, in order to achieve higher accuracy in image processing, most convolutional neural networks (CNNs) have very deep structures, which leads to very large forward reasoning and video memory overhead. Although these large-sized CNNs can be run on powerful GPU (graphics processing unit) clusters, if they are run in real time (>30 frames / s) in real scenes, such as mobile devices and embedded devices, these platforms have few memory resources, low processor performance, and limited power consumption, which makes it difficult to deploy the most accurate models on these platforms and achieve real-time operation speed. Therefore, how to design a model with fewer parameters, less calculation and faster speed while ensuring accuracy is a hot issue at present and a challenging direction.

[0004] The compression and acceleration of deep neural network models refers to the use of the redundancy of neural network parameters and network structure to simplify the model, without affecting the task completion, to obtain a model with fewer parameters and a more streamlined structure. The compressed model has smaller computing resource model and memory requirements, and can meet a wider range of application needs than the original model. As a branch of model compression, neural network pruning technology reduces the computing consumption of large neural networks by removing unimportant branches, so that they can run on mobile devices. Figure 1 A specific network pruning diagram is shown in Figure 1. Network pruning uses a certain metric to remove some weights in the network or remove some connections between weights. The model after channel pruning can be trained using existing hardware and computing libraries, and ultimately can achieve higher compression rates and shorter computing time. The corresponding relationship between the feature map and the convolution kernel after channel pruning is shown in Figure 1. Figure 2 As shown in Figure 2, when a channel of a feature map is subtracted, the convolution kernel connected to it will also be removed.

[0005] In network pruning, there are two main pruning methods, one is static pruning and the other is dynamic pruning. After static pruning, the pruned parameters are permanently removed from the model and no longer participate in reasoning and training. Although most of the parameters in the network are redundant, the static pruning algorithm will still permanently remove some key parameters. No matter what evaluation criteria are adopted, it is difficult to avoid mispruning, which will inevitably lead to loss of network capacity. For this reason, dynamic pruning is proposed; dynamic pruning does not permanently remove any parameters, but dynamically selects part of the network to participate in the calculation according to the type of task to be processed, so as to reduce the complexity of the model. For example, if the type of task to be processed is image processing, a fixed part of the parameters are selected to participate in the calculation. However, even if they are all image processing, the images to be processed are different. Therefore, fixed selection of some parameters cannot select the network for image processing in a more targeted manner.

[0006] In summary, it can be seen that how to reduce the computing power required for image processing and complete image processing tasks more specifically is a problem to be solved in this field. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide an image processing system and method based on dynamic pruning, which can not only reduce the computing power required for image processing, but also complete the image processing task more specifically. The specific scheme is as follows:

[0008] In a first aspect, the present application discloses an image processing system based on dynamic pruning, comprising:

[0009] A feature extraction module, used to extract feature maps of each image to be processed using a feature extraction network;

[0010] An adaptive convolution acquisition module, used to acquire a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after the weight adjustment;

[0011] A channel learning module is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance;

[0012] A dynamic pruning module, used for dynamically pruning the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network;

[0013] The image processing module is used to process each of the images to be processed using the sparse network to obtain a corresponding processed image.

[0014] Optionally, the adaptive convolution acquisition module includes:

[0015] A context fusion submodule, used for mapping the feature map to a feature map of a first preset channel to extract context information of the feature map;

[0016] A channel interaction submodule, configured to convert the feature map of the first preset channel into a feature map of the second preset channel based on the context information and using a group linear layer;

[0017] The coefficient generation submodule is used to obtain a four-dimensional tensor using the feature map of the first preset channel and the feature map of the second preset channel.

[0018] Optionally, the context fusion submodule includes:

[0019] A resolution adjustment unit, configured to reduce the resolution of the feature map by using an average pooling layer to obtain a feature map with reduced resolution;

[0020] A fusion unit is used to map the feature map after the resolution is reduced to the feature map of the first preset channel to extract context information of the feature map after the resolution is reduced.

[0021] Optionally, the adaptive convolution acquisition module includes:

[0022] The target convolution kernel acquisition submodule is used to multiply the weights of each initial convolution kernel in the initial image processing network by the four-dimensional tensor element by element, and determine the product after the multiplication as the adjusted weight, so as to obtain the adaptive target convolution kernel after the adjusted weight.

[0023] Optionally, the channel learning module includes:

[0024] The channel importance acquisition unit is used to project each channel of the feature map into a column vector, and use a fully connected layer to project the column vector into a channel importance vector, so as to learn the corresponding channel importance based on the channel importance vector.

[0025] Optionally, the channel learning module further includes:

[0026] An importance learning unit, used for learning the importance of the channel using a target learning function to obtain a channel importance distribution; wherein the target learning function is an Argmax function;

[0027] The target importance classification unit is used to classify the channel importance into a first target importance and a second target importance based on the channel importance distribution.

[0028] Optionally, the dynamic pruning module includes:

[0029] A pruning unit is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel, the first target importance and the second target importance.

[0030] Optionally, the image processing system based on dynamic pruning further includes:

[0031] The target learning function determination unit is used to learn the importance of historical channels using the Gumbel-Softmax function during the training phase to update the initial temperature parameters, obtain the target temperature parameters, and determine the target learning function based on the target temperature parameters.

[0032] Optionally, the image processing module includes:

[0033] A first processing unit, configured to process each of the to-be-processed images using the current convolution operation if the current convolution operation in the sparse network corresponds to the first target importance level;

[0034] The second processing unit is used to directly jump to the next convolution operation if the current convolution operation in the sparse network corresponds to the second target importance.

[0035] In a second aspect, the present application discloses an image processing method based on dynamic pruning, comprising:

[0036] Using a feature extraction network to extract feature maps of each image to be processed;

[0037] Acquire a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after the weight adjustment;

[0038] Performing importance learning on each channel in the feature map to obtain the corresponding channel importance;

[0039] Dynamically pruning the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network;

[0040] The sparse network is used to process each of the images to be processed to obtain a corresponding processed image.

[0041] The beneficial effects of the present application are as follows: the present application discloses an image processing system based on dynamic pruning, which includes: a feature extraction module, which is used to extract feature maps of each image to be processed by using a feature extraction network; an adaptive convolution acquisition module, which is used to obtain a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after adjusting the weight; a channel learning module, which is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance; a dynamic pruning module, which is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network; an image processing module, which is used to process each of the images to be processed by using the sparse network to obtain a corresponding processed image. It can be seen from this that the present application needs to obtain the corresponding four-dimensional tensor according to the feature map of each image to be processed, and obtain the adaptive target convolution kernel based on the four-dimensional tensor. In this way, the channel importance learning can be carried out autonomously in the future, avoiding the complicated steps brought about by the need to manually preset thresholds for importance degree classification. By performing adaptive dynamic pruning, a sparse network corresponding to each image to be processed is obtained. In other words, the present application can reduce the computing power required for the sparse network when performing image processing. On the one hand, the sparse network obtained by adaptive dynamic pruning can process each image to be processed more specifically and has more application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A specific network pruning schematic diagram disclosed in this application;

[0044] Figure 2 A schematic diagram of the correspondence between a specific feature map after channel pruning and a convolution kernel disclosed in this application;

[0045] Figure 3 A structural diagram of an image processing based on dynamic pruning disclosed in this application;

[0046] Figure 4 A structural diagram of a specific image processing based on dynamic pruning disclosed in this application;

[0047] Figure 5 A schematic diagram of a specific image classification task disclosed in this application;

[0048] Figure 6 A schematic diagram of a specific target detection task disclosed in this application;

[0049] Figure 7 This is a flow chart of an image processing method based on dynamic pruning disclosed in this application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] In network pruning, there are two main pruning methods, one is static pruning and the other is dynamic pruning. After static pruning, the pruned parameters are permanently removed from the model and no longer participate in reasoning and training. Although most of the parameters in the network are redundant, the static pruning algorithm will still permanently remove some key parameters. No matter what evaluation criteria are adopted, it is difficult to avoid mispruning, which will inevitably lead to loss of network capacity. For this reason, dynamic pruning is proposed; dynamic pruning does not permanently remove any parameters, but dynamically selects part of the network to participate in the calculation according to the type of task to be processed, so as to reduce the complexity of the model. For example, if the type of task to be processed is image processing, a fixed part of the parameters are selected to participate in the calculation. However, even if they are all image processing, the images to be processed are different. Therefore, fixed selection of some parameters cannot select the network for image processing in a more targeted manner.

[0052] To this end, the present application provides an image processing solution based on dynamic pruning, which can not only reduce the computing power required for image processing, but also complete image processing tasks in a more targeted manner.

[0053] See also Figure 3 As shown, the embodiment of the present application discloses an image processing system based on dynamic pruning, including:

[0054] A feature extraction module 11 is used to extract a feature map of each image to be processed using a feature extraction network;

[0055] An adaptive convolution acquisition module 12, used to acquire a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after the weight adjustment;

[0056] A channel learning module 13 is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance;

[0057] A dynamic pruning module 14, configured to dynamically prune the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network;

[0058] The image processing module 15 is used to process each of the to-be-processed images using the sparse network to obtain a corresponding processed image.

[0059] It can be understood that after the feature extraction module 11 receives the input images to be processed, it will use the feature extraction network to output the feature maps of each image to be processed. The feature extraction network can, for example, extract the image texture features, border features, etc. of the image to be processed. That is to say, one image to be processed has multiple feature maps, which is determined by the number of network layers.

[0060] The adaptive convolution acquisition module 12 includes: a context fusion submodule 121, which is used to map the feature map to the feature map of the first preset channel to extract the context information of the feature map; a channel interaction submodule 122, which is used to convert the feature map of the first preset channel into the feature map of the second preset channel based on the context information and using a group linear layer; a coefficient generation submodule 123, which is used to obtain a four-dimensional tensor using the feature map of the first preset channel and the feature map of the second preset channel.

[0061] The context fusion submodule 121 includes: a resolution adjustment unit, which is used to reduce the resolution of the feature map by using an average pooling layer to obtain a feature map after the resolution is reduced; and a fusion unit, which is used to map the feature map after the resolution is reduced to the feature map of the first preset channel to extract the context information of the feature map after the resolution is reduced. When obtaining the four-dimensional tensor of the feature map, it is necessary to first use the average pooling layer to reduce the spatial resolution of the input feature map from C i-1 ×H i-1 ×W i-1 Lower to C i-1 ×k×k to obtain the feature map after resolution reduction, where C i Indicates channel, H i and W i Represent the height and width respectively, i represents the number of layers, and k represents the size of the convolution kernel; the feature map after the resolution is reduced is sent to the fusion unit, and the fusion unit maps the feature map after the resolution is reduced to the feature map of the first preset channel In order to reduce the amount of subsequent calculations, R represents the real number domain to extract the context information of the feature map after resolution reduction;

[0062] The channel interaction submodule 122 transforms the feature map of the first preset channel into Convert to the feature map of the second preset channel Channel C is about to be input i-1 Project to output dimension C i , the information contained in the feature map P corresponds to the input channel. Here, P is converted to Q in order to interact the information contained in the feature map on the channel, because the subsequent processing requires information corresponding to the number of output channels, so this conversion is required here;

[0063] Next, the coefficient generation submodule 123 takes P and Q as input and decodes P and Q from the one-dimensional vector d to k×k, and then uses these two sets of decoded feature maps to obtain the four-dimensional tensor after input modulation The specific process is: decode P and Q, that is, use two weights R d×k×k The fully connected layer multiplies P and Q respectively, thus obtaining Then layer M (1) and M (2) Weights are shared between different channels, that is, various dimensions are mapped to Finally, adding the two together can get the four-dimensional tensor M after channel interaction.

[0064] The adaptive convolution acquisition module 12 includes: a target convolution kernel acquisition submodule 124, which is used to multiply the weights of each initial convolution kernel in the initial image processing network by the four-dimensional tensor element by element, and determine the product after the multiplication as the adjusted weight, so as to obtain an adaptive target convolution kernel after the adjusted weight.

[0065] It can be understood that the target convolution kernel acquisition submodule 124 converts the weights of each initial convolution kernel in the initial image processing network into Multiply the four-dimensional tensor M element by element to get the product θ i , and θ i As the adjusted weight θ i , using θ i Replace the weights of the initial convolution kernel You can get the adaptive target convolution kernel after adjusting the weights.

[0066] The beneficial effects of the present application are as follows: the present application discloses an image processing system based on dynamic pruning, which includes: a feature extraction module, which is used to extract feature maps of each image to be processed by using a feature extraction network; an adaptive convolution acquisition module, which is used to obtain a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after adjusting the weight; a channel learning module, which is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance; a dynamic pruning module, which is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network; an image processing module, which is used to process each of the images to be processed by using the sparse network to obtain a corresponding processed image. It can be seen from this that the present application needs to obtain the corresponding four-dimensional tensor according to the feature map of each image to be processed, and obtain the adaptive target convolution kernel based on the four-dimensional tensor. In this way, the channel importance learning can be carried out autonomously in the future, avoiding the complicated steps brought about by the need to manually preset thresholds for importance degree classification. By performing adaptive dynamic pruning, a sparse network corresponding to each image to be processed is obtained. In other words, the present application can reduce the computing power required for the sparse network when performing image processing. On the one hand, the sparse network obtained by adaptive dynamic pruning can process each image to be processed more specifically and has more application significance.

[0067] Reference Figure 4 As shown, the embodiment of the present invention discloses a specific image processing system module framework based on dynamic pruning. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0068] The channel learning module 13 includes: a channel importance acquisition unit 131, which is used to project each channel of the feature map into a column vector, and use a fully connected layer to project the column vector into a channel importance vector, so as to learn the corresponding channel importance based on the channel importance vector.

[0069] The channel importance acquisition unit 131 first compresses the feature map by global average pooling, and projects each channel of the feature map into a scalar, that is, projects the feature map into a column vector Then, a fully connected layer is used to transform the column vector S i Projection into channel importance vector It measures the importance of each channel. The channel importance acquisition unit 131 can adaptively learn important channels according to input features, and can be optimized along with the network to improve the accuracy of prediction.

[0070] The channel learning module 13 also includes: an importance learning unit 132, which is used to learn the channel importance using a target learning function to obtain a channel importance distribution; wherein the target learning function is an Argmax function; and a target importance classification unit 133, which is used to divide the channel importance into a first target importance and a second target importance based on the channel importance distribution. For example, there are 5 channels in total, and the channel importances are 0.9, 0.1, 0.3, 0.8, and 0.6, respectively. Then, in the test phase, the Argmax function can be used to directly determine that 0.9, 0.8, and 0.6 are the first target importances, and 0.1 and 0.3 are the second target importances. That is to say, the closer to 1, the first target importance, and the closer to 0, the second target importance. There is no need to manually preset a number to screen the number of important channels.

[0071] The image processing system based on dynamic pruning further includes: a target learning function determination unit, which is used to use the Gumbel-Softmax function to learn the importance of historical channels during the training phase to update the initial temperature parameters, obtain the target temperature parameters, and determine the target learning function based on the target temperature parameters. The Gumbel-Softmax function expression is as follows:

[0072]

[0073] In the formula, exp represents the exponential function, j is responsible for marking the important channels (marked as 1) and the unimportant channels (marked as 0) in the channel, and g i Indicates the importance of the channel, c i represents the i-th channel, G i represents the noise tensor satisfying the Gumble (0, 1) distribution, and τ represents the temperature parameter. When τ is large, the result obtained by Gumbel-Softmax is close to uniform distribution, that is, the value of each channel is close to 0.5; when τ is small, the result obtained by Gumbel-Softmax is close to binary distribution (0-1 distribution), that is, some channel values ​​are close to 1, and some channel values ​​are close to 0. Here, the value of τ is obtained according to the following formula:

[0074]

[0075] In the formula, max represents the maximum value function, and t represents the tth generation in the training process.

[0076] In the training phase, as the temperature parameter τ decreases, Gumbel-Softmax gradually approaches a binary distribution. Therefore, in the inference phase, Gumbel-Softmax is directly replaced by the Argmax function, that is, the target learning function is the Argmax function. The Argmax function removes the maximum value of the marked channels in j in the column direction to obtain the final channels marked as important or unimportant.

[0077] The dynamic pruning module 14 includes: a pruning unit 141, which is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel, the first target importance and the second target importance. It can be understood that the value of the first target importance is higher than the importance of all channels, and the value of the second target importance is lower than the importance of all channels, that is, the channel corresponding to the first target importance is the important part of all channels, and the channel corresponding to the second target importance is the unimportant part of all channels.

[0078] It is understandable that we get π i After that, we can use π i Filter out unimportant channels to reduce the amount of calculation. Combine the adaptive convolution acquisition module, channel learning module and π i Output x after dynamic pruning i It can be expressed as follows:

[0079] x i =π i norm(conv i (x i-1 ,θ i ))+β i ;

[0080] In the formula, norm represents the batch normalization operation, conv i represents the convolution operation of the i-th layer, β i Indicates bias.

[0081] It should be noted that the adaptive convolution acquisition module, channel learning module, and dynamic pruning module in this application can be applied to tasks such as image classification and target detection, for example Figure 5 A specific image classification task diagram shown in FIG. Figure 6 The schematic diagram of a specific target detection task shown in the figure performs joint optimization learning with a specific task, directly generates a sparse network while training, reduces the amount of calculation, and obtains accurate prediction results. Therefore, the image processing network can be an image classification convolutional neural network or a target detection convolutional neural network.

[0082] The image processing module 15 includes: a first processing unit, which is used to process each of the images to be processed using the current convolution operation if the current convolution operation in the sparse network corresponds to the first target importance; a second processing unit, which is used to directly jump to the next convolution operation if the current convolution operation in the sparse network corresponds to the second target importance. It can be understood that the dynamic pruning module 14 will set the channel corresponding to the second target importance to zero, so the corresponding convolution operation will be skipped later, that is, when the image processing module 15 processes the image to be processed, if the current convolution operation corresponds to the second target importance, it will be ignored and directly jump to the next convolution operation, that is, the image processing module 15 will only use the convolution operation corresponding to the first target importance to process the image to be processed.

[0083] The following is a description of this application:

[0084] 1) The feature extraction module uses the feature extraction network to extract the feature map of each image to be processed, and inputs the feature map distribution into the adaptive convolution acquisition module and the channel learning module;

[0085] 2) The adaptive convolution acquisition module includes a resolution adjustment submodule, a context fusion submodule, a channel interaction submodule, a coefficient generation submodule and a target convolution kernel acquisition submodule. The resolution adjustment submodule uses an average pooling layer to reduce the resolution of the feature map to obtain a feature map after the resolution is reduced. The context fusion submodule includes a fully connected layer, a batch regularization layer and an activation layer, which is used to map the feature map after the resolution is reduced to the feature map P of the first preset channel to extract the context information of the feature map after the resolution is reduced; the channel interaction submodule includes a group fully connected layer, a batch regularization layer and an activation layer, which is used to convert the feature map P of the first preset channel into the feature map Q of the second preset channel based on the context information and using a group linear layer; the coefficient generation submodule includes two fully connected layers M (1) and M (2) , used to obtain a four-dimensional tensor M using the feature map P of the first preset channel and the feature map Q of the second preset channel; the target convolution kernel acquisition submodule converts the weights of each initial convolution kernel in the initial image processing network Multiply the four-dimensional tensor M element by element, and multiply the product θ i Determine the adjusted weights to obtain an adaptive target convolution kernel after adjusting the weights;

[0086] 3) The channel learning module includes a channel importance acquisition unit, in which the global average pooling projects each channel of the feature map into a column vector, and uses a fully connected layer to project the column vector into a channel importance vector, so as to learn the corresponding channel importance based on the channel importance vector;

[0087] 4) The dynamic pruning module receives the adaptive target convolution kernel output by the adaptive convolution acquisition module and the channel importance output by the channel learning module, and performs dynamic pruning based on this to obtain a sparse network. The dynamic pruning module includes an importance classification submodule and a pruning unit, which are used to dynamically adjust the parameters and channels of the convolution kernel to reduce the computational burden while maintaining performance. The importance classification submodule includes an importance sorting unit, a target importance classification unit, and a target importance determination unit. In other words, the importance of each channel can be sorted before classification, or it can be directly classified based on the threshold;

[0088] 5) The image processing module uses a sparse network to process each of the images to be processed to obtain a corresponding processed image.

[0089] See also Figure 7 As shown, the embodiment of the present application discloses an image processing method based on dynamic pruning, comprising:

[0090] Step S11: using a feature extraction network to extract feature maps of each image to be processed.

[0091] Step S12: Obtain a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after adjusting the weights.

[0092] Step S13: performing importance learning on each channel in the feature map to obtain the corresponding channel importance.

[0093] Step S14: dynamically pruning the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network.

[0094] Step S15: using the sparse network to process each of the images to be processed to obtain a corresponding processed image.

[0095] The beneficial effects of the present application are as follows: the present application discloses an image processing system based on dynamic pruning, which includes: a feature extraction module, which is used to extract feature maps of each image to be processed by using a feature extraction network; an adaptive convolution acquisition module, which is used to obtain a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after adjusting the weight; a channel learning module, which is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance; a dynamic pruning module, which is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network; an image processing module, which is used to process each of the images to be processed by using the sparse network to obtain a corresponding processed image. It can be seen from this that the present application needs to obtain the corresponding four-dimensional tensor according to the feature map of each image to be processed, and obtain the adaptive target convolution kernel based on the four-dimensional tensor. In this way, the channel importance learning can be carried out autonomously in the future, avoiding the complicated steps brought about by the need to manually preset thresholds for importance degree classification. By performing adaptive dynamic pruning, a sparse network corresponding to each image to be processed is obtained. In other words, the present application can reduce the computing power required for the sparse network when performing image processing. On the one hand, the sparse network obtained by adaptive dynamic pruning can process each image to be processed more specifically and has more application significance.

[0096] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0097] The above is a detailed introduction to an image processing system and method based on dynamic pruning provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image processing system based on dynamic pruning, characterized in that: include: A feature extraction module, used to extract feature maps of each image to be processed using a feature extraction network; An adaptive convolution acquisition module, used to acquire a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after the weight adjustment; A channel learning module is used to learn the importance of each channel in the feature map to obtain the corresponding channel importance; A dynamic pruning module, used for dynamically pruning the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network; An image processing module, used for processing each of the to-be-processed images using the sparse network to obtain a corresponding processed image; The channel learning module includes: A channel importance acquisition unit, used to project each channel of the feature map into a column vector, and project the column vector into a channel importance vector using a fully connected layer, so as to learn the corresponding channel importance based on the channel importance vector; The channel learning module further includes: An importance learning unit, used for learning the importance of the channel using a target learning function to obtain a channel importance distribution; wherein the target learning function is an Argmax function; The target importance classification unit is used to classify the channel importance into a first target importance and a second target importance based on the channel importance distribution.

2. The image processing system based on dynamic pruning according to claim 1, characterized in that: The adaptive convolution acquisition module comprises: A context fusion submodule, used for mapping the feature map to a feature map of a first preset channel to extract context information of the feature map; A channel interaction submodule, configured to convert the feature map of the first preset channel into a feature map of the second preset channel based on the context information and using a group linear layer; The coefficient generation submodule is used to obtain a four-dimensional tensor using the feature map of the first preset channel and the feature map of the second preset channel.

3. The image processing system based on dynamic pruning according to claim 2, characterized in that: The context fusion submodule includes: A resolution adjustment unit, configured to reduce the resolution of the feature map by using an average pooling layer to obtain a feature map with reduced resolution; A fusion unit is used to map the feature map after the resolution is reduced to the feature map of the first preset channel to extract context information of the feature map after the resolution is reduced.

4. The image processing system based on dynamic pruning according to claim 1, characterized in that: The adaptive convolution acquisition module comprises: The target convolution kernel acquisition submodule is used to multiply the weights of each initial convolution kernel in the initial image processing network by the four-dimensional tensor element by element, and determine the product after the multiplication as the adjusted weight, so as to obtain the adaptive target convolution kernel after the adjusted weight.

5. The image processing system based on dynamic pruning according to claim 1, characterized in that: The dynamic pruning module comprises: A pruning unit is used to dynamically prune the initial image processing network based on the adaptive target convolution kernel, the first target importance and the second target importance.

6. The image processing system based on dynamic pruning according to claim 1, characterized in that: Also includes: The target learning function determination unit is used to learn the importance of historical channels using the Gumbel-Softmax function during the training phase to update the initial temperature parameters, obtain the target temperature parameters, and determine the target learning function based on the target temperature parameters.

7. The image processing system based on dynamic pruning according to claim 1, characterized in that: The image processing module comprises: A first processing unit, configured to process each of the to-be-processed images using the current convolution operation if the current convolution operation in the sparse network corresponds to the first target importance level; The second processing unit is used to directly jump to the next convolution operation if the current convolution operation in the sparse network corresponds to the second target importance.

8. An image processing method based on dynamic pruning, characterized in that: include: Using a feature extraction network to extract feature maps of each image to be processed; Acquire a four-dimensional tensor corresponding to the feature map, and use the four-dimensional tensor to adjust the weights of each initial convolution kernel in the initial image processing network to obtain an adaptive target convolution kernel after the weight adjustment; Performing importance learning on each channel in the feature map to obtain the corresponding channel importance; Dynamically pruning the initial image processing network based on the adaptive target convolution kernel and the channel importance to obtain a sparse network; Using the sparse network to process each of the images to be processed to obtain a corresponding processed image; The importance learning of each channel in the feature map to obtain the corresponding channel importance includes: Projecting each channel of the feature map into a column vector, and using a fully connected layer to project the column vector into a channel importance vector, so as to learn the corresponding channel importance based on the channel importance vector; The importance learning of each channel in the feature map to obtain the corresponding channel importance also includes: The channel importance is learned using a target learning function to obtain a channel importance distribution; wherein the target learning function is an Argmax function; and based on the channel importance distribution, the channel importance is divided into a first target importance and a second target importance.

Images