Image processing method, electronic device, computer readable medium

Abstract

The application provides an image processing method, an electronic device and a computer readable medium. The method comprises: adding a pruning parameter and a mask parameter to a trainable parameter corresponding to a channel in an image processing network; wherein the pruning parameter of the trainable parameter corresponding to the channel is used to represent an importance weight of the trainable parameter corresponding to the channel; the mask parameter of the trainable parameter corresponding to the channel is used to represent whether the trainable parameter corresponding to the channel needs to be pruned or needs to be reserved; performing first training on the image processing network after adding the pruning parameter and the mask parameter according to the mask parameter of the trainable parameter corresponding to the channel; updating the value of the pruning parameter of the trainable parameter corresponding to the channel and the value of the trainable parameter corresponding to the channel during the first training process; and generating a pruned image processing network according to the image processing network obtained through the first training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to pruning methods, image processing methods, electronic devices, and computer-readable media. Background Technology

[0002] The widespread adoption of mobile devices with camera capabilities has increased the demand for image processing technologies running on these devices, leading to ever-increasing requirements for the speed of neural networks. For example, in photography, the effectiveness of neural network noise reduction directly determines the final image quality, and the speed of noise reduction directly impacts the user's shooting experience. Due to the limitations of processor processing power in current mobile devices, neural networks need to be lightweighted during deployment to meet memory and runtime requirements. Network pruning methods, as one of the acceleration techniques for neural networks, have a far-reaching impact on the deployment of neural networks in mobile devices. Current network pruning methods are commonly used in tasks such as object detection and scene recognition, but are not suitable for image noise reduction tasks. Summary of the Invention

[0003] This application provides a pruning method, an image processing method, an electronic device, and a computer-readable medium.

[0004] In a first aspect, embodiments of this application provide a pruning method, comprising: adding pruning parameters and mask parameters to trainable parameters corresponding to channels in a first image processing network; wherein, the pruning parameters of the trainable parameters corresponding to channels are used to characterize the importance weights of the trainable parameters corresponding to channels; the mask parameters of the trainable parameters corresponding to channels are used to characterize whether the trainable parameters corresponding to channels need to be pruned or retained; performing a first training on the image processing network after adding the pruning parameters and the mask parameters according to the mask parameters of the trainable parameters corresponding to channels, wherein the values ​​of the pruning parameters and the values ​​of the trainable parameters corresponding to channels are updated during the first training process; and generating a pruned image processing network based on the image processing network obtained from the first training.

[0005] Secondly, embodiments of this application provide an image processing method, comprising: processing an image to be processed by a pruned image processing network obtained using any of the above-described pruning methods to obtain a processed image.

[0006] Thirdly, embodiments of this application provide an electronic device, including: at least one processor; and a memory storing at least one program, which, when executed by the at least one processor, implements any of the above-described pruning methods or any of the above-described image processing methods.

[0007] Fourthly, embodiments of this application provide a computer-readable medium storing a computer program, which, when executed by a processor, implements any of the above-described pruning methods or any of the above-described image processing methods.

[0008] The pruning method provided in this application reduces information loss during network pruning by updating the values ​​of pruning parameters and trainable parameters that represent importance weights simultaneously during network training. This makes the pruning method more suitable for image processing networks, and eliminates the need for fine-tuning the network after pruning.

[0009] The image processing method of this application uses a pruned image processing network obtained by the above-mentioned pruning method to process the image. Since the above-mentioned pruning method is more suitable for image processing networks with a large output range and a small error tolerance range, it can better preserve the image processing effect. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a pruning method provided in one embodiment of this application; Figure 2 This is a schematic diagram of the pruning method according to an embodiment of this application. Figure 3 A flowchart of a pruning method provided as an example of an embodiment of this application; Figure 4 A block diagram of an electronic device provided in another embodiment of this application. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions of this application, the network pruning method, electronic equipment, and computer-readable medium provided in this application will be described in detail below with reference to the accompanying drawings.

[0012] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this application.

[0013] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0014] As used herein, the term “and / or” includes any and all combinations of at least one related enumerated entry.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of at least one other feature, integral, step, operation, element, component, and / or group thereof is not excluded.

[0016] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0017] Current pruning methods are commonly used in tasks such as object detection and scene recognition. Since the outputs of these tasks are typically binary vectors, the quantization space of the output values ​​has significant redundancy. For example, in a binary output with a threshold of 0.5, 0.6 and 0.9 can both be classified as 1, and 0.4 and 0.1 can both be classified as 0. However, image denoising tasks have a smaller allowable error range in the output space, so current pruning algorithms do not use them.

[0018] Although the pruning method in this application is proposed based on the problems existing in image denoising networks, the pruning method in this application is also applicable to other networks.

[0019] In some exemplary embodiments, the pruning method of this application is applicable to image processing networks whose output range is greater than or equal to a first preset threshold and whose error tolerance range is less than or equal to a second preset threshold, such as image inpainting networks, image denoising networks, etc.

[0020] In some exemplary embodiments, the pruning method of this application is applicable to image processing networks where the ratio of the error tolerance range to the output range is less than or equal to a third preset threshold.

[0021] In some exemplary embodiments, the pruning method of this application is applicable to an error tolerance range of 2. N The image processing network is defined as follows: N is the output range. For example, if each pixel of the image is represented by N bits, then the output range is N bits, and the error tolerance range is 2^N. N .

[0022] Figure 1 A flowchart of a pruning method provided in one embodiment of this application.

[0023] Firstly, referring to Figure 1 One embodiment of this application provides a pruning method, including: Step 100: Add pruning parameters and masking parameters to the trainable parameters corresponding to each channel in the image processing network; wherein, the pruning parameters of the trainable parameters corresponding to each channel are used to characterize the importance weights of the trainable parameters corresponding to each channel; the masking parameters of the trainable parameters corresponding to each channel are used to characterize whether the trainable parameters corresponding to each channel need to be pruned or retained.

[0024] In some exemplary embodiments, trainable parameters refer to parameters whose values ​​can be updated during the training process of the image processing network before pruning and masking parameters are added.

[0025] In some exemplary embodiments, a pruning parameter and a mask parameter need to be added for each trainable parameter corresponding to each channel.

[0026] In some exemplary embodiments, if in certain scenarios the trainable parameters corresponding to certain channels do not need to be pruned, or the trainable parameters corresponding to certain channels are not allowed to be pruned, then pruning parameters and flooding parameters can be added only to the trainable parameters corresponding to the channels that need to be pruned.

[0027] In some exemplary embodiments, a channel may refer to an input channel or an output channel.

[0028] In some exemplary embodiments, pruning parameters and masking parameters are added to the trainable parameters corresponding to channels in the image processing network: such as Figure 2 As shown, pruning parameters are added to the trainable parameters corresponding to each channel in the image processing network; a second training is performed on the image processing network after adding the pruning parameters, during which the values ​​of the trainable parameters corresponding to each channel and the values ​​of the pruning parameters corresponding to each channel are updated; the values ​​of the mask parameters corresponding to each channel are determined based on the updated values ​​of the pruning parameters; and mask parameters are added to the trainable parameters corresponding to each channel in the image processing network after adding the pruning parameters based on the values ​​of the mask parameters corresponding to each channel.

[0029] This exemplary embodiment updates the values ​​of trainable parameters and pruning parameters during the second training process of the image processing network after adding pruning parameters. This allows the values ​​of trainable parameters to dynamically approach the optimal solution corresponding to the given pruning parameter values ​​by updating the values ​​of trainable parameters. Furthermore, updating the values ​​of pruning parameters accelerates the convergence of the image processing network, thereby speeding up the pruning process while ensuring the performance of the image processing network.

[0030] In some exemplary embodiments, adding pruning parameters to the trainable parameters corresponding to a channel in an image processing network means adding a pruning parameter to each trainable parameter corresponding to each channel. That is, each trainable parameter corresponding to each channel corresponds to a pruning parameter, which can be represented as W. ij , where i represents the i-th trainable parameter and j represents the j-th channel.

[0031] In some exemplary embodiments, if in certain scenarios the trainable parameters corresponding to certain channels do not need to be pruned, or if the trainable parameters corresponding to certain channels are not allowed to be pruned, then pruning parameters can be added only to the trainable parameters corresponding to the channels that need to be pruned. That is, each trainable parameter corresponding to each channel that needs to be pruned corresponds to a pruning parameter, which can be represented as W. ij , where i represents the i-th channel that needs to be pruned, and j represents the j-th trainable parameter that can be pruned.

[0032] In some exemplary embodiments, if a pruning parameter is added for each trainable parameter corresponding to each channel, then during the second training of the image processing network with the added pruning parameter, the values ​​of the trainable parameters corresponding to the channels and the values ​​of the pruning parameters for each trainable parameter corresponding to each channel are updated.

[0033] In some exemplary embodiments, if a pruning parameter is added for each trainable parameter that can be pruned for each channel that needs to be pruned, then during the second training of the image processing network with the added pruning parameters, the values ​​of the trainable parameters corresponding to the channels and the values ​​of the pruning parameters for each trainable parameter that can be pruned for each channel that needs to be pruned are updated.

[0034] In some exemplary embodiments, the image processing network with added pruning parameters undergoes a second training process. During this second training, updating the values ​​of the trainable parameters corresponding to each channel and the values ​​of the pruning parameters corresponding to the trainable parameters of each channel includes: updating the values ​​of the trainable parameters corresponding to each channel k times based on a first loss function, with the pruning parameters as constants and the trainable parameters as variables; where k is an integer greater than or equal to 1; updating the values ​​of the pruning parameters corresponding to the trainable parameters of each channel based on a second loss function, with the trainable parameters as constants and the pruning parameters as variables; repeating the above two steps until the value of the first loss function is less than or equal to a first preset threshold.

[0035] The embodiments of this application do not limit the specific forms of the first loss function and the second loss function, but depend on the specific application scenario. For example, the first loss function can be... The second loss function can be Where θ is the trainable parameter, Y is the output of the training set, and X is the input of the training set. Let X be the result of image processing network after adding pruning parameters, and W be the pruning parameters. The norm is represented, which can be, for example, the 1-norm or the 2-norm, and β is a hyperparameter. The regularization term is the pruning parameter. For example, the regularization term can be L1 regularization.

[0036] In some exemplary embodiments, the value of k can be set based on experience, and the value of k is related to the training time and the pruning effect.

[0037] In some exemplary embodiments, determining the value of the mask parameter of the trainable parameters corresponding to the channel based on the updated value of the pruning parameter of the trainable parameters corresponding to the channel includes: determining the pruning preset value of the channel in the image processing network; and determining the value of the mask parameter of the trainable parameters corresponding to the channel based on the updated value of the pruning parameter of the trainable parameters corresponding to the channel and the pruning preset value of the channel.

[0038] In some exemplary embodiments, each channel corresponds to a pruning preset value.

[0039] In some exemplary embodiments, if certain channels do not need to be pruned in certain scenarios, then each channel that needs to be pruned corresponds to a pruning preset value.

[0040] In some exemplary embodiments, the pruning preset value corresponding to a channel is used to characterize the total number or proportion of trainable parameters that need to be pruned for each channel in the image processing network. Correspondingly, the pruning parameter is used to characterize the total network importance weight of the trainable parameters corresponding to each channel. In this case, a channel refers to either the input channel or the output channel of the image processing network.

[0041] In some exemplary embodiments, the pruning preset value corresponding to a channel is used to characterize the total number or proportion of trainable parameters that need to be pruned for each channel in the image processing network that needs to be pruned. Correspondingly, the pruning parameter is used to characterize the total network importance weight of the trainable parameters corresponding to each channel that needs to be pruned. In this case, a channel refers to an input channel or an output channel of the image processing network.

[0042] In some exemplary embodiments, the pruning preset value is used to characterize the number or proportion of trainable parameters that need to be pruned for each channel in each layer of the image processing network. Correspondingly, the pruning parameter is used to characterize the importance weight of the trainable parameters for each channel in each layer. In this case, "channel" refers to the input or output channel of a layer in the image processing network.

[0043] In some exemplary embodiments, the pruning preset value is used to characterize the number or proportion of trainable parameters that need to be pruned for each channel that needs to be pruned in each layer of the image processing network. Correspondingly, the pruning parameter is used to characterize the importance weight of the trainable parameters that can be pruned for each channel that needs to be pruned in each layer. In this case, the channel refers to the input or output channel of a certain layer of the image processing network.

[0044] In some exemplary embodiments, the pruning parameters of trainable parameters corresponding to the same channels in two linked layers have the same value. For example, if layers 1 and 2 are linked, and layer 1 has three channels (channel 1, channel 2, and channel 3) and layer 2 has two channels (channel 4 and channel 5), then the pruning parameter value of the trainable parameter corresponding to channel 1 in layer 1 is the same as the pruning parameter value of the trainable parameter corresponding to channel 4 in layer 2, and the pruning parameter value of the trainable parameter corresponding to channel 3 in layer 1 is the same as the pruning parameter value of the trainable parameter corresponding to channel 5 in layer 2. Other cases follow the same principle.

[0045] During the second training of the image processing network after adding the pruning parameters, when updating the values ​​of the pruning parameters of the trainable parameters corresponding to the channels based on the second loss function with the trainable parameters as constants and the pruning parameters as variables, the values ​​of the pruning parameters with the same values ​​should be kept consistent after the update.

[0046] In some exemplary embodiments, determining the pruning preset value corresponding to a channel in an image processing network includes: determining the connection relationship between layers in the image processing network, the computational cost and the number of trainable parameters for each layer; and determining the pruning preset value corresponding to the channel based on the connection relationship, the computational cost and the number of trainable parameters for each layer, and the target performance index.

[0047] In some exemplary embodiments, the linking relationship includes input-output matching relationships. A linking relationship exists between layers if one input channel of one layer is the same as an input channel of another layer, or if one output channel of one layer is the same as an output channel of another layer. No linking relationship exists between layers if all input channels or output channels of the two layers are different.

[0048] In some exemplary embodiments, the computational load includes the sum of multiplication operations, addition operations, and memory accesses.

[0049] In some exemplary embodiments, the target operating metrics include at least one of the following: runtime, operating effect, processing power of the operating device, storage capacity of the operating device, etc.

[0050] In some exemplary embodiments, the pruning preset value corresponding to a channel is used to characterize the total number or proportion of trainable parameters that need to be pruned for each channel in the image processing network; the pruning parameter is used to characterize the total importance weight of the trainable parameters corresponding to each channel in the network; determining the value of the mask parameter of the trainable parameter corresponding to a channel based on the updated value of the pruning parameter of the trainable parameter corresponding to the channel and the pruning preset value of the channel includes: for each channel, determining the A trainable parameters with the lowest pruning parameter values ​​corresponding to the channel as trainable parameters that need to be pruned; where A is the pruning preset value, or the product of the pruning preset value and the total number of channels of the second trained image processing network; determining the value of the mask parameter of the trainable parameter corresponding to the channel based on the trainable parameters that need to be pruned for each channel in the second trained image processing network.

[0051] In some exemplary embodiments, determining the value of the mask parameter of the trainable parameters corresponding to the channel in the image processing network that needs to be pruned based on the trainable parameters corresponding to the channel obtained in the second training includes: determining the value of the mask parameter of the trainable parameters corresponding to the channel that needs to be pruned to a first value, and determining the value of the mask parameter of the trainable parameters corresponding to the channel that needs to be retained to a second value.

[0052] In some exemplary embodiments, the pruning preset value corresponding to a channel is used to characterize the total number or proportion of trainable parameters that need to be pruned for each channel in the image processing network that needs to be pruned; the pruning parameter is used to characterize the total importance weight of the trainable parameters corresponding to each channel that needs to be pruned; determining the value of the mask parameter of the trainable parameters corresponding to a channel based on the updated value of the pruned parameters of the trainable parameters corresponding to the channel and the pruning preset value of the channel includes: for each channel that needs to be pruned, determining the A trainable parameters with the lowest updated value of the pruned parameters corresponding to the channel that needs to be pruned as the trainable parameters that need to be pruned; where A is the pruning preset value, or the product of the pruning preset value and the total number of channels in the second trained image processing network; determining the value of the mask parameter of the trainable parameters corresponding to the channel based on the trainable parameters that need to be pruned for the channels in the second trained image processing network.

[0053] In some exemplary embodiments, determining the value of the mask parameter of the trainable parameter corresponding to the channel to be truncated in the image processing network obtained by the second training includes: determining the value of the mask parameter of the trainable parameter corresponding to the channel to be truncated to be a first value, and determining the value of the mask parameter of the trainable parameter corresponding to the channel to be truncated to be retained to be a second value.

[0054] In some exemplary embodiments, the pruning preset value is used to characterize the number or proportion of trainable parameters that need to be pruned for each channel in each layer of the image processing network; the pruning parameter is used to characterize the importance weight of the trainable parameters corresponding to each channel in each layer; determining the value of the mask parameter of the trainable parameter corresponding to the channel based on the updated value of the pruned parameter of the trainable parameter corresponding to the channel and the pruning preset value of the channel includes: for each channel in each layer of the second trained image processing network, determining the A trainable parameters with the lowest updated value of the pruned parameter of the channel in the second trained image processing network as the trainable parameters that need to be pruned for the channel in the layer; where A is the pruning preset value, or the pruning preset value and the number of channels in the layer; determining the value of the mask parameter of the trainable parameter corresponding to the channel in the layer based on the trainable parameters that need to be pruned for the channels in all layers of the second trained image processing network.

[0055] In some exemplary embodiments, determining the value of the mask parameter of the trainable parameters corresponding to the channels in all layers of the image processing network obtained in the second training includes: determining the value of the mask parameter of the trainable parameters corresponding to the channels in the layer that need to be pruned as a first value, and determining the value of the mask parameter of the trainable parameters corresponding to the channels in the layer that need to be retained as a second value.

[0056] In some exemplary embodiments, the pruning preset value is used to characterize the number or proportion of trainable parameters that need to be pruned for each channel that needs to be pruned in each layer of the image processing network; the pruning parameter is used to characterize the importance weight of the trainable parameter corresponding to each channel that needs to be pruned in each layer; determining the value of the mask parameter of the trainable parameter corresponding to the channel based on the updated value of the pruning parameter and the pruning preset value of the channel includes: for each channel that needs to be pruned in each layer of the second trained image processing network, determining the A trainable parameters with the lowest pruning parameter values ​​for the channel that needs to be pruned in the layer as the trainable parameters that need to be pruned for the channel that needs to be pruned in the layer; where A is the pruning preset value, or the pruning preset value and the number of channels in the layer; determining the value of the mask parameter of the trainable parameter that needs to be pruned for the channel that needs to be pruned in the layer based on the trainable parameters that need to be pruned for the channel that needs to be pruned in all layers of the second trained image processing network.

[0057] In some exemplary embodiments, determining the mask parameter value of the pruning channel corresponding to the pruning channel in a layer based on the trainable parameters to be pruned corresponding to the pruning channels in all layers of the image processing network obtained in the second training includes: determining that the mask parameter value of the pruning trainable parameter corresponding to the pruning channel in the layer is a first value, and determining that the mask parameter value of the trainable parameter to be retained is a second value.

[0058] In some exemplary embodiments, the first value is 0 and the second value is 1; or, the first value is 1 and the second value is 0.

[0059] In some exemplary embodiments, adding mask parameters for the trainable parameters corresponding to the channels in the second trained image processing network according to the values ​​of the mask parameters for the trainable parameters corresponding to the channels after performing a second training on the image processing network with the added pruning parameters means adding the values ​​of the mask parameters for the trainable parameters corresponding to the channels obtained after a second training on the image processing network with the added pruning parameters to the second trained image processing network to obtain the second image processing network.

[0060] In some exemplary embodiments, adding mask parameters for trainable parameters corresponding to channels in the second trained image processing network means adding a mask parameter for each channel corresponding to each trainable parameter in the second trained image processing network.

[0061] In some exemplary embodiments, if in certain scenarios the trainable parameters corresponding to certain channels do not need to be pruned, or if the trainable parameters corresponding to certain channels are not allowed to be pruned, then a mask parameter can be added only to the trainable parameters corresponding to each channel that needs to be pruned. That is, each trainable parameter corresponding to each channel that needs to be pruned corresponds to a mask parameter, which can be represented as M. ij , where i represents the i-th channel that needs to be pruned, and j represents the j-th trainable parameter that can be pruned.

[0062] Step 101, as follows Figure 2 As shown, the image processing network with added pruning parameters and mask parameters is trained based on the mask parameters of the trainable parameters corresponding to the channels. During the first training process, the values ​​of the pruning parameters and the values ​​of the trainable parameters corresponding to the channels are updated.

[0063] In some exemplary embodiments, the image processing network after adding the pruning parameters and the mask parameters is trained based on the mask parameters of the trainable parameters corresponding to the channels. During the first training process, updating the values ​​of the pruning parameters and the trainable parameters corresponding to the channels includes: updating the values ​​of the pruning parameters of the trainable parameters to be pruned corresponding to the channels based on the mask parameters of the trainable parameters to be pruned; updating the values ​​of the trainable parameters to be pruned corresponding to the channels based on a first loss function, with the pruning parameters as constants and the trainable parameters as variables; updating the values ​​of the pruning parameters of the trainable parameters to be retained corresponding to the channels based on the first loss function, with the trainable parameters as constants and the pruning parameters as variables; repeating the above steps until the value of the first loss function is less than or equal to a first preset threshold.

[0064] In some exemplary embodiments, the image processing network after adding the pruning parameters and the mask parameters is trained based on the mask parameters of the trainable parameters corresponding to the channels. During the first training process, updating the values ​​of the pruning parameters and the trainable parameters corresponding to the channels includes: updating the values ​​of the pruning parameters of the trainable parameters to be pruned for each channel based on the mask parameters of the trainable parameters to be pruned; updating the values ​​of the trainable parameters to be pruned for each channel based on a first loss function, with the pruning parameters as constants and the trainable parameters as variables; updating the values ​​of the pruning parameters of the trainable parameters to be retained for each channel based on the first loss function, with the trainable parameters as constants and the pruning parameters as variables; determining the probability of the trainable parameters corresponding to each channel for each channel; updating the trainable parameter with the highest probability among the trainable parameters to be pruned for each channel as the trainable parameter to be retained; updating the trainable parameter with the highest probability among the trainable parameters to be retained for each channel as the trainable parameter to be pruned; repeating the above steps until the value of the first loss function is less than or equal to a first preset threshold.

[0065] In some exemplary embodiments, updating the value of the pruning parameter of the trainable parameter to be pruned corresponding to the channel based on the mask parameter of the trainable parameter corresponding to the channel includes: updating the value of the pruning parameter of the trainable parameter to be pruned corresponding to the channel based on a third loss function when the value of the mask parameter of the trainable parameter corresponding to the channel is a first value.

[0066] The embodiments of this application do not limit the specific form of the third loss function. For example, the third loss function can be... .

[0067] In some exemplary embodiments, updating the values ​​of the pruning parameters of the trainable parameters that need to be pruned for a channel according to the third loss function means updating the values ​​of the pruning parameters of the trainable parameters that need to be pruned for a channel so as to minimize the value of the third loss function.

[0068] In this embodiment, by updating the values ​​of the pruning parameters of the trainable parameters that need to be pruned for the corresponding channels, the output range of the image processing network can be smoothly transitioned, and the effect changes before and after pruning can be smoothly transitioned.

[0069] In some exemplary embodiments, determining the probability of a trainable parameter corresponding to a channel includes: determining the probability of a trainable parameter corresponding to a channel based on the number of steps of the first training of the image processing network after adding the pruning parameters and the mask parameters, the value of the mask parameter of the trainable parameter corresponding to the channel, and the value of the pruning parameter of the trainable parameter corresponding to the channel.

[0070] In some exemplary embodiments, according to the formula Determine the probability of the trainable parameters corresponding to the channel; where p ij Let s be the probability of the j-th trainable parameter corresponding to the i-th channel, s be the current training step number, T be the preset parameters, and W be the training parameter. ij M represents the pruning parameter value for the j-th trainable parameter corresponding to the i-th channel. ij The mask parameter is the value of the j-th trainable parameter corresponding to the i-th channel.

[0071] In this embodiment, by updating the trainable parameter with the highest probability among the trainable parameters that need to be pruned corresponding to the channel to the trainable parameters that need to be retained, and updating the trainable parameter with the highest probability among the trainable parameters that need to be retained corresponding to the channel to the trainable parameters that need to be pruned, some pruned parameters are allowed to re-participate in training during the training process, thereby further improving the training effect.

[0072] Step 102: Generate a pruned image processing network based on the image processing network obtained from the first training.

[0073] In some exemplary embodiments, generating a pruned image processing network based on a first trained image processing network includes: for each channel, discarding the trainable parameters that need to be pruned in the first trained image processing network; updating the values ​​of the trainable parameters corresponding to the channels in the first trained image processing network to the product of the values ​​of the trainable parameters corresponding to the corresponding channels in the first trained image processing network and the values ​​of the corresponding pruning parameters; and removing the pruning parameters and masking parameters in the first trained image processing network to obtain the pruned image processing network.

[0074] In some exemplary embodiments, the trainable parameters that need to be pruned in the first trained image processing network refer to the values ​​of the corresponding trainable parameters in the first trained image processing network, and the trainable parameters whose product between the pruning parameter value and the mask parameter value is 0.

[0075] The pruning method provided in this application reduces information loss during network pruning by updating the values ​​of pruning parameters and trainable parameters that represent importance weights simultaneously during network training. This makes the pruning method more suitable for image processing networks, and eliminates the need for fine-tuning the network after pruning.

[0076] To more fully illustrate the pruning method of the embodiments of this application, an example is given below for explanation. The example given is not intended to limit the scope of protection of this application.

[0077] Example like Figure 3 As shown, the pruning methods include: Step 300: Determine the connection relationships between layers in the image processing network, the computational cost of each layer, and the number of trainable parameters.

[0078] Step 301: Determine the pruning preset value corresponding to each channel based on the connection relationship, the computational cost and number of trainable parameters of each layer, and the target performance index; wherein, the pruning preset value corresponding to each channel is used to characterize the total number of trainable parameters that need to be pruned for each channel in the image processing network.

[0079] Step 302: Add a pruning parameter to each trainable parameter corresponding to each channel in the image processing network.

[0080] Step 303: Using the pruning parameter as a constant and the trainable parameter as a variable, update the value of the trainable parameter corresponding to the channel k times based on the first loss function; where k is an integer greater than or equal to 1.

[0081] Step 304: Using the trainable parameters as constants and the pruning parameters as variables, update the values ​​of the pruning parameters of the trainable parameters corresponding to the channel based on the second loss function.

[0082] Step 305: Repeat steps 303 and 304 until the value of the first loss function is less than or equal to the first preset threshold.

[0083] Step 306: For each channel, determine the A trainable parameters with the lowest updated pruning parameters corresponding to the channel as the channels that need to be pruned; where A is the preset pruning value.

[0084] Step 307: Determine the mask parameter value of the trainable parameters that need to be pruned for the corresponding channel to be 0, and determine the mask parameter value of the trainable parameters that need to be retained for the corresponding channel to be 1.

[0085] Step 308: Add the values ​​of the mask parameters of the trainable parameters corresponding to the channels to the image processing network after adding the pruning parameters.

[0086] Step 309: When the mask parameter value of the trainable parameter corresponding to the channel is 0, update the pruning parameter value of the trainable parameter that needs to be pruned according to the third loss function.

[0087] Step 310: Using the pruning parameters as constants and the trainable parameters as variables, update the values ​​of the trainable parameters that need to be pruned for each channel based on the first loss function.

[0088] Step 311: Using the trainable parameters as constants and the pruning parameters as variables, update the values ​​of the pruning parameters of the trainable parameters that need to be retained for the corresponding channels based on the first loss function.

[0089] Step 312: For each channel, determine the probability of the trainable parameter corresponding to the channel; update the trainable parameter with the highest probability among the trainable parameters that need to be pruned for the channel to the trainable parameter that needs to be retained; update the trainable parameter with the highest probability among the trainable parameters that need to be retained for the channel to the trainable parameter that needs to be pruned.

[0090] Step 313: Repeat steps 309-312 until the value of the first loss function is less than or equal to the first preset threshold.

[0091] Step 314: For each channel, discard the trainable parameters that need to be pruned in the first trained image processing network; update the values ​​of the trainable parameters corresponding to the channels in the first trained image processing network to the product of the values ​​of the trainable parameters corresponding to the channels in the first trained image processing network and the values ​​of the corresponding pruning parameters; remove the pruning parameters and mask parameters in the first trained image processing network to obtain the pruned image processing network.

[0092] Secondly, another embodiment of this application provides an image processing method, including: The pruned image processing network obtained by using any of the above pruning methods processes the image to be processed to obtain the processed image.

[0093] In some exemplary embodiments, the processing includes at least one of the following: noise reduction processing and repair processing.

[0094] The image processing method of this application uses a pruned image processing network obtained by the above-mentioned pruning method to process the image. Since the above-mentioned pruning method is more suitable for image processing networks with a large output range and a small error tolerance range, it can better preserve the image processing effect.

[0095] Thirdly, referring to Figure 4 Another embodiment of this application provides an electronic device, including: at least one processor 401; and a memory 402, on which at least one program is stored, which, when executed by at least one processor 401, implements any of the above-described pruning methods or any of the above-described image processing methods.

[0096] In some exemplary embodiments, the electronic device further includes one or more I / O interfaces 403 connected between the processor 401 and the memory 402, configured to enable information interaction between the processor 401 and the memory 402.

[0097] Among them, processor 401 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 402 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 403 is connected between processor 401 and memory 402, and can realize information interaction between processor 401 and memory 402, including but not limited to data bus (Bus).

[0098] In some embodiments, the processor 401, memory 402, and I / O interface 403 are interconnected via bus 404, and thus connected to other components of the computing device.

[0099] Fourthly, another embodiment of this application provides a computer-readable medium storing a computer program, which, when executed by a processor, implements any of the above-described pruning methods or any of the above-described image processing methods.

[0100] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0101] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. An image processing method, comprising: In an image processing network, pruning parameters and masking parameters are added to the trainable parameters corresponding to each channel. The pruning parameters of the trainable parameters corresponding to each channel are used to characterize the importance weights of the trainable parameters corresponding to each channel. The masking parameters of the trainable parameters corresponding to each channel are used to characterize whether the trainable parameters corresponding to each channel need to be pruned or retained. The image processing network after adding the pruning parameters and the mask parameters is trained based on the mask parameters of the trainable parameters corresponding to the channel. During the first training process, the values ​​of the pruning parameters and the values ​​of the trainable parameters corresponding to the channel are updated. A pruned image processing network is generated based on the image processing network obtained from the first training. The pruned image processing network is used to process the image to be processed to obtain the processed image. The processing includes at least one of the following: noise reduction processing and restoration processing. The first training of the image processing network after adding the pruning parameters and the mask parameters based on the mask parameters of the trainable parameters corresponding to the channel includes updating the values ​​of the pruning parameters and the values ​​of the trainable parameters corresponding to the channel during the first training process: The values ​​of the pruning parameters of the trainable parameters corresponding to the channel that need to be pruned are updated based on the mask parameters of the trainable parameters corresponding to the channel. Using the pruning parameters as constants and the trainable parameters as variables, the values ​​of the trainable parameters corresponding to the channel that need to be pruned are updated based on the first loss function; Using the trainable parameters as constants and the pruning parameters as variables, the values ​​of the pruning parameters of the trainable parameters that need to be retained for the corresponding channel are updated based on the first loss function. Repeat the above steps until the value of the first loss function is less than or equal to the first preset threshold; The process of repeating the above steps until the value of the first loss function is less than or equal to the first preset threshold, and the process of performing a first training on the image processing network after adding the pruning parameters and the mask parameters based on the mask parameters of the trainable parameters corresponding to the channel, further includes updating the values ​​of the pruning parameters and the values ​​of the trainable parameters corresponding to the channel during the first training process: For each of the channels, determine the probability of the trainable parameters corresponding to the channel; Update the trainable parameter with the highest probability among the trainable parameters that need to be pruned corresponding to the channel to the trainable parameter that needs to be retained; The trainable parameter with the highest probability among the trainable parameters that need to be retained for the corresponding channel is updated to the trainable parameter that needs to be pruned.

2. The image processing method according to claim 1, wherein, The addition of pruning parameters and mask parameters to the trainable parameters corresponding to channels in the image processing network includes: In the image processing network, the pruning parameters are added to the trainable parameters corresponding to the channel; The image processing network with the added pruning parameters is trained a second time, and during the second training process, the values ​​of the trainable parameters corresponding to the channel and the values ​​of the pruning parameters corresponding to the trainable parameters of the channel are updated. The value of the mask parameter corresponding to the trainable parameter of the channel is determined based on the updated value of the pruning parameter of the trainable parameter corresponding to the channel; Based on the value of the mask parameter of the trainable parameter corresponding to the channel, the mask parameter is added to the image processing network after adding the pruning parameter for the trainable parameter corresponding to the channel.

3. The image processing method according to claim 2, wherein, The second training of the image processing network after adding the pruning parameters, including updating the values ​​of the trainable parameters corresponding to the channel and the values ​​of the pruning parameters corresponding to the trainable parameters of the channel during the second training process, includes: Using the pruning parameters as constants and the trainable parameters as variables, the values ​​of the trainable parameters corresponding to the channel are updated k times based on the first loss function; where k is an integer greater than or equal to 1. Using the trainable parameters as constants and the pruning parameters as variables, the values ​​of the pruning parameters of the trainable parameters corresponding to the channel are updated based on the second loss function. Repeat the above two steps until the value of the first loss function is less than or equal to the first preset threshold.

4. The image processing method according to claim 2, wherein, The step of determining the mask parameter value of the trainable parameters corresponding to the channel based on the updated value of the pruned parameters of the trainable parameters corresponding to the channel includes: Determine the preset pruning value corresponding to the channel in the image processing network; The mask parameter value of the trainable parameter corresponding to the channel is determined based on the updated value of the pruning parameter of the trainable parameter corresponding to the channel and the preset pruning value of the channel.

5. The image processing method according to claim 4, wherein, Determining the preset pruning value corresponding to the channel in the image processing network includes: Determine the connection relationships between layers in the image processing network, the computational cost of each layer, and the number of trainable parameters; The pruning preset value corresponding to the channel is determined based on the link relationship, the amount of computation and trainable parameters of each layer, and the target operating index.

6. The image processing method according to claim 4, wherein, The pruning preset value corresponding to the channel is used to characterize the total number or proportion of trainable parameters that need to be pruned for each channel in the image processing network. The pruning parameters are used to characterize the total network importance weight of the trainable parameters corresponding to each channel; The step of determining the mask parameter value of the trainable parameters corresponding to the channel based on the updated value of the pruning parameters of the trainable parameters corresponding to the channel and the preset pruning value of the channel includes: For each channel, the A trainable parameters with the lowest updated pruning parameters corresponding to that channel are identified as trainable parameters that need to be pruned; where A is the pruning preset value, or the product of the pruning preset value and the total number of channels of the image processing network obtained by the second training. The mask parameter value of the trainable parameter corresponding to the channel is determined based on the trainable parameters that need to be pruned in the image processing network obtained by the second training.

7. The image processing method according to claim 4, wherein, The pruning preset value is used to characterize the number or proportion of trainable parameters that need to be pruned for each channel in each layer of the image processing network; the pruning parameter is used to characterize the importance weight of the trainable parameters for each channel in each layer. The step of determining the mask parameter value of the trainable parameters corresponding to the channel based on the updated value of the pruning parameters of the trainable parameters corresponding to the channel and the preset pruning value of the channel includes: For each channel in each layer of the image processing network obtained by the second training, the A trainable parameters with the lowest values ​​of the pruning parameters corresponding to the channel in the layer are determined as the trainable parameters that need to be pruned for the channel in the layer; where A is the pruning preset value, or the product of the pruning preset value and the number of trainable parameters of the channel in the layer; The mask parameter value of the trainable parameter corresponding to the channel in all layers of the image processing network obtained by the second training is determined based on the trainable parameters that need to be pruned in all layers.

8. The image processing method according to any one of claims 1-7, wherein, The step of generating the pruned image processing network based on the first trained image processing network includes: For each of the channels, the trainable parameters that need to be truncated in the first trained image processing network are discarded; The values ​​of the trainable parameters corresponding to the channels in the first trained image processing network are updated to the product of the values ​​of the trainable parameters corresponding to the channels in the first trained image processing network and the values ​​of the corresponding pruning parameters. The pruning parameters and mask parameters in the first trained image processing network are then removed to obtain the pruned image processing network.

9. An electronic device, comprising: At least one processor; A memory storing at least one program that, when executed by the at least one processor, implements the image processing method according to any one of claims 1-8.

10. A computer-readable medium storing a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-8.

Images