Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to the technical field of computer vision, and specifically provides an image processing method and device, electronic equipment and a storage medium. An image processing method comprises: performing image detection on a to-be-processed image to obtain a noise estimation map of the to-be-processed image; determining a noise residual map of the to-be-processed image according to the noise estimation map and the to-be-processed image; and determining a target image based on a difference value between the noise residual map and the to-be-processed image. The present disclosure improves the image denoising effect and improves the image quality.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image denoising refers to the process of reducing noise in an image. It can effectively enhance image information, enrich image details, and improve image quality. Among related technologies, image denoising based on deep neural networks (DNNs) has been widely used. However, the denoising effect of neural networks in these technologies is generally limited, and the improvement in image quality is only slight. Summary of the Invention

[0003] To improve image denoising performance, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide an image processing method, including:

[0005] Image detection is performed on the image to be processed to obtain a noise estimation map of the image to be processed;

[0006] Based on the noise estimation map and the image to be processed, determine the noise residual map of the image to be processed;

[0007] The target image is determined based on the difference between the noise residual map and the image to be processed.

[0008] In some implementations, the step of performing image detection on the image to be processed to obtain a noise estimation map of the image to be processed includes:

[0009] The image to be processed is input into a pre-trained noise estimation network to obtain the noise estimation map predicted by the noise estimation network.

[0010] The step of determining the noise residual map of the image to be processed based on the noise estimation map and the image to be processed includes:

[0011] The noise estimation map and the image to be processed are input into a pre-trained noise residual network to obtain the noise residual map.

[0012] In some implementations, before inputting the image to be processed into a pre-trained noise estimation network to obtain the noise estimation map predicted by the noise estimation network, the method further includes:

[0013] Perform channel conversion on the image to be processed, converting the channel order of the image to the target channel order; and / or,

[0014] The image to be processed is scaled to obtain an image of a preset scale.

[0015] In some implementations, the process of training the noise estimation network and the noise residual network includes:

[0016] Obtain a sample dataset, wherein each sample data in the sample dataset includes a sample image, a noise estimation label, and a noise residual label;

[0017] For any sample data, the sample image is input into the noise estimation network to be trained, and the noise estimation map output by the noise estimation network is obtained.

[0018] The noise estimation map and the sample image are input into the noise residual network to be trained to obtain the noise residual map output by the noise residual network.

[0019] Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, the network parameters of the noise estimation network and / or the noise residual network are adjusted until the convergence condition is met.

[0020] In some implementations, obtaining the sample dataset includes:

[0021] For any sample data, the noise residual label corresponding to the sample image is obtained based on the difference between the sample image and the sample label corresponding to the sample image;

[0022] Based on the noise residual label, the noise estimation label corresponding to the sample image is obtained.

[0023] In some implementations, obtaining the noise estimation label corresponding to the sample image based on the noise residual label includes:

[0024] The standard deviation is calculated by traversing the noise residual labels using a sliding window based on a preset scale to obtain an intermediate image;

[0025] Edge pixel padding is performed on the intermediate image to obtain the noise estimation label with the same scale as the sample image.

[0026] In some implementations, the process of determining a first difference between the noise estimation map and the noise estimation label includes:

[0027] In response to the difference between the noise estimation map and the noise estimation label being greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference;

[0028] In response to the fact that the difference between the noise estimation map and the noise estimation label is not greater than a preset threshold, the product of the difference and the second parameter is determined as the first difference; the first parameter is less than the second parameter.

[0029] In a second aspect, embodiments of this disclosure provide an image processing apparatus, including:

[0030] The image detection module is configured to perform image detection on the image to be processed to obtain a noise estimation map of the image to be processed;

[0031] The residual determination module is configured to determine the noise residual map of the image to be processed based on the noise estimation map and the image to be processed.

[0032] The image determination module is configured to determine the target image based on the difference between the noise residual map and the image to be processed.

[0033] In some implementations, the image detection module is configured to:

[0034] The image to be processed is input into a pre-trained noise estimation network to obtain the noise estimation map predicted by the noise estimation network.

[0035] In some implementations, the residual determination module is configured to:

[0036] The noise estimation map and the image to be processed are input into a pre-trained noise residual network to obtain the noise residual map.

[0037] In some implementations, the image detection module is configured to:

[0038] Perform channel conversion on the image to be processed, converting the channel order of the image to the target channel order; and / or,

[0039] The image to be processed is scaled to obtain an image of a preset scale.

[0040] In some embodiments, the apparatus described in this disclosure further includes a training module, which is configured to:

[0041] Obtain a sample dataset, wherein each sample data in the sample dataset includes a sample image, a noise estimation label, and a noise residual label;

[0042] For any sample data, the sample image is input into the noise estimation network to be trained, and the noise estimation map output by the noise estimation network is obtained.

[0043] The noise estimation map and the sample image are input into the noise residual network to be trained to obtain the noise residual map output by the noise residual network.

[0044] Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, the network parameters of the noise estimation network and / or the noise residual network are adjusted until the convergence condition is met.

[0045] In some implementations, the training module is configured to:

[0046] For any sample data, the noise residual label corresponding to the sample image is obtained based on the difference between the sample image and the sample label corresponding to the sample image;

[0047] Based on the noise residual label, the noise estimation label corresponding to the sample image is obtained.

[0048] In some implementations, the training module is configured to:

[0049] The standard deviation is calculated by traversing the noise residual labels using a sliding window based on a preset scale to obtain an intermediate image;

[0050] Edge pixel padding is performed on the intermediate image to obtain the noise estimation label with the same scale as the sample image.

[0051] In some implementations, the training module is configured to:

[0052] In response to the difference between the noise estimation map and the noise estimation label being greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference;

[0053] In response to the fact that the difference between the noise estimation map and the noise estimation label is not greater than a preset threshold, the product of the difference and the second parameter is determined as the first difference; the first parameter is less than the second parameter.

[0054] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0055] Processor; and

[0056] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.

[0057] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.

[0058] The image processing method of this disclosure includes performing image detection on an image to be processed to obtain a noise estimation map, determining a noise residual map of the image to be processed based on the noise estimation map and the processed image, and determining a target image based on the difference between the noise residual map and the image to be processed. In this disclosure, by obtaining the noise residual map jointly from the noise estimation map and the image to be processed, the original image information can be fully considered during the denoising process, alleviating the problem of losing original image feature information when extracting deep features, thereby improving the image denoising effect. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0060] Figure 1 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0061] Figure 2 This is a schematic diagram of the structure of an image processing network according to some embodiments of the present disclosure.

[0062] Figure 3 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0063] Figure 4 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0064] Figure 5 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0065] Figure 6 These are comparison diagrams showing the effects of image processing methods according to some embodiments of this disclosure.

[0066] Figure 7 This is a structural block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0067] Figure 8 This is a structural block diagram of an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0068] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0069] Currently, image denoising techniques based on deep neural networks (DNNs) are widely used in various scenarios. In these techniques, a pre-trained denoising network is typically used to estimate noise in the input image, obtaining a noise residual map. Then, the original input image is subtracted from the noise residual map to obtain a high-quality denoised image. However, the inventors in this study found that in related techniques, the denoised images obtained using DNN networks retain a significant amount of noise and have poor image details, making it difficult to further optimize and improve the denoising effect.

[0070] In view of the deficiencies existing in the above-mentioned related technologies, the present disclosure provides an image processing method, apparatus, electronic device and storage medium, which aims to improve the image denoising effect and improve image quality.

[0071] In a first aspect, embodiments of this disclosure provide an image processing method that can be applied to an electronic device. The electronic device described in this disclosure can be any suitable type of device, such as a smartphone, tablet computer, wearable device, in-vehicle device, mobile computer, server, cloud platform, etc., and this disclosure does not impose any limitations on it.

[0072] like Figure 1 As shown, in some embodiments, the image processing method of this disclosure includes:

[0073] S110. Perform image detection on the image to be processed to obtain a noise estimation map of the image to be processed.

[0074] In this embodiment of the disclosure, the image to be processed refers to the original image to which denoising processing is desired, that is, the image to be processed is the original image containing noise. The purpose is to use the method of this embodiment of the disclosure to denoise the image to be processed, thereby obtaining the denoised target image.

[0075] In some embodiments, the image to be processed can be a scene image captured by an electronic device through an image acquisition device. For example, in the case of a smartphone, the image to be processed can be a natural scene image captured by the phone's camera. In other embodiments, the image to be processed can be an image obtained by the electronic device through any other means, such as images obtained through network download, external device transmission, etc. In the embodiments of this disclosure, the image to be processed can be a single image or a frame image in a video stream, and this disclosure does not impose any limitations on this.

[0076] In this embodiment of the disclosure, instead of directly performing noise prediction based on the image to be processed to obtain the noise residual map, image detection is first performed on the image to be processed to obtain the noise estimation map.

[0077] It is worth noting that related technologies often rely on image detection techniques to directly detect the image to be processed, thereby predicting the noise residual map of the image to be processed. The inventors of this case discovered through research that directly predicting and regressing the noise residual map based on the image to be processed leads to the loss of a large amount of original image details during feature extraction, resulting in a noise residual map that fails to meet expectations and ultimately produces a poor denoising effect.

[0078] Therefore, in this embodiment, a regression noise estimation map can first be predicted based on the shallow features of the image to be processed. The noise estimation map can retain a large amount of prior information related to the content of the image to be processed. In this way, by combining the noise estimation map with the original image to be processed to jointly predict the regression noise residual map, the deep network can fully learn the features of the original image and obtain a noise residual map with better performance. The steps of this disclosure will be described below and will not be detailed here.

[0079] In some implementations, an image processing network can be pre-established based on a DNN. The image to be processed is input into the noise estimation network in the pre-trained image processing network to obtain a noise estimation map predicted by the noise estimation network. The following implementations will be described in detail, but will not be elaborated upon here.

[0080] In some embodiments, the image to be processed can be preprocessed before being input into the noise estimation network, such as converting image channels or scaling the image, to obtain an input image that meets the network requirements. The embodiments described below will be further elaborated upon, and will not be detailed here.

[0081] S120. Based on the noise estimation map and the image to be processed, determine the noise residual map of the image to be processed.

[0082] In this embodiment of the disclosure, after obtaining the noise estimation map of the image to be processed, as mentioned above, the noise estimation map is a noise map obtained from the shallow layer of the image to be processed, so that a large amount of prior information related to the original content of the image to be processed can be retained in the noise estimation map.

[0083] When regressing the noise residual map, the noise residual map of the image to be processed can be predicted jointly based on the noise estimation map and the original image to be processed. The noise estimation map can be understood as a shallow feature map for noise prediction of the image to be processed. Superimposing the noise estimation map with the image to be processed can effectively alleviate the problem of losing the original feature information of the image when extracting deep features, thereby improving the effect of the noise residual map.

[0084] In some implementations, an image processing network can be pre-established based on a DNN. The obtained noise estimation map and the image to be processed are jointly input into the noise residual network in the pre-trained image processing network to obtain the noise residual map predicted by the noise residual network. The following implementations will describe the embodiments in detail, and will not be elaborated here.

[0085] S130. Determine the target image based on the difference between the noise residual map and the image to be processed.

[0086] As can be understood, the noise residual map represents the noise in the image to be processed, while the original image to be processed represents the scene image containing noise. Therefore, the final denoised target image can be obtained based on the difference between the image to be processed and the noise residual map. In one example, the target image can be obtained by subtracting the noise residual map from the image to be processed pixel by pixel.

[0087] In some embodiments, because the image to be processed is preprocessed before denoising, the target image can be restored to its original scale or pixel channel order after obtaining the target image, thus obtaining the final target image. The embodiments described below will not be detailed here.

[0088] As can be seen from the above, in this embodiment of the present disclosure, the noise residual map is obtained by combining the noise estimation map and the image to be processed, which can fully consider the original image information during the denoising process, alleviate the problem of losing the original feature information of the image when extracting deep features, and thus improve the image denoising effect.

[0089] In some implementations, the image processing method exemplified in this disclosure may be implemented using a DNN network. Figure 2 The structure of the image processing network in some embodiments of this disclosure is shown below. Figure 2 Please provide an explanation.

[0090] like Figure 2As shown, in some embodiments, the image processing network of this disclosure example may include a noise estimation network 100 and a noise residual network 200.

[0091] The noise estimation network 100 takes the image to be processed as input and outputs a noise estimation map corresponding to the image to be processed. It can be understood that in this embodiment, the noise estimation network 100 is mainly used to extract shallow features; therefore, the noise estimation network 100 can be configured with a structure that has a relatively low network depth.

[0092] For example, in one instance, the noise estimation network 100 may include a shallow neural network consisting of two 3x3 convolutional layers. Taking a 4-channel image as an example, the image is raised from 4 channels to 16 channels after passing through the first convolutional layer, then passed through a non-linear activation layer (ReLU) before being input into the second convolutional layer, and finally output as a noise estimation map.

[0093] The input to the noise residual network 200 is the original image to be processed and a noise estimation map, and the output is the noise residual map corresponding to the image to be processed. It can be understood that in this embodiment of the disclosure, the noise residual network 200 is used to extract deep features, so the noise residual network 200 can be set to a structure with more network layers.

[0094] For example, in one instance, the noise residual network 200 may include a deep neural network consisting of five 3x3 convolutional layers. After obtaining the noise estimation map, the noise estimation map can be concatenated with the original image to be processed and then input into the noise residual network 200. It will then pass through the first four convolutional layers for deep feature extraction, and after passing through the non-linear activation layer ReLU, it will be input into the last convolutional layer, and then the noise residual map will be output.

[0095] After obtaining the noise residual map, the noise residual map can be subtracted from the original image to be processed to obtain the final denoised image, which is the target image described in this disclosure.

[0096] As can be seen from the above, in this embodiment of the present disclosure, the noise residual network obtains a noise residual map based on the image to be processed and the noise estimation map, which alleviates the problem of losing the original feature information of the image when extracting deep features, thereby improving the image denoising effect.

[0097] The image processing network described above can be deployed in any type of electronic device. Taking a smartphone as an example, when the image processing network is deployed in a smartphone, the image to be processed can be a natural scene image captured in real time by the smartphone's camera module. The captured natural scene image is denoised through the above process, thereby obtaining and outputting a clearer target image to the user, thus improving the mobile phone's imaging effect.

[0098] However, different camera modules use different Bayer arrays, resulting in different pixel channel orders in the acquired images. In order to improve the universality of the image processing network disclosed herein, the image to be processed can be pre-converted to a uniform pixel channel order before inputting it into the image processing network.

[0099] A Bayer array refers to the pixel arrangement of the image sensor in a camera module. For example, an image taken by a certain brand of mobile phone might have an RGGB image channel arrangement; another example is an image taken by a certain brand of mobile phone with a GRBG image channel arrangement; and so on.

[0100] In one example of this disclosure, after acquiring the image to be processed, channel conversion can be performed on the image. For example, the aforementioned RGGB and GRBG arrangements can be uniformly converted to RGBG arrangements, that is, the channel order can be uniformly converted to the target channel order. In this way, the image processing network of this embodiment can be deployed in devices from any manufacturer, thus having higher versatility.

[0101] Furthermore, during the preprocessing of the image to be processed, the image can be scaled to uniformly scale images of different sizes into an input image that meets the requirements of the image processing network. Those skilled in the art will understand and fully implement this, and this disclosure will not elaborate further.

[0102] In one example, assuming the original image to be processed is a 1000 pixel * 1000 pixel image in RGGB arrangement, after the preprocessing described in this embodiment, a 500 pixel * 500 pixel * 4 image matrix is ​​obtained, where each layer of the matrix represents the R, G, B, G color channels in sequence. That is, the image size is scaled down to 500 pixels * 500 pixels, and the Bell arrangement is converted to RGBG arrangement.

[0103] As can be seen from the above, by preprocessing the image to be processed in this embodiment, the image processing network can be applied to various electronic devices, which is conducive to the deployment of the image processing network.

[0104] based on Figure 2 The structure of the image processing network shown is illustrated below. The network training process of the image processing network described in this disclosure will be explained below.

[0105] like Figure 3 As shown, in some embodiments, the training process of the image processing network of this disclosure example may include:

[0106] S310. Obtain the sample dataset.

[0107] First, the sample dataset required for training can be obtained. The sample dataset includes multiple sample data, where each sample data may include a sample image, the sample label corresponding to the sample image, the noise estimation label, and the noise residual label.

[0108] Taking a sample dataset as an example, the sample image represents an input image I containing noise. input The sample label represents the ground truth (GT, GroundThrough) image I after denoising the sample labels. gt The noise estimation label represents the ground truth image I corresponding to the noise estimation map. esgt The noise residual label represents the ground truth image I corresponding to the noise residual map. diff .

[0109] In some implementations, the sample dataset can be an existing, well-known dataset, SIDD (Smartphone Image Denoising Dataset), which contains tens of thousands of noisy images and their corresponding noise-free images. Since the noise-free images in the SIDD dataset are obtained by smoothing hundreds of noisy images of the same scene, the image quality of the noise-free images is very high, without issues such as subpixel shift. Using this as a training sample set can improve the network's denoising performance.

[0110] For a single sample data point in SIDD, the noise map is the sample image I. input The noise-free image is the sample label I. gt In this embodiment, it is also necessary to obtain the noise residual label I of the sample image. diff and noise estimation label I esgt The following is combined with Figure 4 The implementation method describes the calculation process.

[0111] like Figure 4 As shown, in some embodiments, the training process of the image processing network of this disclosure example, obtaining the sample dataset includes:

[0112] S410. For any sample data, obtain the noise residual label corresponding to the sample image based on the difference between the sample image and the sample label corresponding to the sample image.

[0113] S420. Based on the noise residual label, obtain the noise estimation label corresponding to the sample image.

[0114] In some implementations, taking a sample from the SIDD dataset as an example, the sample data includes sample image I. input and sample label I gt .

[0115] First, it can be obtained through sample image I input and sample label I gt The difference value is used to obtain the noise residual label I diff It is understandable that the sample image I... input This represents an image containing noise, while the sample label I... gt Indicates the sample image I input The noise-free image after denoising, thus making the sample image I input and sample label I gt The noise residual label I is obtained by subtracting and taking the absolute value. diff This refers to the true image representing the noise residual.

[0116] Secondly, after obtaining the noise residual label I diff Next, it is necessary to base the noise residual label I diff Determine the noise estimation label I corresponding to the noise estimation map. esgt In some implementations, the standard deviation can be calculated based on a sliding window, according to the noise residual label I. diff The noise estimation label I is calculated. esgt The following is combined with Figure 5 The implementation method is described below.

[0117] like Figure 5 As shown, in some embodiments, the process of obtaining noise estimation labels during the training of the image processing network of this disclosure includes:

[0118] S421. Solve the standard deviation of the noise residual labels by sliding window traversal based on the preset scale to obtain the intermediate image.

[0119] S422. Perform edge pixel padding on the intermediate image to obtain noise estimation labels with the same scale as the sample images.

[0120] In this embodiment, sliding windows of different sizes can be pre-set according to the image scale, thereby selecting an appropriate sliding window for noise residual labels of different scales. For example, in one example, for an image with noise residual labels of 500 pixels * 500 pixels, the sliding window can be selected to be 15 pixels * 15 pixels in size.

[0121] In this example, the sliding window traverses the entire noise residual label with a step size of 1. When the sliding window moves to a certain position, within the 15-pixel * 15-pixel sliding window range, the standard deviation of a total of 225 pixels within the sliding window range is calculated based on each pixel value, and the calculated standard deviation is used as the pixel value of the center point of the sliding window region.

[0122] After traversing the entire 500 pixel * 500 pixel noise residual label, an intermediate image with a length and width of 492 pixels * 492 pixels is obtained. This is because the sliding window itself has a size of 15 pixels * 15 pixels, so the obtained intermediate image will have (15 + 1) / 2 = 8 pixels less edge in both length and width directions than the noise residual label. Therefore, in this embodiment, pixel padding can be performed on the edges of the intermediate image, for example, by zero-padding, to fill in the missing pixel values ​​and obtain a noise estimation label I with the same scale as the noise residual label. esgt .

[0123] It is worth noting that, for the specific calculation of noise estimation label I esgt The method of calculating the standard deviation is not limited to the above-mentioned method. Any other suitable calculation method can be used, such as using gradient operators. This disclosure does not limit this.

[0124] The above describes the process of obtaining a single sample dataset. For each sample in the SIDD dataset, this process is repeated sequentially to obtain multiple sample datasets, which constitute the sample dataset described in this disclosure. In one example of this disclosure, approximately 200,000 training sample datasets are prepared during the training phase.

[0125] Furthermore, it is worth noting that the images in the SIDD dataset were taken by different devices, therefore the Bell array order and image scale may vary. Therefore, in some implementations, the aforementioned image preprocessing method can also be used to convert the Bell array of each sample data into the target channel order, and / or scale the image to a preset scale. Those skilled in the art will undoubtedly understand and fully implement this based on the above description, and this disclosure will not elaborate further.

[0126] S320. For any sample data, input the sample image into the noise estimation network to be trained to obtain the noise estimation map output by the noise estimation network.

[0127] S330. Input the noise estimation map and sample image into the noise residual network to be trained to obtain the noise residual map output by the noise residual network.

[0128] In this embodiment of the disclosure, network training can begin after the data preparation work in S310 is completed. In some embodiments, four sample data points can be randomly selected from the sample dataset as a batch and input into the image processing network for training. The following describes the one-iteration training process of the image processing network using one sample data point as an example.

[0129] Combination Figure 2The network structure shown, for a given sample data, includes the sample image I. input The noise estimation graph I is obtained by inputting the noise estimation network 100 to be trained. pred Then, the sample image I input And noise estimation diagram I pred By inputting the noise residual network to be trained, the noise residual map I output by the noise residual network 200 can be obtained. resi .

[0130] For example, in one sample image I input An input image of size 512 pixels * 512 pixels * 4 channels is processed by the first convolutional layer of a noise estimation network 100 to extract features, resulting in an intermediate feature map of size 512 pixels * 512 pixels * 16 channels. After ReLU activation, this intermediate feature map is input to the second convolutional layer, outputting a noise estimation map of size 512 pixels * 512 pixels * 4 channels. pred .

[0131] Then the noise estimation diagram I pred With sample image I input An image sequence concatenated to 512 pixels * 512 pixels * 8 channels is input into the noise residual network 200 to be trained. The first four convolutional layers of the noise residual network 200 extract features sequentially, with the number of channels in the intermediate feature maps changing sequentially to 8, 16, 32, 64, and 32. Finally, the fifth convolutional layer outputs a noise residual map I of 512 pixels * 512 pixels * 4 channels. resi .

[0132] S340. Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, adjust the network parameters of the noise estimation network and / or the noise residual network until the convergence condition is met.

[0133] It is understandable that the noise estimation diagram I... pred This represents the predicted output of the noise estimation network 100, while the noise estimation label I... esgt This represents the true value of the noise estimation graph; therefore, the first difference between the two represents the difference between the prediction result of the noise estimation network 100 and the true result.

[0134] Similarly, noise residual plot I resi This represents the prediction output of the noisy residual network 200, while the noise residual label I... diff This represents the true value of the noise residual map; therefore, the second difference between the two represents the difference between the prediction result of the noise residual network 200 and the true result.

[0135] Therefore, when training the image processing network in this embodiment, the training loss term can include the first difference and the second difference mentioned above. Based on the first difference and the second difference, backpropagation is performed to optimize and adjust the network parameters of the noise estimation network 100 and / or the noise residual network 200.

[0136] The above describes the training process for a single sample data. During network training, this process is repeated sequentially for each sample data. By continuously optimizing and adjusting the network parameters until the convergence condition is met, network training can be stopped, resulting in a trained image processing network. For any aspects of the network training not detailed above, those skilled in the art can understand and fully implement them by referring to supervised training processes in related technologies; this disclosure will not elaborate further.

[0137] It is worth noting that the inventors in this case discovered through research that, during the network testing phase, if the noise estimation diagram I... pred The value is higher than the noise estimation label I esgt The final target image obtained is relatively better; conversely, if the noise estimation image I is poor, the result will be worse. pred The value is lower than the noise estimation label I esgt If this happens, the final target image will be relatively worse. That is, the noise estimation image I... pred The value is higher than the noise estimation label I esgt Compared to noise estimation diagram I pred The value is lower than the noise estimation label I esgt This results in better noise reduction. Therefore, in some embodiments of this disclosure, during the training process of the image processing network, noise estimation image I is targeted. pred and noise estimation label I esgt The first difference is set as the segmented loss.

[0138] Specifically, in this embodiment of the disclosure, noise estimation map I is determined. pred and noise estimation label I esgt The process of the first difference includes:

[0139] If the difference between the noise estimation map and the noise estimation label is greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference; if the difference between the noise estimation map and the noise estimation label is not greater than the preset threshold, the product of the difference and the second parameter is determined as the first difference, wherein the first parameter is less than the second parameter.

[0140] It is understood that in the embodiments of this disclosure, the preset threshold represents the noise estimation graph I. pred The value is greater than the noise estimation label I esgt The preset value, for example, can be set to a preset threshold of 0, so that when the noise estimation diagram I...pred With noise estimation label I esgt When the difference is greater than a preset threshold, it indicates noise estimation (Figure I). pred The value is greater than the noise estimation label I esgt Conversely, the opposite is also true.

[0141] Based on the foregoing, when the noise estimation diagram I... pred The value is greater than the noise estimation label I esgt When the noise estimation graph I is at its lowest, the network's denoising effect will be better, thus allowing for a lower penalty to be applied to the first difference. Conversely, when the noise estimation graph I is at its lowest, the network's denoising effect will be better. pred The value is not greater than the noise estimation label I esgt At this time, the network's noise reduction effect will deteriorate, thus allowing for a higher penalty to be applied to the first difference. Specifically, this can be expressed as:

[0142]

[0143] In equation (1) above, L represents the first difference, α represents the first parameter, and (1-α) represents the second parameter. α can be a number between 0 and 0.5. For example, in one example, α can be 0.3.

[0144] Combining equation (1), it can be seen that when the noise estimation diagram I... pred The value is greater than the noise estimation label I esgt When the first parameter α is small, the penalty for the first difference is low. Conversely, when the noise estimation graph I... pred The value is not greater than the noise estimation label I esgt When the value of the second parameter (1-α) is large, the penalty for the first difference is higher. This allows the network to learn better noise estimation map features, thus improving image denoising performance.

[0145] As can be seen from the above, in this embodiment of the disclosure, for noise estimation diagram I pred and noise estimation label I esgt The first difference is set as a segmented loss, which allows the network to learn better noise estimation map features as much as possible, further improving the network's denoising effect.

[0146] After the image processing network is trained, it can be used for image denoising. In one example scenario, the trained image processing network can be deployed in a smartphone. When the smartphone captures the image to be processed through its camera module, the image to be processed can be input into the image processing network based on the aforementioned process to obtain the denoised target image.

[0147] Figure 6 A set of comparison images is shown; see [link / reference]. Figure 6 As shown, Figure 6In the image (a), the original image without any noise reduction processing is shown, which is the image to be processed. As you can see, the image contains a lot of noise and the image details are very poor. Figure 6 In the image (b), the denoised image obtained by traditional DNN denoising is shown. As can be seen, a certain degree of noise is still retained in the image, and the image details are also very poor. Figure 6 In (c), the target image is obtained using the image processing network of the present disclosure. It can be seen that compared with the image in (b), a large amount of noise has been removed, and the image details are richer and the image quality is higher.

[0148] As can be seen from the above, in this embodiment of the present disclosure, the noise residual map is obtained by combining the noise estimation map and the image to be processed, which can fully consider the original image information during the denoising process, alleviate the problem of losing the original feature information of the image when extracting deep features, and thus improve the image denoising effect.

[0149] Secondly, embodiments of this disclosure provide an image processing apparatus that can be applied to an electronic device. The electronic device described in this disclosure can be any suitable type of device, such as a smartphone, tablet computer, wearable device, in-vehicle device, mobile computer, server, cloud platform, etc., and this disclosure does not impose any limitations on it.

[0150] like Figure 7 As shown, in some embodiments, the image processing apparatus of this disclosure includes:

[0151] The image detection module 10 is configured to perform image detection on the image to be processed to obtain a noise estimation map of the image to be processed;

[0152] The residual determination module 20 is configured to determine the noise residual map of the image to be processed based on the noise estimation map and the image to be processed.

[0153] The image determination module 30 is configured to determine the target image based on the difference between the noise residual map and the image to be processed.

[0154] As can be seen from the above, in this embodiment of the present disclosure, the noise residual map is obtained by combining the noise estimation map and the image to be processed, which can fully consider the original image information during the denoising process, alleviate the problem of losing the original feature information of the image when extracting deep features, and thus improve the image denoising effect.

[0155] In some embodiments, the image detection module 10 is configured to:

[0156] The image to be processed is input into a pre-trained noise estimation network to obtain the noise estimation map predicted by the noise estimation network.

[0157] In some implementations, the residual determination module 20 is configured to:

[0158] The noise estimation map and the image to be processed are input into a pre-trained noise residual network to obtain the noise residual map.

[0159] In some embodiments, the image detection module 10 is configured to:

[0160] Perform channel conversion on the image to be processed, converting the channel order of the image to the target channel order; and / or,

[0161] The image to be processed is scaled to obtain an image of a preset scale.

[0162] In some embodiments, the apparatus described in this disclosure further includes a training module, which is configured to:

[0163] Obtain a sample dataset, wherein each sample data in the sample dataset includes a sample image, a noise estimation label, and a noise residual label;

[0164] For any sample data, the sample image is input into the noise estimation network to be trained, and the noise estimation map output by the noise estimation network is obtained.

[0165] The noise estimation map and the sample image are input into the noise residual network to be trained to obtain the noise residual map output by the noise residual network.

[0166] Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, the network parameters of the noise estimation network and / or the noise residual network are adjusted until the convergence condition is met.

[0167] In some implementations, the training module is configured to:

[0168] For any sample data, the noise residual label corresponding to the sample image is obtained based on the difference between the sample image and the sample label corresponding to the sample image;

[0169] Based on the noise residual label, the noise estimation label corresponding to the sample image is obtained.

[0170] In some implementations, the training module is configured to:

[0171] The standard deviation is calculated by traversing the noise residual labels using a sliding window based on a preset scale to obtain an intermediate image;

[0172] Edge pixel padding is performed on the intermediate image to obtain the noise estimation label with the same scale as the sample image.

[0173] In some implementations, the training module is configured to:

[0174] In response to the difference between the noise estimation map and the noise estimation label being greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference;

[0175] In response to the fact that the difference between the noise estimation map and the noise estimation label is not greater than a preset threshold, the product of the difference and the second parameter is determined as the first difference; the first parameter is less than the second parameter.

[0176] As can be seen from the above, in this embodiment of the present disclosure, the noise residual map is obtained by combining the noise estimation map and the image to be processed, which can fully consider the original image information during the denoising process, alleviate the problem of losing the original feature information of the image when extracting deep features, and thus improve the image denoising effect.

[0177] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0178] Processor; and

[0179] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.

[0180] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.

[0181] Specifically, Figure 8 A schematic diagram of the structure of an electronic device 600 suitable for implementing the method of this disclosure is shown. Figure 8 The electronic device shown can perform the corresponding functions of the processor and storage medium described above.

[0182] like Figure 8 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded into the memory 602 from a storage section 608. The memory 602 also stores various programs and data required for the operation of the electronic device 600. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0183] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0184] In particular, according to embodiments of this disclosure, the above-described method process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this disclosure.

Claims

1. An image processing method, characterized in that, include: The image to be processed is input into a pre-trained noise estimation network to obtain a noise estimation map predicted by the noise estimation network. The noise estimation map and the image to be processed are input into a pre-trained noise residual network to obtain the noise residual map; The target image is determined based on the difference between the noise residual map and the image to be processed; The process of training the noise estimation network and the noise residual network includes: Obtain a sample dataset, wherein each sample data in the sample dataset includes a sample image, a noise estimation label, and a noise residual label; For any sample data, the sample image is input into the noise estimation network to be trained, and the noise estimation map output by the noise estimation network is obtained. The noise estimation map and the sample image are input into the noise residual network to be trained to obtain the noise residual map output by the noise residual network. Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, the network parameters of the noise estimation network and / or the noise residual network are adjusted until the convergence condition is met. The process of determining a first difference between the noise estimation map and the noise estimation label includes: In response to the difference between the noise estimation map and the noise estimation label being greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference; In response to the fact that the difference between the noise estimation map and the noise estimation label is not greater than a preset threshold, the product of the difference and the second parameter is determined as the first difference; the first parameter is less than the second parameter.

2. The method according to claim 1, characterized in that, Before inputting the image to be processed into a pre-trained noise estimation network to obtain the noise estimation map predicted by the noise estimation network, the method further includes: Perform channel conversion on the image to be processed, converting the channel order of the image to the target channel order; and / or, The image to be processed is scaled to obtain an image of a preset scale.

3. The method according to claim 1, characterized in that, The obtained sample dataset includes: For any sample data, the noise residual label corresponding to the sample image is obtained based on the difference between the sample image and the sample label corresponding to the sample image; Based on the noise residual label, the noise estimation label corresponding to the sample image is obtained.

4. The method according to claim 3, characterized in that, The step of obtaining the noise estimation label corresponding to the sample image based on the noise residual label includes: The standard deviation is calculated by traversing the noise residual labels using a sliding window based on a preset scale to obtain an intermediate image; Edge pixel padding is performed on the intermediate image to obtain the noise estimation label with the same scale as the sample image.

5. An image processing apparatus, characterized in that, include: The image detection module is configured to input the image to be processed into a pre-trained noise estimation network to obtain a noise estimation map predicted by the noise estimation network. The residual determination module is configured to input the noise estimation map and the image to be processed into a pre-trained noise residual network to obtain the noise residual map; The image determination module is configured to determine the target image based on the difference between the noise residual map and the image to be processed; The training module is configured as follows: Obtain a sample dataset, wherein each sample data in the sample dataset includes a sample image, a noise estimation label, and a noise residual label; For any sample data, the sample image is input into the noise estimation network to be trained, and the noise estimation map output by the noise estimation network is obtained. The noise estimation map and the sample image are input into the noise residual network to be trained to obtain the noise residual map output by the noise residual network. Based on the first difference between the noise estimation map and the noise estimation label, and the second difference between the noise residual map and the noise residual label, the network parameters of the noise estimation network and / or the noise residual network are adjusted until the convergence condition is met; The training module is configured as follows: In response to the difference between the noise estimation map and the noise estimation label being greater than a preset threshold, the product of the difference and the first parameter is determined as the first difference; In response to the fact that the difference between the noise estimation map and the noise estimation label is not greater than a preset threshold, the product of the difference and the second parameter is determined as the first difference; The first parameter is less than the second parameter.

6. An electronic device, characterized in that, include: processor; as well as The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any one of claims 1 to 4.

7. A storage medium, characterized in that, Used to store computer-readable instructions for causing a computer to perform the method according to any one of claims 1 to 4.

Images