Image processing method and device for decoupling denoising and enhancing details

Abstract

The application relates to the field of image processing and discloses an image processing method and device for decoupling noise removal and detail enhancement. The method inputs an image to be processed into a trained image processing model, so that a backbone network of the image processing model performs noise removal on the image to be processed by increasing a response value and reducing a pixel value, obtains a denoised image, increases pixel values and gradients of textures and deformation edges in the denoised image, obtains a second feature map after detail enhancement, and then obtains an image after noise removal and detail enhancement of the image to be processed through a main branch of the image processing model. The application can decouple the two abilities of noise removal and detail enhancement, respectively performs noise removal and detail enhancement on a low-resolution image, and enhances the noise removal effect and the detail enhancement effect of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an image processing method and apparatus for decoupling noise reduction and enhancing details. Background Technology

[0002] Current image super-resolution methods mainly use convolutional neural network models. These models take low-resolution images as input and output high-resolution images. During model training, the low-resolution images are degraded by introducing noise (such as JPEG compression, additive white Gaussian noise, blurring, etc.) so that the model can learn to both reduce noise and improve resolution.

[0003] Although noise and detail signals have their own pattern distributions, both belong to high-frequency information, so their distributions overlap. Current methods teach the model to simultaneously learn the ability to denoise and enhance detail clarity. Denoising requires weakening the signal, while enhancing detail requires strengthening it. Therefore, learning these two abilities simultaneously can be antagonistic, often failing to effectively remove noise and enhance detail at the same time. The output either has strong denoising capabilities, resulting in excessive loss of detail and poor detail enhancement leading to blurry image quality; or weak denoising capabilities, preserving some noise, while strong detail enhancement amplifies the remaining noise, creating artifacts. Summary of the Invention

[0004] This invention provides an image processing method and apparatus that decouples noise reduction and detail enhancement capabilities, thereby improving both noise removal rate and detail enhancement to improve image clarity during image processing.

[0005] To address the aforementioned technical problems, a first embodiment of the present invention provides an image processing method for decoupling noise reduction and enhancing details, comprising:

[0006] The image to be processed is input into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to obtain a first feature map after denoising. The pixel value and gradient of the texture and deformed edge in the first feature map are increased to obtain a second feature map after enhancing the details. Then, the main branch of the image processing model processes the second feature map to obtain the image after denoising and detail enhancement.

[0007] This invention inputs the image to be processed into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to obtain a denoised image. It also increases the pixel value and gradient of the texture and deformed edges in the denoised image to obtain a second feature map with enhanced details. Then, through the main branch of the image processing model, the image to be processed after denoising and detail enhancement is obtained, realizing the decoupling of denoising and detail enhancement capabilities. It performs denoising and detail enhancement processing on low-resolution images respectively, thereby enhancing the denoising effect and the detail enhancement effect of the image.

[0008] Furthermore, the trained image processing model is specifically as follows:

[0009] Construct an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches;

[0010] The training image is input into the image processing training model;

[0011] The training images are denoised using an optimization algorithm, and the auxiliary branch is trained to output the denoised first target image.

[0012] An optimization algorithm is used to enhance the details of the first target image, and the main branch is trained to output the enhanced second target image.

[0013] Once the model training is complete, the parameters of the image processing training model are determined, auxiliary branches are removed, and the trained image processing model is formed.

[0014] This invention constructs an image processing model for image processing. The model includes not only a backbone network and a main branch, but also an auxiliary branch. The model is trained using an optimization algorithm. The auxiliary branch is trained to output the denoised image, while the main branch is trained to output the image with enhanced details. After the model training is completed, the auxiliary branch needs to be removed when using the model. By using the optimization algorithm to train different parts of the model separately, the training results of the model in terms of denoising and detail enhancement can be better reflected, thereby enhancing the training effect of the model.

[0015] Furthermore, the algorithm is optimized as follows:

[0016] A loss function is used as the objective function of the optimization algorithm. The loss function consists of two parts, which guide the auxiliary branch and the main branch to train the image processing model, respectively.

[0017] The expression for the loss function is:

[0018]

[0019] in, DN is the loss function; DN is the first training target image output by the auxiliary branch; HR is the second training target image output by the main branch; GT dn It is an image of the same size as the training image, but with noise removed; GT hr It is an image with a higher resolution than the training image and no noise; the function l1 represents the norm; λ is the weight coefficient.

[0020] Furthermore, the construction of the image processing training model specifically involves:

[0021] A deep neural network model is used as the image processing training model; wherein, the image processing training model includes a backbone network, a main branch, and an auxiliary branch;

[0022] The backbone network adopts the ESRNet structure, which consists of an upsampling layer and an downsampling layer.

[0023] The auxiliary branch is used to output the image after the upsampled layer of the backbone network;

[0024] The main branch is used to output the image after passing through the downsampled layer of the backbone network.

[0025] The image processing training model of this invention employs a deep neural network model, consisting of a backbone network, auxiliary branches, and a main branch. The backbone network adopts an ESRNet structure, comprising upsampling and downsampling layers. The auxiliary and main branches are used to output the images processed by the upsampling and downsampling layers of the backbone network, respectively. This invention outputs denoised images and enhanced detail images separately, providing better feedback on the training results of the image processing training model in both denoising and detail enhancement capabilities, thereby enhancing the training effect of the image processing training model.

[0026] Further, the step of having the main branch of the image processing model process the second feature map to obtain the image after denoising and detail enhancement is specifically as follows:

[0027] The second feature map is mapped to the color space through the main branch of the image processing model, resulting in an image of the image to be processed after denoising and detail enhancement.

[0028] When using a trained image processing model to output a high-resolution image after denoising and detail enhancement, this invention maps the second feature map to the color space through the main branch of the image processing model, outputting a noise-free and detail-enhanced high-resolution image.

[0029] Furthermore, the noise in the image to be processed includes: grid artifacts, ringing phenomena, and additive Gaussian noise introduced during the recording and propagation of the image; wherein, the noise appears uniformly across the entire frame of the image to be processed or appears at the texture edges of the image to be processed.

[0030] This invention provides an image processing method that decouples denoising and detail enhancement. The method inputs the image to be processed into a trained image processing model, where the backbone network of the model increases the response value and decreases the pixel value of the noise in the image to obtain a denoised image. The method also increases the pixel values ​​and gradients of textures and deformed edges in the denoised image to obtain a second feature map with enhanced details. Then, through the main branch of the image processing model, the image after denoising and detail enhancement is obtained. This method achieves decoupled denoising and detail enhancement capabilities, performing denoising and detail enhancement on low-resolution images respectively, thus improving both the denoising and detail enhancement effects.

[0031] Accordingly, the present invention provides an image processing apparatus for decoupling noise reduction and enhancing details, comprising: a processing module;

[0032] The processing module is used to input the image to be processed into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to be processed, thereby obtaining a first feature map after denoising, and increases the pixel value and gradient of the texture and deformed edge in the first feature map to obtain a second feature map after enhancing the details. Then, the main branch of the image processing model processes the second feature map to obtain the image to be processed after denoising and detail enhancement.

[0033] Furthermore, the processing module includes: a construction unit, an input unit, a first training unit, a second training unit, and a removal unit;

[0034] The building unit is used to build an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches;

[0035] The input unit is used to input training images into the image processing training model;

[0036] The first training unit is used to denoise the training image using an optimization algorithm and to train the auxiliary branch to output the denoised first target image.

[0037] The second training unit is used to perform detail enhancement processing on the first target image using an optimization algorithm, and trains the main branch to output the second target image with enhanced details;

[0038] The removal unit is used to determine the parameters of the image processing training model after the model training is completed, remove auxiliary branches, and form a trained image processing model.

[0039] Furthermore, the construction of the image processing training model specifically involves:

[0040] A deep neural network model is used as the image processing training model; wherein, the image processing training model includes a backbone network, a main branch, and an auxiliary branch;

[0041] The backbone network adopts the ESRNet structure, which consists of an upsampling layer and an downsampling layer.

[0042] The auxiliary branch is used to output the image after the upsampled layer of the backbone network;

[0043] The main branch is used to output the image after passing through the downsampled layer of the backbone network.

[0044] Furthermore, the processing module includes: a mapping unit;

[0045] The mapping unit is used to map the second feature map to the color space through the main branch of the image processing model, so as to obtain the image to be processed after denoising and detail enhancement.

[0046] This invention provides an image processing apparatus that decouples noise reduction and detail enhancement. Based on the organic combination of modules, it can decouple the two capabilities of noise reduction and detail enhancement, so that when processing images, it can improve the noise removal rate and enhance details to improve clarity. Attached Figure Description

[0047] Figure 1 A flowchart illustrating an embodiment of the image processing method for decoupling, denoising, and enhancing details provided by the present invention;

[0048] Figure 2 A schematic flowchart of another embodiment of the image processing method for decoupling, denoising, and enhancing details provided by the present invention;

[0049] Figure 3 This is a schematic diagram of one embodiment of the image processing training model provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In the description of this invention, it should be understood that the following terminology is used:

[0052] 1. "LR": Low Resolution Image.

[0053] 2. "HR": High Resolution Image.

[0054] 3. "GT": Ground Truth, target image.

[0055] 4. "DN": Denoise Image.

[0056] like Figure 1 The diagram shown is a flowchart of an embodiment of the image processing method for decoupling, denoising, and enhancing details provided by the present invention. The method includes step 101, as follows:

[0057] Step 101: Input the image to be processed into the trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to be processed, thereby obtaining a first feature map after denoising, and increases the pixel value and gradient of the texture and deformed edge in the first feature map to obtain a second feature map after enhancing the details, and then the main branch of the image processing model processes the second feature map to obtain the image after denoising and detail enhancement.

[0058] In this embodiment of the invention, the noise in the image to be processed is: grid artifacts, ringing phenomena, and additive Gaussian noise introduced during the recording and transmission of the image; wherein, the noise appears uniformly in the full frame of the image to be processed or appears at the texture edges of the image to be processed.

[0059] In this embodiment of the invention, the main branch of the image processing model processes the second feature map to obtain an image of the image to be processed after denoising and detail enhancement, specifically as follows:

[0060] The second feature map is mapped to the color space through the main branch of the image processing model, resulting in an image of the image to be processed after denoising and detail enhancement.

[0061] In this embodiment of the invention, a trained image processing model is used to process the image to be processed, as well as perform inverse denoising and detail enhancement. First, the image to be processed is input into the model. The backbone network of the model can denoise the image to obtain the denoised image. Then, the denoised image is subjected to detail enhancement. The main branch of the model maps the denoised and detail-enhanced image to the color space and outputs the processed high-resolution image.

[0062] like Figure 2 The diagram shown is a flowchart of another embodiment of the image processing method for decoupling noise reduction and enhancing details provided by the present invention, mainly including steps 201 to 205, each step as follows:

[0063] In this embodiment of the invention, step 101 includes steps 201 to 205.

[0064] Step 201: Construct an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches.

[0065] In this embodiment of the invention, the image processing training model is constructed as follows:

[0066] A deep neural network model is used as the image processing training model; wherein, the image processing training model includes a backbone network, a main branch, and an auxiliary branch;

[0067] The backbone network adopts the ESRNet structure, which consists of an upsampling layer and an downsampling layer.

[0068] The auxiliary branch is used to output the image after the upsampled layer of the backbone network;

[0069] The main branch is used to output the image after passing through the downsampled layer of the backbone network.

[0070] In this embodiment of the invention, a deep neural network model is used as the image processing training model. The backbone network of this model can be a commonly used ESRNet structure, consisting of several convolutional blocks used to extract features from the input LR. These convolutional blocks can be divided into upsampling layers and downsampling layers. The upsampling layer is used to denoise the input image, and the denoised image serves as the input to the downsampling layer for detail enhancement. The auxiliary branch and the main branch of the model serve as the output images. The auxiliary branch outputs the denoised image, and the main branch outputs the image with enhanced details. Outputting both the denoised and enhanced images provides better feedback on the model's training results in both denoising and detail enhancement, thereby enhancing the model's training effectiveness.

[0071] Step 202: Input the training image into the image processing training model.

[0072] Step 203: Use an optimization algorithm to denoise the training image and train the auxiliary branch to output the denoised first target image.

[0073] Step 204: Use an optimization algorithm to enhance the details of the first target image, and train the main branch to output the enhanced second target image.

[0074] In this embodiment of the invention, the optimization algorithm is specifically as follows:

[0075] A loss function is used as the objective function of the optimization algorithm. The loss function consists of two parts, which guide the auxiliary branch and the main branch to train the image processing model, respectively.

[0076] The expression for the loss function is:

[0077]

[0078] in, DN is the loss function; DN is the first training target image output by the auxiliary branch; HR is the second training target image output by the main branch; GT dn It is an image of the same size as the training image, but with noise removed; GTh r It is an image with a higher resolution than the training image and no noise; the function l1 represents the norm; λ is the weight coefficient.

[0079] In this embodiment of the invention, since a deep neural network-based image processing model is used and solved by an optimization algorithm, as long as the objective function is reasonable, the stochastic gradient descent algorithm can iteratively solve for local optima for each layer of the model. The model trained in this way can provide larger response values ​​to areas with noise, and then reduce their pixel values, resulting in a cleaner image without artifacts. This invention selects a loss function as the objective function for the priority algorithm. The loss function consists of two parts, guiding the auxiliary branch and the main branch respectively. After determining the model structure and the objective function, the image processing training model can be trained on the training dataset.

[0080] Step 205: After the model training is completed, determine the parameters of the image processing training model, remove the auxiliary branches, and form a trained image processing model.

[0081] In this embodiment of the invention, after the image processing training model is trained, the parameters of the model are determined, duplicate branches are removed, and a usable image processing model is formed. When the image processing model is used, only the backbone network and the main branch are retained. The low-resolution image (LR) is input into the model, and after denoising and detail enhancement are performed by the backbone network, the high-resolution image (HR) is output by the main branch.

[0082] As an example of an embodiment of the present invention, such as Figure 3 The diagram shown is a schematic representation of an embodiment of the image processing training model provided by this invention. The image processing training model consists of a backbone network, auxiliary branches, and a main branch. The backbone network can be an ESRNet structure, composed of several convolutional blocks, used to extract features from the input LR. The feature map output by the intermediate layer. This is the final output feature map. The main branch takes f2 as input and outputs HR after passing through a convolutional layer. This invention proposes adding an auxiliary branch at f1, taking f1 as input and outputting a denoised image DN after passing through a convolutional layer. During model training, the loss function guides the backbone network before f1 to learn the ability to remove noise, while the structure between f1 and f2 learns the ability to enhance details. This design method explicitly decouples the super-resolution task into denoising and detail enhancement, and models them in the same model. During training, an auxiliary branch is added to the intermediate layers to output the denoised image, and the loss function guides the structure before the intermediate layers to learn the denoising ability, while the main branch outputs the super-resolution image, guiding the latter half to learn the ability to enhance details. In use, the auxiliary branch is removed, retaining only the backbone network and the main branch, without increasing the additional computational cost.

[0083] Accordingly, the present invention provides an image processing apparatus for decoupling noise reduction and enhancing details, the apparatus including a processing module;

[0084] The processing module is used to input the image to be processed into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to be processed, thereby obtaining a first feature map after denoising, and increases the pixel value and gradient of the texture and deformed edge in the first feature map to obtain a second feature map after enhancing the details. Then, the main branch of the image processing model processes the second feature map to obtain the image to be processed after denoising and detail enhancement.

[0085] In this embodiment of the invention, the processing module includes: a construction unit, an input unit, a first training unit, a second training unit, and a removal unit;

[0086] The building unit is used to build an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches;

[0087] The input unit is used to input training images into the image processing training model;

[0088] The first training unit is used to denoise the training images using an optimization algorithm and to train the auxiliary branch to output the denoised first target image.

[0089] The second training unit is used to enhance the details of the first target image using an optimization algorithm, and trains the main branch to output the enhanced details of the second target image.

[0090] The removal unit is used to determine the parameters of the image processing training model after the model training is completed, remove auxiliary branches, and form a trained image processing model.

[0091] In this embodiment of the invention, the image processing training model is constructed as follows:

[0092] A deep neural network model is used as the image processing training model; the image processing training model includes a backbone network, a main branch, and an auxiliary branch.

[0093] The backbone network adopts the ESRNet structure, which consists of upsampling layers and downsampling layers;

[0094] The auxiliary branch is used to output the image after the upsampled layer of the backbone network;

[0095] The main branch is used to output the image after passing through the downsampled layer of the backbone network.

[0096] In this embodiment of the invention, the processing module includes: a mapping unit;

[0097] The mapping unit is used to map the second feature map to the color space through the main branch of the image processing model, so as to obtain the image to be processed after denoising and detail enhancement.

[0098] In summary, the image processing method and apparatus for decoupling denoising and detail enhancement provided in this invention inputs the image to be processed into a trained image processing model. The backbone network of the image processing model then increases the response value and decreases the pixel value of the noise in the image to obtain a denoised image. Furthermore, the pixel values ​​and gradients of textures and deformed edges in the denoised image are increased to obtain a second feature map with enhanced details. Finally, through the main branch of the image processing model, the image after denoising and detail enhancement is obtained. This invention explicitly decouples the image super-resolution task into denoising and detail enhancement, ensuring that noise in the image is removed before enhancement, preventing amplification and artifacts. The model can simultaneously handle both denoising and detail enhancement, making it easier to optimize and resulting in a clearer image.

[0099] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An image processing method for decoupling noise reduction and enhancing details, characterized in that, include: The image to be processed is input into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to obtain a first feature map after denoising. The pixel value and gradient of the texture and deformed edge in the first feature map are increased to obtain a second feature map after enhancing the details. Then, the main branch of the image processing model processes the second feature map to obtain the image after denoising and detail enhancement.

2. The image processing method for decoupling, denoising, and enhancing details according to claim 1, characterized in that, The trained image processing model is specifically as follows: Construct an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches; The training image is input into the image processing training model; The training images are denoised using an optimization algorithm, and the auxiliary branch is trained to output the denoised first target image. An optimization algorithm is used to enhance the details of the first target image, and the main branch is trained to output the enhanced second target image. Once the model training is complete, the parameters of the image processing training model are determined, auxiliary branches are removed, and the trained image processing model is formed.

3. The image processing method for decoupling, denoising, and enhancing details according to claim 2, characterized in that, The optimization algorithm is as follows: A loss function is used as the objective function of the optimization algorithm. The loss function consists of two parts, which guide the auxiliary branch and the main branch to train the image processing model, respectively. The expression for the loss function is: in, DN is the loss function; DN is the first training target image output by the auxiliary branch; HR is the second training target image output by the main branch; GT dn It is an image of the same size as the training image, but with noise removed; GTh r It is an image with a higher resolution than the training image and no noise; the function l1 represents the norm; λ is the weight coefficient.

4. The image processing method for decoupling, denoising, and enhancing details according to claim 2, characterized in that, The construction of the image processing training model specifically involves: A deep neural network model is used as the image processing training model; wherein, the image processing training model includes a backbone network, a main branch, and an auxiliary branch; The backbone network adopts the ESRNet structure, which consists of an upsampling layer and an downsampling layer. The auxiliary branch is used to output the image after the upsampled layer of the backbone network; The main branch is used to output the image after passing through the downsampled layer of the backbone network.

5. The image processing method for decoupling, denoising, and enhancing details according to claim 1, characterized in that, The process of having the main branch of the image processing model process the second feature map to obtain the image after denoising and detail enhancement is specifically as follows: The second feature map is mapped to the color space through the main branch of the image processing model, resulting in an image of the image to be processed after denoising and detail enhancement.

6. The image processing method for decoupling, denoising, and enhancing details according to claim 1, characterized in that, The noise in the image to be processed includes: grid artifacts, ringing phenomena, and additive Gaussian noise introduced during the recording and transmission of the image; wherein, the noise appears uniformly across the entire frame of the image to be processed or appears at the texture edges of the image to be processed.

7. An image processing apparatus for decoupling noise reduction and enhancing details, characterized in that, include: Processing module; The processing module is used to input the image to be processed into a trained image processing model, so that the backbone network of the image processing model increases the response value and decreases the pixel value of the noise in the image to be processed, thereby obtaining a first feature map after denoising, and increases the pixel value and gradient of the texture and deformed edge in the first feature map to obtain a second feature map after enhancing the details. Then, the main branch of the image processing model processes the second feature map to obtain the image to be processed after denoising and detail enhancement.

8. The image processing apparatus for decoupling noise reduction and enhancing details according to claim 7, characterized in that, The processing module includes: a construction unit, an input unit, a first training unit, a second training unit, and a removal unit; The building unit is used to build an image processing training model, wherein the image processing training model includes a backbone network, a main branch, and auxiliary branches; The input unit is used to input training images into the image processing training model; The first training unit is used to denoise the training image using an optimization algorithm and to train the auxiliary branch to output the denoised first target image. The second training unit is used to perform detail enhancement processing on the first target image using an optimization algorithm, and trains the main branch to output the second target image with enhanced details; The removal unit is used to determine the parameters of the image processing training model after the model training is completed, remove auxiliary branches, and form a trained image processing model.

9. The image processing apparatus for decoupling, denoising, and enhancing details according to claim 8, characterized in that, The construction of the image processing training model specifically involves: A deep neural network model is used as the image processing training model; wherein, the image processing training model includes a backbone network, a main branch, and an auxiliary branch; The backbone network adopts the ESRNet structure, which consists of an upsampling layer and an downsampling layer. The auxiliary branch is used to output the image after the upsampled layer of the backbone network; The main branch is used to output the image after passing through the downsampled layer of the backbone network.

10. The image processing apparatus for decoupling, denoising, and enhancing details according to claim 7, characterized in that, The processing module includes: a mapping unit; The mapping unit is used to map the second feature map to the color space through the main branch of the image processing model, so as to obtain the image to be processed after denoising and detail enhancement.

Images