Denoising tone mapping method and device based on deep learning

Abstract

The present invention proposes a denoising tone mapping method and device based on deep learning. The method comprises the following steps: S1, calibrating camera noise parameters; S2, building a denoising tone mapping network model; S3, decomposing the high-dynamic images in the training set into a series of low-dynamic images with different exposures, adding noise to each of them, resynthesizing the noisy high-dynamic images, and preprocessing the noisy high-dynamic images as input to the denoising tone mapping network; S4, training the denoising tone mapping network through backpropagation using a structural loss function and a perceptual loss function until the network converges. The present invention can implement denoising tone mapping end-to-end, directly converting noisy high-dynamic images into high-quality denoised tone mapped images, effectively solving the problem of uneven noise distribution in high-dynamic images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a denoising tone mapping method and device based on deep learning. Background Art

[0002] High dynamic range (HDR) images and videos can display more brightness and color intensity, better reflecting the visual effects of real-world environments. However, common display devices have a very limited dynamic range, which is insufficient to display these images and videos. Tone mapping can compress these vast pixel values ​​into a range that can be displayed by common devices while ensuring satisfactory visual effects. Although tone mapping methods have been well studied over the past few years, problems still exist in practical applications.

[0003] First, the complex noise distribution of various high dynamic range sources may affect performance. In practical applications, the most widely used high dynamic range cameras use bracketing to generate high dynamic range images through non-uniform weighted combination. Therefore, the noise distribution in the generated high dynamic range images no longer meets the commonly used noise assumptions (such as Gaussian distribution or Poisson distribution). In addition, existing research does not take into account the subsequent tone mapping process, which is often highly nonlinear and may amplify noise. The noise distribution of tone mapped images can be very uneven. Directly denoising the tone mapped images is not the best choice because common denoising methods have difficulty dealing with both small and large noises. In addition, denoising before high dynamic range synthesis may degrade many details and lead to unnecessary information loss. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention proposes a denoising tone mapping method and device based on deep learning.

[0005] The technical solution adopted in the present invention is as follows:

[0006] A denoising tone mapping method based on deep learning, the method comprises the following steps:

[0007] S1, calibrate camera noise parameters;

[0008] S2, build a denoising tone mapping network model;

[0009] S3, decomposing the high dynamic range image in the training set into a series of low dynamic range images with different exposures and adding noise to each of them, re-synthesizing the noisy high dynamic range image, and pre-processing the noisy high dynamic range image as the input of the denoising tone mapping network;

[0010] S4, using the structural loss function and the perceptual loss function, the denoising tone mapping network is trained by backpropagation until the network converges.

[0011] The present invention also provides a denoising tone mapping device based on deep learning, the device comprising:

[0012] Camera calibration module, used to calibrate camera noise parameters;

[0013] Denoising tone mapping network, used to convert noisy high-dynamic images into denoised tone-mapped images;

[0014] An image processing module, configured to decompose the high-dynamic image into a series of low-dynamic images of different exposures, add noise to each of the images, recompose the noisy high-dynamic image, and pre-process it before inputting it into the denoising tone mapping network;

[0015] A training module is used to train the denoising tone mapping network through back propagation.

[0016] The method of the present invention does not require any manual parameter adjustment and can quickly convert noisy high-dynamic images of various scenes into high-resolution, high-quality denoised tone-mapped images, effectively solving the problem of uneven noise distribution in high-dynamic images. The specific effects are as follows:

[0017] (1) The spatial adaptive denoising module (SADM) consists of multiple cascaded denoising units. The dilated convolution kernels in these units gradually expand to increase the receptive field, which can better cope with a wider range of noise.

[0018] (2) The attention mechanism in the denoising unit enables the network to adaptively handle different levels of noise in high-dynamic images.

[0019] (3) The camera calibration method in the present invention only requires shooting a dark field, which is simpler than other calibration methods and has good accuracy.

[0020] (4) The image decomposition and noise addition method in the method of the present invention can more realistically introduce high dynamic image noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a diagram of the denoising tone mapping network framework of the present invention;

[0022] Figure 2 It is a module diagram of the denoising tone mapping network of the present invention;

[0023] Figure 3 Flowchart of the denoising tone mapping method of the present invention. DETAILED DESCRIPTION

[0024] To describe the purpose and method of the present invention in detail, the following is a detailed description with reference to the accompanying drawings. Figure 1-Figure 3, the present invention provides some embodiments of a denoising tone mapping method based on deep learning.

[0025] The present invention is based on the principle that when a low-dynamic image stack contains noise, if these low-dynamic images are denoised separately (the noise distribution of these images conforms to conventional assumptions in the art) and then subjected to high-dynamic synthesis and tone mapping, the resulting tone-mapped image will lose significant detail because denoising inevitably blurs some structural information. Artifacts generated during the denoising process will also affect image quality. This approach is acceptable when only one image requires denoising. However, a stack typically contains multiple images, and the blurred information and artifacts generated by denoising multiple images will have a cumulative effect, severely degrading the quality of the final tone-mapped image. If denoising is performed on the noisy image after tone mapping, since the noise in the high-dynamic image is non-uniformly distributed, stronger noise will be further amplified by the tone mapping. Different tone mapping methods amplify noise to varying degrees, but the amplified noise cannot be effectively removed in the subsequent denoising process. This is because conventional denoising methods do not target this extremely non-uniformly distributed noise and are therefore unable to effectively remove it.

[0026] Previous tone mapping methods primarily focused on achieving high-quality output that adapts to the human visual system, while ignoring image noise. Previous high-dynamic range denoising methods also failed to account for noise amplification caused by subsequent tone mapping, and therefore were unable to effectively remove strong noise. The method of the present invention aims to simultaneously perform tone mapping while denoising, directly converting noisy high-dynamic range images into noise-free, high-quality tone-mapped images.

[0027] like Figure 2 As shown, the present invention proposes a denoising tone mapping method based on deep learning, comprising the following steps:

[0028] 1. Calibrate camera noise parameters.

[0029] The camera noise is modeled as:

[0030] V i =g(S i +D i )+R i

[0031] Where V is the pixel value, S is the photon noise, D is the dark current noise, R is the readout noise, i is the pixel position index, and g is the camera gain.

[0032]

[0033] Among them, N d is the number of electrons excited by the photoelectric effect, N eis the number of electrons generated by dark current, is the variance of the readout noise, P(·) indicates Poisson distribution, and N(·) indicates Gaussian distribution.

[0034] Using the camera to capture a dark frame gives:

[0035]

[0036] Among them, E[·] represents the mean, Var[·] represents the variance. From the above formula, we can deduce By taking multiple dark frames with different exposure times, a straight line can be fitted with an intercept of The slope is g. And the dark current noise N per unit exposure time can be derived d In addition, when the exposure time t approaches 0, the dark current noise also approaches 0, and the image variance is the readout noise variance. In summary, the readout noise variance is

[0037]

[0038] Adding expectation to the clean image is Poisson noise (photon noise and dark current noise) is multiplied by g and added with a mean of 0 and a variance of Gaussian noise (readout noise).

[0039] 2. Build a denoising tone mapping network model.

[0040] Construct a denoising tone mapping network structure. The network of this embodiment is an end-to-end U-net-like architecture, which is mainly composed of an encoder network, several spatial adaptive denoising modules and a decoder network. Specifically, the convolution layer is first used to extract image features, and then the encoder network composed of a set of downsampling layers is used to learn the deep features of the input high-dynamic image, and feed it back to the denoising module to remove non-uniformly distributed noise. In order to better extract features, a residual block (RB) is introduced in the encoder, and then the features are input into the spatial adaptive denoising module SADM for denoising. Finally, the output result of the denoising module is input into a decoder network composed of a series of upsampling layers, and fused into the final decoding layer to generate a tone mapping image. In addition, jump connections are used to help the network better utilize more complete information and produce clearer results.

[0041] The SADM module of the present invention introduces an attention mechanism. High levels of noise tend to appear in brighter areas of the image. Therefore, the SADM module helps the network focus on areas with stronger noise (brighter areas) and compress the values ​​in these areas, guiding the subsequent network to better automatically remove noise. In addition, the attention mechanism also helps to adjust brightness and ensure global and local contrast.

[0042] 3. Decompose the high-dynamic images in the training set into a series of low-dynamic images with different exposures and add noise to each of them, then resynthesize the noisy high-dynamic images. After preprocessing, the noisy high-dynamic images are used as the input of the denoising tone mapping network.

[0043] The clean high-dynamic image is decomposed into a series of low-dynamic stacks with different exposures, and noise is added to each of these low-dynamic images. The specific process of high-dynamic image decomposition is as follows: the high-dynamic image pixel values ​​are truncated and normalized; the Newton iteration method is used to find an α such that avg(floor(clamp((αV) 12.2 )·255))=127, where avg(·) is the averaging operation, floor(·) is rounding down, clamp(·) limits the value range to [0,1], and V is the pixel value. Taking the current image as the reference image, assuming that the current image exposure time is 1, increasing or decreasing the exposure time to obtain a low-dynamic image stack, the pixel value of the j-th image is floor(clamp((αt j V) 12.2 )·255), t j is the exposure time of the jth image.

[0044] The formula for re-synthesizing a noisy high-dynamic image is:

[0045] in

[0046] Where v is the pixel value, t is the exposure time, i is the index of the exposure time, the total number of exposures is n, and f -1 represents the camera response function and ω is an arbitrary weighting function.

[0047] Preprocessing of noisy high-dynamic images: Random cropping and geometric transformations such as horizontal flipping and vertical flipping of input image pairs to expand the dataset; Normalization of high-dynamic images: Among them, I represents the high dynamic image pixel value, I min Represents the minimum pixel value of the image, I max Represents the maximum pixel value of the image.

[0048] 4. Use the structural loss function and the perceptual loss function to train the denoising tone mapping network through backpropagation until the network converges.

[0049] Among them, the structural loss function is:

[0050]

[0051] in, and y are the tone-mapped image and true value output by the network, respectively. for The average value, μ y is the mean value of y, for The variance of is the variance of y, for and the covariance of y, c1 and c2 are two constants used to avoid division by zero.

[0052] The perceptual loss function is:

[0053]

[0054] where φ represents the network used to extract image features, i is the index of layer M, and U is the element-wise sum.

[0055] The total loss function is:

[0056]

[0057] Among them, λ is the weight coefficient.

[0058] The network structure proposed in this embodiment is as follows:

[0059] (1) High dynamic image input denoising tone mapping network, the image first enters a convolution layer with a stride of 1, an output channel of 64, and a convolution kernel size of 3*3, and then passes through the ReLU activation function.

[0060] (2) Input residual block RB is used to extract shallow features. The specific structure of RB is shown in Figure 3 . The feature layer (the output of step (1)) first enters a convolution layer with a stride of 1, an output channel of 64, and a convolution kernel size of 3*3, and then passes through instance normalization and ReLU activation function. After passing through a convolution layer with a stride of 1, an output channel of 64, a convolution kernel size of 3*3 and instance normalization, the output is added to the input of RB, and a residual connection is introduced to better transfer features and prevent gradient disappearance. After passing through a ReLU and a convolution layer with a stride of 1, an output channel of 64, and a convolution kernel size of 1*1, the final output of RB is obtained by the ReLU activation function. RB uses residual connections to better extract features, and using RB at different depths can extract features of different depths.

[0061] (3) The output features of step (2) enter a convolution layer with a stride of 1, an output channel of 64, and a convolution kernel size of 3*3, and then undergo instance normalization and ReLU activation function.

[0062] (4) Input the RB module to extract deeper features.

[0063] (5) The output of step (4) enters a convolutional layer with a stride of 2, an output channel of 128, and a convolution kernel size of 3*3, and then undergoes instance normalization and ReLU activation function.

[0064] (6) Enter a convolutional layer with a stride of 2, an output channel of 256, and a convolution kernel size of 3*3, and then undergoes instance normalization and ReLU activation function.

[0065] (7) After 3 SADM modules. See the detailed structure for details. Figure 3 The SADM module consists of multiple cascaded denoising units. First, it enters a dilated convolution layer with a stride of 1, an output channel of 256, and a ReLU activation function. Then it passes through a convolution layer with a stride of 1, an output channel of 256, a convolution kernel size of 1*1, and a Sigmoid activation function. The output is multiplied by the output of the previous ReLU and then concatenated with the original feature layer to obtain the output of a denoising unit. The Sigmoid activation function can output an attention map with a value between [0,1] to guide the network to allocate different attention to unevenly distributed noise to obtain better denoising effects. These denoising units in SADM are connected by dense connections. These cascaded units can greatly increase the receptive field of the network and cooperate with the attention mechanism to achieve excellent denoising and tone mapping effects. The convolution kernel sizes of the dilated convolutions in the three SADM modules are 1, 3, and 5, respectively.

[0066] (7) Enter a deconvolution layer with a stride of 1, an output channel of 256, and a convolution kernel size of 3*3, and then undergo instance normalization and ReLU activation function.

[0067] (8) Enter a deconvolution layer with a stride of 2, an output channel of 128, and a convolution kernel size of 3*3, and then undergoes instance normalization and ReLU activation function.

[0068] (9) Enter a deconvolution layer with a stride of 2, an output channel of 64, and a convolution kernel size of 3*3, and then undergoes instance normalization and ReLU activation function.

[0069] (10) Enter a convolutional layer with a stride of 1, an output channel of 64, and a convolution kernel size of 3*3, and then pass through the ReLU activation function.

[0070] (11) Enter a convolution layer with a stride of 1, an output channel of 64, and a convolution kernel size of 1*1, and then pass through the Sigmoid activation function.

[0071] The tone-mapped image output by the network and the true value of the tone-mapped image are used to calculate the loss using the above-mentioned structural loss function.

[0072] The neural network model in the present invention can realize end-to-end high dynamic image denoising and tone mapping, and directly obtain high-quality tone mapping images.

[0073] It should be noted that the application of the present invention is not limited to the above embodiments. Those skilled in the art can make improvements or changes based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A denoising tone mapping method based on deep learning, characterized in that: The method comprises the following steps: S1, calibrate camera noise parameters; S2, building a denoising tone mapping network model; the denoising tone mapping network includes an encoder network, a spatially adaptive denoising module, and a decoder network. First, convolutional layers are used to extract image features. Then, the encoder network, consisting of a set of downsampling layers, learns the deep features of the input high-dynamic range image and feeds them back to the spatially adaptive denoising module to remove non-uniformly distributed noise. Finally, the output of the spatially adaptive denoising module is input into a decoder network consisting of a series of upsampling layers and fused into the final decoding layer to generate a tone-mapped image. S3, decomposing the high dynamic range image in the training set into a series of low dynamic range images with different exposures and adding noise to each of them, re-synthesizing the noisy high dynamic range image, and pre-processing the noisy high dynamic range image as the input of the denoising tone mapping network; S4, using the structural loss function and the perceptual loss function, the denoising tone mapping network is trained by backpropagation until the network converges.

2. The denoising tone mapping method based on deep learning according to claim 1, characterized in that Step S1 specifically includes: adding the expected Poisson noise is multiplied by g and then added with a mean of 0 and a variance of Gaussian noise; where V is the pixel value, i is the pixel position index, g is the camera gain, N d is the number of electrons excited by the photoelectric effect, and t is the exposure time.

3. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S2, a residual block is introduced into the encoder network.

4. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S2, the spatial adaptive denoising module is composed of a plurality of cascaded denoising units.

5. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S3, decomposing the high dynamic range images in the training set specifically includes: Cut off the parts of the high-dynamic image with excessively high pixel values ​​and normalize them; Using Newton's method, find an α such that avg(floor(clamp((αV) 12.2 )·255))=127, where avg(·) is the averaging operation, floor(·) is rounding down, clamp(·) limits the value range to [0,1], and V is the pixel value; Taking the current image as the reference image, assuming that the exposure time of the current image is 1, increasing or decreasing the exposure time to obtain a low-dynamic image stack, the pixel value of the j-th image is floor(clamp((αt j V) 12.2 )·255), t j is the exposure time of the jth image.

6. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S3, the formula for re-synthesizing the noisy high-dynamic image is: Where v is the pixel value, t is the exposure time, i is the index of the exposure time, the total number of exposures is n, and f -1 represents the camera response function and ω is an arbitrary weighting function.

7. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S4, the structural loss function is: in, and y are the tone-mapped image and true value output by the network, respectively. for The average value, μ y is the mean value of y, for The variance of is the variance of y, for and the covariance of y, c1 and c2 are two constants used to avoid division by zero.

8. The denoising tone mapping method based on deep learning according to claim 1, characterized in that In step S4, the perceptual loss function is: Among them, φ represents the network used to extract image features, and y are the tone-mapped image and true value output by the network, respectively, i is the index of layer M, and U is the element-wise sum.

9. A device for implementing the deep learning-based denoising tone mapping method according to claim 1, characterized in that: The device includes: Camera calibration module, used to calibrate camera noise parameters; The denoising tone mapping network is used to convert the noisy high-dynamic image into a denoised tone-mapped image; An image processing module, configured to decompose the high-dynamic image into a series of low-dynamic images of different exposures, add noise to each of the images, recompose the noisy high-dynamic image, and pre-process it before inputting it into the denoising tone mapping network; A training module is used to train the denoising tone mapping network through back propagation.

Images