A dual-domain joint image denoising method based on neural network

Abstract

The present invention relates to a neural network-based dual-domain joint image denoising method. The method comprises a training phase, in which a given simulated noise model is used to generate a set of purely simulated images to train a dual-domain image denoising network; and an inference phase, in which the trained dual-domain image denoising network is directly applied to the denoising locations in the raw and sRGB domains of any camera's image processing to perform image denoising. Compared with existing technologies, this method offers advantages such as significantly improved denoising performance and generalization capabilities.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a dual-domain joint image denoising method based on a neural network. Background Art

[0002] The smart camera's sensor receives the scene's light information and forms a raw image. This image is then processed through the image signal processing chain to produce an sRGB image, which is then presented to the user. During the imaging process of the smart camera, noise is introduced in the raw domain, amplified by the image signal processing chain, and accumulated in the sRGB image, reducing the final image quality. Existing high-quality image denoising neural networks can be divided into two categories: one that performs noise reduction solely in the raw domain, and the other that performs noise reduction solely in the sRGB domain. These neural networks are trained on large-scale datasets obtained through simulation or real-world photography, ultimately achieving excellent noise reduction results in the corresponding raw / sRGB domains. In addition to neural network-based noise reduction solutions, there are also a number of traditional algorithms based on artificial priors and statistical theory, but their noise reduction performance is far inferior to that of neural network-based image denoising solutions.

[0003] Denoising solutions based on traditional algorithms have low computational overhead and are suitable for most scenarios. However, their disadvantage is poor noise reduction performance, especially inability to handle severe noise. Currently, mobile phones and other devices have begun to use neural network-based noise reduction solutions. Although neural network-based noise reduction solutions have good noise reduction performance in the single raw / sRGB domain, they still have certain shortcomings:

[0004] 1) Although the denoising network in the raw domain can directly access and remove the original noise, the presence of the subsequent image signal processing chain causes the residual noise after denoising in the raw domain to be amplified or entangled with the image signal after passing through the image signal processing chain, resulting in aliasing noise or blocking artifacts in the sRGB domain.

[0005] 2) Although denoising networks in the sRGB domain are directly related to the final image, they lack information about the original image noise. When faced with unknown noise levels and types after the image signal processing chain, the denoising effect is not very ideal, resulting in blurred local details and residual noise.

[0006] 3) Both of the above-mentioned networks are highly dependent on the matching of training and inference data. However, in practical applications, not only is it expensive, time-consuming and labor-intensive to obtain specific high-quality training data, but the noise level and type in the actual shooting scene and the image signal processing chain associated with the imaging device are in a state of continuous change. Once the objective conditions during training and inference do not match, the noise reduction performance of the network will be greatly reduced.

[0007] Therefore, improving the denoising ability and generalization of denoising networks in practical applications has become a technical problem that needs to be solved. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a neural network-based dual-domain joint image denoising method that greatly improves the denoising performance and generalization ability.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] According to one aspect of the present invention, a dual-domain joint image denoising method based on a neural network is provided, the method comprising:

[0011] During the training phase, a pure simulated image set is generated using a given simulated noise model to train the dual-domain image denoising network;

[0012] During the inference phase, the trained dual-domain image denoising network is directly applied to the denoising positions of the raw domain and sRGB domain of any camera's image processing to perform image denoising.

[0013] As a preferred technical solution, the dual-domain image denoising network is specifically:

[0014] A denoising network is introduced in both the raw and sRGB domains, using the noise distribution prior in the raw domain for preliminary noise reduction. The residual noise amplified by the image signal processing chain is then removed in the subsequent sRGB domain.

[0015] As a preferred technical solution, the training phase process specifically includes:

[0016] Step S101: Generate a simulated noisy raw image using a simulated noise model, and convert the noise-free raw image into a noise-free sRGB image for subsequent calculation of the loss function using a differentiable image signal processing link;

[0017] Step S102: The noisy raw image is input into a raw domain denoising network to obtain an output raw image. The output raw image is processed through a differentiable image signal processing chain to obtain a corresponding sRGB image, which is then input into an sRGB domain denoising network to obtain an output sRGB image.

[0018] Step S103 : Calculate the loss function values of the output raw image and the noise-free raw image, and the output sRGB image and the noise-free sRGB image, and perform gradient backpropagation to simultaneously optimize the raw domain denoising network and the sRGB domain denoising network.

[0019] As a preferred technical solution, the simulation noise model in step S101 is specifically as follows:

[0020]

[0021] Where K is the total gain coefficient of the imaging system (also regarded as the noise level), R * is the noise-free raw image, R is the generated noisy raw image, σ r 2 is the variance of the Gaussian noise, kind Represent Poisson and Gaussian distributions respectively. In actual practice, the noise model can adopt a more appropriate form according to the actual situation. The Gaussian-Poisson distribution form used here is just a general expression.

[0022] As a preferred technical solution, the total gain coefficient K and variance σ of the imaging system r 2 Obey the joint probability density distribution, as follows:

[0023]

[0024]

[0025] in and Represent uniform and Gaussian distributions respectively, K min / K max Represent the minimum / maximum noise level obtained by the noise model, log(σ r 2 ) is the Gaussian noise variance σ r 2 The logarithmic function of log(K) is the logarithmic function of the total gain coefficient K of the imaging system, log(σ r 2 ) is approximately linearly related to log(K), through the slope α, intercept β and Gaussian variable σ 2 To control, coefficient K min , K max , α, β, and σ can be obtained by performing noise calibration on specific equipment during actual processing.

[0026] As a preferred technical solution, step S102 specifically includes the following steps:

[0027] Step S1021: transform the single-channel noisy raw image into a four-channel noisy raw image using channel transformation and input it into the raw domain denoising neural network to obtain a four-channel output raw image;

[0028] Step S1022: The four-channel output raw image is subjected to channel inverse transformation to obtain a single-channel output raw image;

[0029] In step S1023, the single-channel output raw image is converted into an input sRGB image of the sRGB domain denoising neural network after passing through the differentiable image signal processing link, and the output sRGB image is obtained after further denoising.

[0030] As a preferred technical solution, the raw domain denoising network and the sRGB domain denoising network are both equipped with a noise prior fusion module for fusing the noise prior with the input image and obtaining a fused feature vector.

[0031] As an optimal technical solution, the differentiable image signal processing chain includes a black level correction module, a normalization module, a white balance module, a demosaicing module, a color space conversion module, a gamma correction module and an adjustable tone mapping module, which is responsible for the conversion process from raw image to sRGB image.

[0032] As a preferred technical solution, the demosaicing module adopts a differentiable version of the adaptive homogeneity-guided demosaicing algorithm; the adjustable tone mapping module adopts a quadratic curve whose curvature is controlled by the gain coefficient α, which is expressed as follows:

[0033] R n =R n-1 +α·R n-1 (1-R n-1 ),

[0034] Among them, R n Represents the tone mapping result image after n iterations, R0 is the original image, and α represents the gain coefficient.

[0035] As a preferred technical solution, this method uses the L1 loss function for gradient backpropagation, where the loss function is as follows:

[0036] L=λ·||RR * ||1+||II * ||1,

[0037] Among them, ||||1 represents the L1 loss function, R, R * ,I,I * They represent the output raw image, noise-free raw image, output sRGB image, and noise-free sRGB image respectively, and λ represents the adjustment coefficient of raw domain supervision in the loss function.

[0038] Compared with the existing technology, the present invention is comparable to the previous single-domain image denoising solution in two aspects: the denoising ability and generalization of the denoising network, as follows

[0039] 1) In terms of generalization, the present invention achieves that a denoising neural network trained on a simulated image set can successfully achieve excellent denoising performance on a real image set, even if the noise level of the images in this real image set and the image signal processing chain they pass through are unknown to the denoising neural network of the present invention; the best existing denoising neural network solutions do not yet have this generalization.

[0040] 2) In terms of image denoising capability, the present invention achieves the following: regardless of whether the noise level of the noisy image is high or low, the denoised image is closer to the noise-free image in terms of both objective evaluation indicators (PSNR, SSIM) and subjective evaluation than the best existing single-domain denoising neural network. The present invention has the best image denoising capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an overall flow chart of the method of the present invention;

[0042] Figure 2 It is a specific training flow chart of the method of the present invention;

[0043] Figure 3 Schematic diagram of the simulated noise model of the present invention, where (a) is the joint probability density distribution function and (b) is the approximate probability density distribution function of R;

[0044] Figure 4 Schematic diagram of the noise prior fusion mechanism of the present invention;

[0045] Figure 5 is a schematic diagram of an adjustable tone mapping module of the present invention;

[0046] Figure 6 This is a test result diagram obtained under different noise levels K and gain coefficients α of the present invention;

[0047] Figure 7 This is a raw image test result diagram of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0049] The present invention introduces a denoising network in both the raw domain and the sRGB domain, uses the noise distribution prior in the raw domain to perform preliminary denoising, and removes the residual noise amplified by the image signal processing link in the subsequent sRGB domain. A differentiable image signal processing link is designed to connect the dual-domain image denoising network for end-to-end training, thereby overcoming the shortcomings of the single-domain denoising network and significantly improving the denoising performance. In addition, the present invention introduces input images with different noise levels and a varying image signal processing link during the training process, reducing the objective condition differences that may exist during training and inference, and significantly improving the generalization ability of the system. This dual-domain joint denoising method enables the denoising network trained on simulated images of the present invention to achieve plug-and-play high-quality denoising performance on unseen intelligent imaging devices, and has good practical capabilities.

[0050] The dual-domain joint image denoising method of the present invention covers two stages: training process and reasoning process. Figure 1 As shown in the figure. During the training phase, the present invention does not require the preparation of a real image set. Instead, it simply generates a purely simulated image set using a given simulated noise model and trains the dual-domain image denoising network of the present invention. During the inference phase, the present invention uses the trained dual-domain image denoising network and directly applies it to the raw and sRGB domains of the unknown camera's image signal processing chain, achieving excellent image denoising results.

[0051] The specific training process is as follows Figure 2 As shown, it includes four parts: simulated image generation, raw domain noise reduction, sRGB domain noise reduction, and differentiable image signal processing link. Given a noise-free raw image data set, the present invention first uses a simulated noise model to generate a large number of simulated noisy raw images, and uses a differentiable image signal processing link to convert the noise-free raw images into noise-free sRGB images for subsequent verification. Then, the noisy raw image is input into the raw domain noise reduction network to obtain the output raw image. The output raw image is processed through the differentiable image signal processing link to obtain the corresponding sRGB image, which is input into the sRGB domain noise reduction network to obtain the output sRGB image. Finally, the image quality difference between the output sRGB image and the noise-free sRGB image is verified.

[0052] In the training process, the simulated noise model is as shown in the attached Figure 3 Here, the present invention defines the noise-free raw image as R * The generated noisy raw image is defined as R, and the Gaussian-Poisson distribution is used to construct the noise model. The simulated noise model distribution is as follows:

[0053]

[0054] Among them, K is the total gain coefficient of the imaging system, which can represent the noise level of the camera imaging, σ r2 is the variance of the Gaussian noise. and Represent Poisson and Gaussian distributions respectively. Using the above simulation noise model, the present invention can use a single noise-free raw image R * , generating a noisy raw image R with random simulated noise levels. In practice, the simulated noise model can adopt a more appropriate form according to the actual situation. The Gaussian-Poisson distribution used here is just a general expression. The coefficients K and σ that control the simulated noise level are r 2 Obey the joint probability density distribution:

[0055]

[0056]

[0057] in, and Represent uniform and Gaussian distributions respectively, K min / K max Represent the minimum / maximum noise level obtained by the noise model, log(σ r 2 ) is approximately linearly related to log(K), through the slope α, intercept β and Gaussian variable σ 2 To control. Among them, the coefficient K min , K max , α, β, σ can be obtained by performing noise calibration on a specific device in the actual processing process. Here, for the sake of detailed description, the present invention intends to take α = 2.540, β = 1.218, σ = 0.268 to simulate the situation of most real scenes. In the training stage, the present invention takes K min =0.0002, K max =0.02, and let K be randomly and uniformly sampled, thereby simulating the raw image R with random noise.

[0058] After obtaining the simulated noisy raw image R, the present invention first uses channel transformation to transform it into a four-channel raw image and input it into the raw domain denoising neural network to obtain a four-channel output raw image, and then performs channel inverse transformation to obtain a single-channel output raw image. The output raw image is converted into the input sRGB image of the sRGB domain denoising neural network after passing through the differentiable image signal processing link, and the output sRGB image is obtained after further denoising. The denoising networks in the raw domain and sRGB domain are both equipped with a noise prior fusion module, which fuses the noise prior and the input image to obtain the fused feature vector. The specific flow chart is shown in the attached figure. Figure 4 shown.

[0059] As a connecting module in the entire process, the differentiable image signal processing chain designed by the present invention includes black level correction, normalization, white balance, demosaicing, color space conversion, gamma correction, and adjustable tone mapping, a total of seven differentiable modules, which are responsible for the conversion process from raw image to sRGB image. For the demosaicing module, the present invention designs a simplified differentiable version of the Adaptive Homogeneity-Directed (AHD) demosaicing algorithm. For the adjustable tone mapping module, the present invention uses a quadratic curve whose curvature is controlled by the gain coefficient α, and the formula is expressed as:

[0060] R n =R n-1 +α·R n-1 (1-R n-1 ),

[0061] Among them, R n Represents the tone mapping result after n iterations, R0 is the original image, and α represents the gain coefficient. Here, the present invention adopts the quadratic curve form, that is, n=2. The specific curve form is shown in the attached figure. Figure 5 The larger the gain coefficient α, the greater the overall brightening gain of the image. In the training process, the present invention controls α to randomly take values within a given range [0, 1], so that the trained dual-domain image denoising network can adapt to a variety of gain coefficients, thereby improving the generalization of the denoising network.

[0062] Finally, in order to automatically optimize the dual-domain image denoising network, the present invention uses the L1 loss function for gradient backpropagation. Using the obtained output raw image and output sRGB image, the loss function used by the present invention is as follows:

[0063] L=λ·||RR * ||1+||II * ||1,

[0064] Among them, ||||1 represents the L1 loss function, R, R * ,I,I * They represent the output raw image, the noise-free raw image, the output sRGB image, and the noise-free sRGB image, respectively. Here, the coefficient λ is set to 1.

[0065] During training, the present invention uses raw images from the MIT-Adobe FiveK dataset as noise-free raw images for training, and randomly selects 200 of them for training and 20 for testing. Each noise-free raw image has a high resolution of 4K. The present invention randomly crops them into square image blocks with a length and width of 128 pixels for training and uses the AdamW optimizer for training. The initial parameters of the optimizer are β1=0.9, β2=0.999, and weight decay=1e -2 The total number of training rounds is 120,000, each batch of input image blocks contains 2 images, and the initial learning rate is 1e -5 , and when the number of training rounds is [40, 60, 80, 100,000], the learning rate becomes 0.6 times the learning rate of the previous round.

[0066] The present invention has the following key innovations:

[0067] The proposed dual-domain image denoising network's training and inference processes are key. This network is trained entirely on simulated images, enabling high-quality denoising on any real-world image. Previous image denoising solutions have been unable to achieve such high generalization.

[0068] 2. This paper proposes for the first time to place the denoising network in the raw domain and the sRGB domain respectively, and for the first time to propose a training scheme for simultaneously and jointly optimizing the dual-domain image denoising network, which has been proven to have better denoising capabilities than all previous single-domain denoising schemes.

[0069] 3. The adjustable noise prior fusion mechanism used in the dual-domain image denoising network of the present invention is the first solution proposed and proven to effectively improve the denoising capability of the dual-domain image denoising network.

[0070] 4. The entire differentiable image signal processing link designed by the present invention is a complete set of Python language codes designed and written by ourselves. Compared with previous solutions, it has realized the characteristics of differentiability, variability, and fast inference speed for the first time, and realized end-to-end optimization of the dual-domain network.

[0071] The present invention will be described in detail below with reference to the accompanying drawings:

[0072] like Figure 1As shown, the overall process of the present invention consists of two phases: training and inference. During the training phase, the present invention uses a given simulated noise model to generate simulated noisy images, which are then input into the dual-domain joint image denoising system of the present invention for training. During the testing phase, the present invention utilizes the trained dual-domain image denoising network and applies it to the raw and sRGB domains of the unknown camera's image signal processing chain for inference, achieving ideal image denoising results.

[0073] like Figure 2 As shown, the training process specifically includes four parts: (1) simulated image generation, (2) raw domain denoising, (3) sRGB domain denoising and (4) differentiable image signal processing link. In the first part, the present invention first extracts raw parameters from the noise-free raw image, and uses the predefined simulated noise model and differentiable image signal processing link of the present invention to generate a noisy input raw image and a noise-free sRGB image with random noise levels respectively. Among them, the input raw image is used as the input of the second part, and the noise-free sRGB image is used as the true value to calculate the loss function of the gradient return and verify the quality of the output image of the third part. In the second part, the present invention first transforms the single-channel input raw image into a four-channel input raw image through a channel transformation operation, and then inputs the four-channel input raw image into a denoising network with a noise prior fusion mechanism and jump connection to obtain a four-channel output raw image. Finally, the four-channel output raw image is transformed into a single-channel output raw image through a channel inverse transformation operation. In Part 3, the present invention uses the input sRGB image to directly input a denoising network with a noise prior fusion mechanism and jump connection to obtain the output sRGB image, and calculates the loss function of the output sRGB image and the noise-free sRGB image and verifies the image quality difference. Part 4 serves as the conversion module between the raw domain and the sRGB domain in the entire process, and includes a series of differentiable image signal processing modules (black level correction, normalization, white balance, de-mosaicing, color space conversion, gamma correction, and adjustable tone mapping), forming an entire differentiable image signal processing chain. In the entire process, Part 4 not only converts the noise-free raw image into a noise-free sRGB image in Part 1; it also connects Parts 2 and 3, and converts the output raw image of Part 2 into the input sRGB image of Part 3.

[0074] like Figure 3 As shown in the figure, it is an example of a simulated noise model. The present invention uses a noise-free raw image R * Generate a noisy raw image R, R obeys the Gaussian-Poisson distribution: As shown in Figure (b), the coefficients K and σ that control the noise level r 2 Obey the joint probability density distribution: As shown in Figure (a), during the training process, the present invention controls K to randomly select values within a given range of [0.0002, 0.2] to generate a noisy raw image R with random level noise. In actual practice, the simulation noise model and the setting and range of noise parameters can adopt more appropriate forms based on actual conditions. This is only an example of the invention process and does not represent the only feasible form.

[0075] like Figure 4 As shown, it is a noise prior fusion mechanism, because the coefficients K and σ related to the noise level r 2 It is known in the training stage and can also be obtained through actual calibration in the inference stage. The present invention designs a noise prior fusion mechanism to improve the noise reduction ability. Using the input raw image R and the coefficients K and σ to control the noise level r 2 Get the noise prior Finally, it is input into the noise prior fusion module. In the noise prior fusion module, the noise prior passes through a domain selection branch, 1×1 convolution, and 3×3 convolution in sequence to obtain a feature vector of size HW×N (H is length, W is width, and N is the number of channels). The input image also passes through 1×1 convolution and 3×3 convolution in sequence to obtain a feature vector of size HW×N (H is length, W is width, and N is the number of channels). The two feature vectors are superimposed by channels to obtain a feature vector of size HW×2N. Among them, the domain selection branch uses a direct connection method in the raw domain and a differentiable image signal processing link connection method in the sRGB domain. After passing through the noise fusion prior module, the feature vector obtained by channel superposition is transformed into a high-dimensional feature vector of HW×C through 1×1 convolution and input into the denoising network.

[0076] like Figure 5 The figure shows an adjustable tone mapping module. The present invention uses a quadratic curve whose curvature is controlled by a gain coefficient α. The left side of the figure shows a schematic diagram of the tone-mapped quadratic curve. The horizontal axis corresponds to the pixel values of the original image R0, and the vertical axis corresponds to the pixel values of the tone-mapped image R2. The larger the gain coefficient α, the steeper the curve. The right side of the figure shows a portion of the tone-mapped image R, at different gain coefficients α. The larger the gain coefficient α, the brighter the overall image brightness.

[0077] like Figure 6 As shown in the figure, the test results are obtained under different noise levels K and gain coefficients α. From the above result comparison diagram, it can be seen that the result obtained by the present invention is closest to the noise-free image, and there is almost no residual noise and image artifacts that affect the visual quality.

[0078] like Figure 7As shown in the figure, the sRGB result image is obtained by inferring the raw image. As can be seen from the comparison of the above results, although the solution of the present invention is trained entirely on a simulated dataset, the sRGB result image processed by the solution of the present invention is nearly noise-free. It also has rich details and high contrast, which is superior to the results of the single-domain noise reduction solution and even surpasses the results of the mobile phone's native processing.

[0079] After verification by experiments and simulation data sets, the dual-domain joint image denoising method based on neural networks in the present invention has the best image denoising ability and generalization ability to be applied in real scenarios. It is the most promising image denoising neural network solution.

[0080] In the experiment to test the denoising ability, the present invention calls three of the best existing denoising neural networks and conducts inference tests on a simulation data set constructed with random noise level K∈[0.0002, 0.02] and gain coefficient α∈[0, 1]. The results obtained by the dual-domain joint image denoising neural network proposed in the present invention comprehensively surpass the previous single-domain denoising network. The results are shown in Tables 1 and 2 and in the attached table. Figure 6 middle.

[0081] In the experiment of testing the generalization ability, the present invention first tested the results of the test on the simulation image set when the training conditions and the objective conditions of reasoning did not match, as shown in Appendix 3 and 4. The present invention selected a smaller range of noise level K and gain coefficient α during training, but tested under a larger range. The results in the table show that the solution of the present invention has better generalization under mismatched reasoning conditions. Then, the present invention used flagship mobile phones of different brands (Apple, Xiaomi, Huawei) to shoot multiple groups of real raw pictures with different sensitivity settings, and tested the generalization ability of the denoising model. As shown in the Appendix Figure 7 As shown in the figure, the visual results obtained by the dual-domain image denoising network of the present invention not only surpass the previous single-domain denoising scheme, but are also better than the result images obtained after the image processing link built into the flagship mobile phone, which is specifically reflected in less noise, artifacts, and clearer details and colors.

[0082] Table 1

[0083]

[0084] Table 1 shows the noise reduction capability under different noise levels K. In the table, SwinIR, MIRNet-v2, and Restormer are three different noise reduction networks, "Raw Domain" represents noise reduction only in the raw domain, "sRGB Domain" represents noise reduction only in the sRGB domain, and "Invention" represents the dual-domain joint image denoising system proposed in this invention. The experiments in the table demonstrate that the proposed solution achieves the best objective evaluation metrics (PSNR, SSIM) and exhibits the best noise reduction capability under different noise levels K.

[0085] Table 2

[0086]

[0087]

[0088] Table 2 shows the test of noise reduction capability under different gain coefficients α. In the table, SwinIR, MIRNet-v2, and Restormer are three different noise reduction networks, "Raw domain" represents noise reduction only in the raw domain, "sRGB domain" represents noise reduction only in the sRGB domain, and "the present invention" represents the dual-domain joint image denoising system proposed in the present invention. The experiments in the table show that the scheme proposed in the present invention achieves the best objective evaluation indicators (PSNR, SSIM) and has the best noise reduction capability under different gain coefficients α.

[0089] Table 3

[0090]

[0091] In Table 3, we fixed K∈[0.0004,0.01] when training the dual-domain image denoising network. However, during inference, we selected K=0.0002,0.02, outside the training range, to test the generalization ability of the denoising network. The experiments in the table demonstrate that our proposed solution achieves the best objective evaluation metrics (PSNR, SSIM) and maintains the best generalization ability under different noise levels K.

[0092] Table 4

[0093]

[0094] Table 4 shows that the proposed method maintains a fixed value of α∈[0.4,0.6] when training the dual-domain image denoising network. However, during inference, α=0.2,0.8, outside the training range, was selected to test the generalization capability of the denoising network. The experiments in the table demonstrate that the proposed method achieves the best objective evaluation metrics (PSNR, SSIM) and maintains the best generalization capability under different gain coefficients α.

[0095] Alternative embodiment

[0096] The training and inference processes proposed in this invention: training on a simulated image set using a realistic noise model, and inference testing on both simulated and real-world image sets, are intended solely to maximize resource conservation. Any use of this solution to implement other training and inference processes should be considered a "circumvention of this technology to achieve the same invention objectives." For example, completing training and inference testing entirely on a simulated image set or a real-world image set, or completing training on a real-world image set and inference testing on a simulated image set, etc.

[0097] The dual-domain joint image denoising method proposed in this invention is an integrated whole. Using other technical means to implement the four components mentioned in this invention should be considered as "circumventing this technology to achieve the same purpose of the invention." First, the present invention's solution achieves simulated image synthesis by synthesizing Gaussian Poisson noise onto a noise-free image. However, there are many ways to synthesize noisy images, and any other method of synthesizing noisy images should be considered as the "simulated image synthesis" proposed in this invention. Second and third, the present invention's solution incorporates image denoising networks in both the raw and sRGB domains. In inference testing, the present invention only uses three denoising neural networks, but there are many types of denoising neural networks. Using any other denoising neural network (or using different neural networks in the raw and sRGB domains) should be considered as the "dual-domain image denoising network" proposed in this invention. Fourth, the present invention proposes the construction of a complete differentiable image signal processing chain. However, the modules and order of modules in this chain can vary greatly. This present invention only uses one solution for training; any other image signal processing chain should be considered as the "differentiable image signal processing chain" proposed in this invention.

[0098] Because the present invention is a dual-domain joint image denoising method based on a neural network, and neural networks have been proven to be applicable to multiple image tasks in recent years, such as super-resolution, deblurring, rain and fog removal, image completion, etc., some feasible alternatives may include: using the framework of the dual-domain joint image denoising system to achieve some of the other tasks mentioned above, rather than just the image denoising solution as described in the present invention.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A dual-domain joint image denoising method based on neural network, characterized in that: The method includes: During the training phase, a pure simulated image set is generated using a given simulated noise model to train the dual-domain image denoising network; During the inference phase, the trained dual-domain image denoising network is directly applied to the denoising positions of the raw and sRGB domains of any camera's image processing to perform image denoising. The dual-domain image denoising network is specifically: A denoising network is introduced in both the raw and sRGB domains. The noise distribution prior in the raw domain is used for preliminary noise reduction, and the residual noise amplified by the image signal processing chain is subsequently removed in the sRGB domain. The training phase process specifically includes: Step S101: Generate a simulated noisy raw image using a simulated noise model, and convert the noise-free raw image into a noise-free sRGB image for subsequent calculation of the loss function using a differentiable image signal processing link; Step S102: The noisy raw image is input into a raw domain denoising network to obtain an output raw image. The output raw image is processed through a differentiable image signal processing chain to obtain a corresponding sRGB image, which is then input into an sRGB domain denoising network to obtain an output sRGB image. Step S103, calculating the loss function values of the output raw image and the noise-free raw image, and the output sRGB image and the noise-free sRGB image, and performing gradient backpropagation to simultaneously optimize the raw domain denoising network and the sRGB domain denoising network; The simulation noise model in step S101 is specifically as follows: Where K is the total gain coefficient of the imaging system, R * is the noise-free raw image, R is the generated noisy raw image, σ r 2 is the variance of the Gaussian noise, and represent Poisson and Gaussian distributions, respectively.

2. The dual-domain joint image denoising method based on a neural network according to claim 1, characterized in that: The total gain coefficient K and variance σ of the imaging system r 2 Obey the joint probability density distribution, as follows: in and Represent uniform and Gaussian distributions respectively, K min / K max Represent the minimum / maximum noise level obtained by the noise model, log(σ r 2 ) is the Gaussian noise variance σ r 2 The logarithmic function of log(K) is the logarithmic function of the total gain coefficient K of the imaging system, log(σ r 2 ) is approximately linearly related to log(K), through the slope α, intercept β and Gaussian variable σ 2 to control.

3. The dual-domain joint image denoising method based on neural network according to claim 1, characterized in that: The step S102 specifically includes the following steps: Step S1021: transform the single-channel noisy raw image into a four-channel noisy raw image using channel transformation and input it into the raw domain denoising neural network to obtain a four-channel output raw image; Step S1022: The four-channel output raw image is subjected to channel inverse transformation to obtain a single-channel output raw image; In step S1023, the single-channel output raw image is converted into an input sRGB image of the sRGB domain denoising neural network after passing through the differentiable image signal processing link, and the output sRGB image is obtained after further denoising.

4. The dual-domain joint image denoising method based on neural network according to claim 1, characterized in that: The raw domain denoising network and the sRGB domain denoising network are both equipped with a noise prior fusion module for fusing the noise prior with the input image and obtaining a fused feature vector.

5. The dual-domain joint image denoising method based on neural network according to claim 1, characterized in that: The differentiable image signal processing chain includes a black level correction module, a normalization module, a white balance module, a demosaicing module, a color space conversion module, a gamma correction module and an adjustable tone mapping module, which is responsible for the conversion process from raw images to sRGB images.

6. The dual-domain joint image denoising method based on neural network according to claim 5, characterized in that: The demosaicing module uses a differentiable version of the adaptive homogeneity-guided demosaicing algorithm; the adjustable tone mapping module uses a quadratic curve whose curvature is controlled by the gain coefficient α, which is expressed as follows: R n =R n-1 +α·R n-1 (1-R n-1 ), Among them, R n Represents the tone mapping result image after n iterations, R0 is the original image, and α represents the gain coefficient.

7. The dual-domain joint image denoising method based on neural network according to claim 1, characterized in that: This method uses the L1 loss function to do gradient backpropagation, where the loss function is as follows: L=λ·‖R-R * ‖1+‖I-I * ‖1, Among them, ‖‖1 represents the L1 loss function, R, R * ,I,I * They represent the output raw image, noise-free raw image, output sRGB image, and noise-free sRGB image respectively, and λ represents the adjustment coefficient of raw domain supervision in the loss function.

Images