Noise model acquisition method, denoising neural network training method and device, electronic equipment and storage medium

Abstract

The invention discloses a noise model obtaining method, a denoising neural network training method, a noise model obtaining device, a denoising neural network training device, electronic equipment and a computer readable storage medium, and the noise model obtaining method comprises the steps: shooting a flat field frame under different sensitivities and different light incoming amounts, modeling shot noise NP, and obtaining a noise model NP; the relation between the ISO and the system gain K is calibrated; shooting dark field frames under different ISOs, and modeling black level error noise NBLE to calibrate the black level error noise under different ISOs; modeling row noise Nrow and calibrating the relationship between different ISO and the noise parameter sigma row; modeling a fixed pattern noise (NFPN); modeling and reading noise Nread, and calibrating the relationship between different ISO and the shape parameter lambda and the noise intensity sigma TL; and obtaining a target noise model according to the model of the NP, the relationship between the ISO and the system gain K, the model of the NBLE, the model of the Nrow, the relationship between the different ISO and the sigma row, the model of the NFPN, the model of the Nread and the relationship between the different ISO and the shape parameters lambda and sigma TL.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for acquiring a noise model, a method for training a denoising neural network, a device for acquiring a noise model, a device for training a denoising neural network, an electronic device, and a computer-readable storage medium. Background Art

[0002] Electronic devices with shooting functions, such as mobile phones, PADs, and laptops, generally use image processors (ISPs) to perform various processing on the actual digital image signals. A general ISP includes multiple links, such as image denoising, color interpolation, color correction, white balance, and gamma conversion. Image denoising is a very important link. Since each link in the ISP will cause some noise to be generated or the existing noise to be amplified, most ISPs will design multiple denoising modules to remove the noise generated by different links.

[0003] In recent years, more attention has been paid to the denoising of the signal directly output by the image sensor (also called RAW domain signal or RAW domain image), because at this time the signal has only passed through a small part of the ISP link, the noise modeling is relatively accurate, and it is easier to analyze the noise characteristics. At the same time, since the RAW domain image appears at a relatively early position in the ISP process, sufficient noise removal in this part can effectively reduce the impact of noise amplification on image quality in subsequent processes.

[0004] Various noises are introduced into RAW domain images during the imaging process of the image sensor, including noise related to the light signal (for example, shot noise) and noise unrelated to the light signal (for example, noise caused by the physical mechanism of the image sensor). Currently, when denoising RAW domain images, the noise unrelated to the light signal is not modeled, resulting in low denoising accuracy of RAW domain images and inability to obtain high-quality RAW domain images. Summary of the invention

[0005] Embodiments of the present application provide a method for acquiring a noise model, a method for training a denoising neural network, a device for acquiring a noise model, a device for training a denoising neural network, an electronic device, and a computer-readable storage medium.

[0006] In a first aspect, a method for obtaining a noise model according to an embodiment of the present application includes:

[0007] Shooting flat-field frames at different ISOs and different amounts of incoming light to model shot noise N P , and calibrate the relationship between ISO and system gain K;

[0008] At different ISOs, capture dark field frames and perform the following operations:

[0009] Modeling black level error noise N BLE , to calibrate the black level error noise under different ISO;

[0010] Modeling row noise N row , and calibrate different ISO and noise parameters σ row the relationship between;

[0011] Modeling fixed pattern noise N FPN , the fixed pattern noise N FPN including a fixed pattern of dark current noise and a fixed pattern of column noise, wherein the dark current noise is related to ISO;

[0012] Modeling readout noise N read , and calibrate different ISO with shape parameter λ and noise intensity σ TL the relationship between them; and

[0013] According to the shot noise N P The model, the relationship between the ISO and the system gain K, the black level error noise N BLE The model, the row noise N row The model, the different ISO and noise parameters σ row The relationship between the fixed pattern noise N FPN The model, the read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

[0014] In a second aspect, a noise model acquisition device according to an embodiment of the present application includes a shooting module, a first calibration module, a second calibration module, a third calibration module, a fourth calibration module, a fifth calibration module and an acquisition module;

[0015] The shooting module is used to shoot flat-field frames at different sensitivities (ISO) and different amounts of incoming light; the shooting module is also used to shoot dark-field frames at different ISOs;

[0016] The first calibration module is used to model the shot noise N according to the flat field frame P , and calibrate the relationship between ISO and system gain K;

[0017] The second calibration module is used to model the black level error noise N according to the dark field frame BLE , to calibrate the black level error noise under different ISO;

[0018] The third calibration module is used to model the row noise N according to the dark field frame row , and calibrate different ISO and noise parameters σ row the relationship between;

[0019] The fourth calibration module is used to model the fixed pattern noise N according to the dark field frame FPN , the fixed pattern noise N FPN including a fixed pattern of dark current noise and a fixed pattern of column noise, wherein the dark current noise is related to ISO;

[0020] The fifth calibration module is used to model the readout noise N according to the dark field frame read , and calibrate different ISO with shape parameter λ and noise intensity σ TL the relationship between;

[0021] The acquisition module is used to obtain the shot noise N P The model, the relationship between the ISO and the system gain K, the black level error noise N BLE The model, the row noise N row The model, the different ISO and noise parameters σ row The relationship between the fixed pattern noise N FPN The model, the read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

[0022] In a third aspect, the training method of the denoising neural network according to the embodiment of the present application comprises:

[0023] Obtain multiple reference images;

[0024] Acquire noise images corresponding to the plurality of reference images according to the plurality of reference images and a preset target noise model to obtain a set of paired reference-noise images, wherein in each pair of reference-noise images, the noise of the reference image is smaller than the noise of the noise image, and the target noise model is obtained by the above-mentioned noise model acquisition method; and

[0025] The reference-noise image set is used to train a neural network to obtain a denoising neural network.

[0026] In a fourth aspect, a denoising neural network training device according to an embodiment of the present application includes:

[0027] A first acquisition unit, the first acquisition unit is used to acquire a plurality of reference images;

[0028] a second acquisition unit, the second acquisition unit being used to acquire noise images corresponding to the plurality of reference images according to the plurality of reference images and a preset target noise model, so as to acquire a set of paired reference-noise images, wherein in each pair of reference-noise images, the noise of the reference image is smaller than the noise of the noise image, and the target noise model is acquired by the above-mentioned noise model acquisition method; and

[0029] A training unit is used to train a neural network using the reference-noise image set to obtain a denoising neural network.

[0030] In a fifth aspect, the electronic device of an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned method for acquiring the noise model, or executes the above-mentioned method for training the noisy neural network.

[0031] In a sixth aspect, a computer-readable storage medium of an embodiment of the present application stores a computer program thereon, characterized in that when the computer program is executed by a processor, it implements the above-mentioned method for acquiring a noise model, or implements the above-mentioned method for training a noisy neural network.

[0032] The noise model acquisition method, denoising neural network training method, noise model acquisition device, denoising neural network training device, electronic device and computer readable storage medium of the present invention are not only for shot noise N P Modeling is also performed for the row noise N row , black level error noise N BLE , fixed pattern noise N FPN and read noise N read All are modeled. Among them, for the row noise N row Modeling can effectively avoid the existence of horizontal stripe artifacts in the denoised image; for the black level error noise N BLE Modeling can effectively reduce color deviation. For fixed pattern noise N FPN and read noise N read Modeling, adding a denoising network for fixed pattern noise N FPN and read noise N read The processing capability of the target noise model is more accurate, which can improve the quality of the denoised image after denoising.

[0033] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 It is a flowchart of a method for acquiring a noise model and a method for training a denoising neural network in certain embodiments of the present application;

[0036] Figure 2 is a schematic diagram of the structure of a noise model acquisition device in some embodiments of the present application;

[0037] Figure 3 The shot noise N modeled in the noise model acquisition method of certain embodiments of the present application P , and calibrate the flow chart of the relationship between ISO and system gain K;

[0038] Figure 4 It is a schematic diagram of the principle of acquiring two flat-field frames in some embodiments of the present application;

[0039] Figure 5 It is a schematic diagram of the principle of obtaining a difference image of two flat-field frames in some embodiments of the present application;

[0040] Figure 6 It is a schematic diagram of the principle of multiple mean Mean and variance Var relationship pairs in certain implementation modes of the present application;

[0041] Figure 7 is a schematic diagram of a Mean-Var curve of certain embodiments of the present application;

[0042] Figure 8 is a schematic diagram of the corresponding relationship between multiple ISOs and gain K in some embodiments of the present application;

[0043] Fig. 9 It is a schematic diagram of the principle of acquiring two dark field frames in some embodiments of the present application;

[0044] Fig.10 is the modeled black level error noise N of certain embodiments of the present application BLE , a flow chart of calibrating black level error noise under different ISO;

[0045] Fig.11 is the modeled black level error noise N of certain embodiments of the present application BLE , a schematic diagram of the principle of calibrating the black level error noise under different ISOs;

[0046] Fig.12 is the modeled row noise N of certain embodiments of the present application row , and calibrate different ISO and noise parameters σ rowA flowchart of the relationship between

[0047] Fig.13 is the modeled row noise N of certain embodiments of the present application row , and calibrate different ISO and noise parameters σ row Schematic diagram of the relationship between

[0048] Fig.14 is the modeled fixed pattern noise N of certain embodiments of the present application FPN Schematic diagram of the process;

[0049] Fig.15 is the modeled fixed pattern noise N of certain embodiments of the present application FPN Schematic diagram of the principle;

[0050] Fig.16 is the modeled fixed pattern noise N of certain embodiments of the present application FPN Schematic diagram of the principle;

[0051] Fig.17 is the modeled fixed pattern noise N of certain embodiments of the present application FPN Schematic diagram of the principle;

[0052] Fig.18 is the modeled fixed pattern noise N of certain embodiments of the present application FPN Schematic diagram of the principle;

[0053] Fig.19 is the modeled read noise N of certain embodiments of the present application read , and calibrate different ISO with shape parameter λ and noise intensity σ TL A flowchart of the relationship between

[0054] Fig. 20 It is a schematic diagram of the structure of a training device for a denoising neural network according to certain embodiments of the present application;

[0055] Fig.21 is a flowchart of a training method for a denoising neural network in certain embodiments of the present application;

[0056] Fig. 22 is a schematic diagram of denoising a Raw image using a denoising neural network in certain embodiments of the present application;

[0057] Fig.23 It is a comparison diagram of a clear image, an image denoised using the denoising neural network of an embodiment of the present application, and an image denoised using other noise models;

[0058] Fig.24 It is a schematic diagram of the structure of an electronic device of certain embodiments of the present application.

[0059] Description of main component symbols:

[0060] A noise model acquisition device 10, a shooting module 17, a first calibration module 11, a second calibration module 12, a third calibration module 13, a fourth calibration module 14, a fifth calibration module 15, an acquisition module 16, a denoising neural network training device 30, a first acquisition unit 31, a second acquisition unit 33, and a training unit 35. DETAILED DESCRIPTION

[0061] The following is a further description of the embodiments of the present application in conjunction with the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements or elements with the same or similar functions from beginning to end. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the present application.

[0062] RAW domain images will introduce various noises during the imaging process of the image sensor, including noise related to the light signal (for example, shot noise) and noise unrelated to the light signal (for example, noise caused by the physical mechanism of the image sensor). At present, when denoising RAW domain images, the noise unrelated to the light signal is not modeled, resulting in low denoising accuracy of RAW domain images and failure to obtain high-quality RAW domain images. In order to solve this problem, the present application provides a method for obtaining a noise model ( Figure 1 As shown), the training method of denoising neural network ( Figure 1 As shown), a noise model acquisition device ( Figure 2 As shown), a training device for a denoising neural network ( Fig. 20 As shown), electronic equipment ( Fig.24 ) and a computer readable storage medium ( Fig.24 shown).

[0063] See also Figure 1 , the method for obtaining the noise model in the embodiment of the present application includes:

[0064] 01: Shooting flat-field frames at different ISOs and different amounts of incoming light to model shot noise N P , and calibrate the relationship between ISO and system gain K;

[0065] At different ISOs, capture dark field frames and perform the following operations:

[0066] 02: Modeling black level error noise N BLE , to calibrate the black level error noise under different ISO;

[0067] 03: Modeling row noise N row, and calibrate different ISO and noise parameters σ row the relationship between;

[0068] 04: Modeling fixed pattern noise N FPN , fixed pattern noise N FPN Includes fixed-mode dark current noise and fixed-mode column noise. Dark current noise is related to ISO.

[0069] 05: Modeling readout noise N read , and calibrate different ISO with shape parameter λ and noise intensity σ TL the relationship between them; and

[0070] 06: According to the shot noise N P The model, the relationship between ISO and system gain K, the black level error noise N BLE Model, row noise N row Model, different ISO and noise parameter σ row The relationship between the fixed pattern noise N FPN Model, read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

[0071] Please combine Figure 2 The noise model acquisition method of the embodiment of the present application can be applied to the noise model acquisition device 10. Specifically, the noise model acquisition device 10 includes a shooting module 17, a first calibration module 11, a second calibration module 12, a third calibration module 13, a fourth calibration module 14, a fifth calibration module 15 and an acquisition module 16. The shooting module 17 is used to shoot flat field frames at different sensitivities (ISO) and different amounts of incoming light. The shooting module 17 is also used to shoot dark field frames at different ISOs. The first calibration module 11 is used to model the shot noise N according to the flat field frame. P , and calibrate the relationship between ISO and system gain K. The second calibration module 12 is used to model the black level error noise N according to the dark field frame BLE , to calibrate the black level error noise under different ISO. The third calibration module 13 is used to model the row noise N according to the dark field frame row , and calibrate different ISO and noise parameters σ row The fourth calibration module 14 is used to model the fixed pattern noise N according to the dark field frame FPN , fixed pattern noise N FPN The fifth calibration module 15 is used to model the readout noise N according to the dark field frame. read, and calibrate different ISO with shape parameter λ and noise intensity σ TL The acquisition module 16 is used to obtain the relationship between the shot noise N P The model, the relationship between ISO and system gain K, the black level error noise N BLE Model, row noise N row Model, different ISO and noise parameter σ row The relationship between the fixed pattern noise N FPN Model, read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

[0072] Among them, "ISO" is the sensitivity of the photosensitive element of the image sensor to light. The larger the ISO value, the higher the sensitivity, and the more sensitive the photosensitive element is to light; the smaller the ISO value, the lower the sensitivity, and the lower the sensitivity of the photosensitive element to light. ISO values ​​are directly expressed in numbers, such as: 100, 200, 400, 800, etc. The smaller the number, the lower the sensitivity, and the larger the number, the higher the sensitivity. ISO affects the exposure of the image sensor and the quality of the image captured by the image sensor. Generally, under the same other shooting conditions, the higher the sensitivity, the brighter the image, and the lower the sensitivity, the darker the image. On the other hand, under the same other shooting conditions, the higher the sensitivity, the lower the signal-to-noise ratio, and the worse the image quality (the more noise / noise on the image).

[0073] "Light intake" refers to the amount of light that hits the photosensitive element. The amount of light intake is related to the aperture size. Usually, when other shooting conditions remain unchanged, the larger the aperture, the greater the amount of light intake.

[0074] A "flat field frame" is an image frame obtained by photographing an object with uniform brightness, such as white paper or a white board. Figure 4 As shown. "Shooting flat field frames at different sensitivities (ISO) and different amounts of incoming light" means: for example, shooting at least two flat field frames at ISO1 and an amount of incoming light Q1, shooting at least two flat field frames at ISO1 and an amount of incoming light Q2, shooting at least two flat field frames at ISO1 and an amount of incoming light Q3, and continuing to maintain the ISO1 value while changing the amount of incoming light Q to shoot at least two flat field frames. Then, the ISO value is changed to repeat the previous operation, that is, shooting at least two flat field frames at ISO2 and an amount of incoming light Q1, shooting at least two flat field frames at ISO2 and an amount of incoming light Q2, shooting at least two flat field frames at ISO2 and an amount of incoming light Q3, and continuing to maintain the ISO2 value while changing the amount of incoming light Q to shoot at least two flat field frames. In this way, multiple flat field frames are obtained.

[0075] A “dark field frame” refers to an image frame obtained by shooting in a dark environment, for example, when the lens cover of the shooting module 17 is closed, and the amount of light entering Q at this time is 0. Fig. 9 "Shooting dark field frames at different ISOs" means: for example, shooting a dark field frame at ISO1, shooting a dark field frame at ISO2, shooting a dark field frame at ISO3, and so on, to obtain multiple dark field frames.

[0076] In addition to the shot noise N P Other noises, such as black level error noise N BLE , row noise N row , fixed pattern noise N FPN , read noise N read All of them are noises unrelated to the light signal and are caused by the physical mechanism of the shooting module 17. In the shooting scene with extremely dark light, if the row noise N is ignored, row , it will cause the restoration result (i.e. denoised image) to contain horizontal stripe artifacts. If the black level error noise N is ignored BLE , which will lead to a certain color deviation in the restored result. The current noise model does not model the noise unrelated to the light signal in detail. There is still noise unrelated to the light signal that is ignored or the correlation between noise and ISO is not considered, which leads to the inaccuracy of the noise model and the low quality of the denoised image after denoising using the noise model.

[0077] The noise model acquisition method and noise model acquisition device 10 of the embodiment of the present application are not only for the shot noise N P Modeling is also performed for the row noise N row , black level error noise N BLE , fixed pattern noise N FPN and read noise N read All are modeled. Among them, for the row noise N row Modeling can effectively avoid the existence of horizontal stripe artifacts in the denoised image; for the black level error noise N BLE Modeling can effectively reduce color deviation. For fixed pattern noise N FPN and read noise N read Modeling, adding a denoising network for fixed pattern noise N FPN and read noise N read The processing capability of the target noise model is more accurate, which can improve the quality of the denoised image after denoising.

[0078] See also Figure 3 , in some embodiments, 01: Modeling shot noise N P , and calibrate the relationship between ISO and system gain K, including:

[0079] 011: Use Poisson distribution P(I) to model shot noise to obtain shot noise N P Model: (I+N P )~P(I), where I represents the number of photoelectrons proportional to the scene illumination;

[0080] On each ISO, execute:

[0081] 012: Take two shots of the same uniformly illuminated scene to obtain two flat-field frames;

[0082] 013: Get the mean value of the pixel values ​​of two flat field frames;

[0083] 014: Obtain the difference image of two flat-field frames, and obtain the variance Var of the pixel values ​​of the difference image;

[0084] 015: Change the light input amount Q, and repeat the method from 012 to 014 under multiple light input amounts Qi to obtain multiple mean Mean and variance Var relationship pairs;

[0085] 016: Fitting a Mean-Var curve according to the relationship between multiple mean values ​​Mean and variance Var, and taking the slope of the Mean-Var curve as the system gain K under the ISO; and

[0086] 017: For different ISOs, repeat the method from 012 to 016 to obtain the system gain K under different ISOs.

[0087] Correspondingly, please combine Figure 2 The first calibration module 11 is also used to execute the method in 011 to 017. Specifically, the first calibration module 11 is also used to use Poisson distribution P(I) to perform shot noise modeling to obtain the shot noise N P Model: (I+N P )~P(I), where I represents the number of photoelectrons proportional to the scene illumination; performing at each ISO: shooting the same uniformly illuminated scene twice to obtain two flat-field frames; obtaining the mean Mean of the pixel values ​​of the two flat-field frames; obtaining a difference image of the two flat-field frames, and obtaining the variance Var of the pixel values ​​of the difference image; changing the amount of light input Q, and repeatedly performing the methods from 012 to 014 at multiple amounts of light input Qi to obtain multiple pairs of mean Mean and variance Var relationships; fitting a Mean-Var curve based on the multiple pairs of mean Mean and variance Var relationships, and using the slope of the Mean-Var curve as the system gain K at the ISO; and for different ISOs, repeatedly performing the methods from 012 to 016 to obtain the system gain K at different ISOs.

[0088] See also Figure 4 , Figure 4 The left side shows a schematic diagram of the distribution of pixels in the image sensor, and the right side shows a schematic diagram of the pixel values ​​of the pixels of two frames of images captured by the left image sensor. Specifically, the same uniformly luminous scene, such as a whiteboard, is captured twice to obtain two flat field frames, namely flat field frame 1 and flat field frame 2. The shooting conditions of the two shots are the same, for example: the sensitivity values ​​are both ISO1 and the light input amount is both Q1.

[0089] Get the mean value Mean of the pixel values ​​of the two flat-field frames. Specifically, assume that the pixels of flat-field frame 1 are distributed in an M*N array, and the pixels of flat-field frame 2 are also distributed in an M*N array. The mean value Mean(A) of the pixel values ​​of flat-field frame 1 = ∑A(i, j) / (M*N), where A(i, j) is the pixel value of the pixel point with coordinates (i, j) in flat-field frame 1, i is the row coordinate of the pixel point, i is an integer in [0, M], and j is the column coordinate of the pixel point, j is an integer in [0, N]. Assume that flat-field frame 1 only contains Figure 4 Taking the pixel values ​​of the pixels shown as an example, the mean value of the pixel values ​​of the flat field frame 1, Mean(A) = (A1+A2+A3+A4+A5+A6+A7+A8+A9+A10+A11+A12+A13+A14+A15+A16) / 16. Similarly, assuming that the pixels of the flat field frame 2 are in an M*N array, the mean value of the pixel values ​​of the flat field frame 2, Mean(B) = ∑B(i, j) / (M*N), where B(i, j) is the pixel value of the pixel with coordinates (i, j) in the flat field frame 2, i is the row coordinate of the pixel, i is an integer in [0, M], and j is the column coordinate of the pixel, j is an integer in [0, N]. Assuming that the flat field frame 2 only contains Figure 4 Taking the pixel values ​​of the pixels shown as an example, the mean value of the pixel values ​​of the flat field frame 2, Mean(B) = (B1+B2+B3+B4+B5+B6+B7+B8+B9+B10+B11+B12+B13+B14+B15+B16) / 16. Based on this, the mean value of the pixel values ​​of the two flat field frames, Mean1 = [Mean(A)+Mean(B)] / 2 = [∑A(i, j) / (M*N)+∑B(i, j) / (M*N)] / 2 = [∑A(i, j)+∑B(i, j)] / (2M*N).

[0090] See also Figure 5, the two figures on the left are schematic diagrams of the pixel values ​​of the pixels of the flat-field frame 1 and the flat-field frame 2, respectively, and the figure on the right is a schematic diagram of the pixel values ​​of the pixel points of the difference image, wherein the pixel value Δ(i, j) of each pixel point (i, j) in the difference image is the difference between the pixel value A(i, j) of the corresponding position pixel point in the flat-field frame 1 and the pixel value B(i, j) of the corresponding position pixel point in the flat-field frame 2. That is, Δ(i, j) = A(i, j) - B(i, j). For example, the pixel value Δ(1, 1) of the pixel point (1, 1) in the difference image is the difference between the pixel value A(1, 1) of the corresponding position pixel point (1, 1) in the flat-field frame 1 and the pixel value B(1, 1) of the corresponding position pixel point (1, 1) in the flat-field frame 2: A1 - B1. After obtaining the pixel values ​​Δ(i, j) of all the pixel points (i, j) in the difference image, the variance Var1 of these pixel values ​​Δ(i, j) can be calculated.

[0091] The value of the incoming light amount is changed to Q2, and steps 012, 013, and 014 are repeated, that is, under the condition of ISO1 and the incoming light amount Q2, two flat-field frames are obtained, and the mean Mean2 of the pixel values ​​of the two newly obtained flat-field frames is calculated according to the method in 013, and a new difference image is obtained based on the two newly obtained flat-field frames, and the variance Var2 of the new difference image is obtained. Next, the value of the incoming light amount is changed to Q3, and steps 012, 013, and 014 are repeated, that is, under the condition of ISO1 and the incoming light amount Q2, the mean Mean3 and the variance Var3 are obtained.

[0092] Continuing in this way, we can get the mean value Meanf and variance Varf under the same ISO1 and different light input amounts Qf, where f is a natural number, and the light input amount Qf, mean value Meanf and variance Varf are one-to-one corresponding, such as Figure 6 As shown, multiple pairs of mean Mean and variance Var can be obtained, for example, mean Mean1 and variance Var1 are a pair, mean Mean2 and variance Var2 are a pair, mean Mean3 and variance Var3 are a pair, and so on.

[0093] See also Figure 7 After obtaining multiple mean Mean and variance Var relationship pairs, a Mean-Var curve is fitted, which corresponds to ISO1, and the slope of the Mean-Var curve is used as the system gain K1 under ISO1.

[0094] Repeat the method in 012 to 016 to obtain the system gain K2 under ISO2, the system gain K3 under ISO3, and so on. That is, the system gain Ke under different ISOe can be obtained, where e is a natural number, and ISOe corresponds to the system gain Ke one by one, such as Figure 8shown.

[0095] During exposure, incident light in the form of photons strikes the pixel area of ​​the image sensor. Due to the uncertainty of the photon impact, the number of electrons recorded on the image sensor follows a Poisson distribution. P It belongs to the noise associated with the optical signal, that is, shot noise N P The noise intensity depends on the light intensity. Shot noise N P The system gain K is often calibrated by obtaining a flat field frame by shooting a scene with uniform illumination, such as white paper or a white board. However, there are often detailed textures on white paper or a white board, and it is difficult to obtain such a uniformly illuminated image, which brings certain difficulties to the calibration of the system gain K.

[0096] The noise model acquisition method and noise model acquisition device 10 of the embodiment of the present application are used to model the shot noise N P The calibration is performed by shooting flat field frames at different sensitivities (ISO) and different amounts of light Q, which can make up for the requirement of the calibration system gain K for the flat field frames. Even if the white paper or white board shot by the shooting module 17 has certain detail textures, it does not affect the accuracy of the calibration parameters (system gain K). The target noise model obtained thereby is more accurate, which can further improve the quality of the denoised image after denoising.

[0097] See also Fig. 9 and Fig.10 In some embodiments, 02: At different ISOs, capture dark field frames and perform modeling of black level error noise N BLE , to calibrate the black level error noise at different ISOs, including:

[0098] 021: Under different ISO conditions, the pixel values ​​of each sub-channel in the preset area of ​​the dark field frame are averaged and used as the black level error noise on the corresponding channel under the ISO. The preset area is an area extending outward from the center of the dark field frame to a predetermined range.

[0099] Correspondingly, please combine Figure 2 , the second calibration module 12 is also used to execute the method in 021. Specifically, the second calibration module 12 is also used to average the pixel values ​​of each sub-channel in the preset area of ​​the dark field frame under different ISOs, respectively as the black level error noise on the corresponding channel under the ISO, and the preset area is an area extending outward from the center of the dark field frame to a predetermined range.

[0100] Among them, the "preset area of ​​the dark field frame" is usually the central area of ​​the dark field frame, that is, the outermost circle of pixels is not included. The dark field frame usually includes four sub-channels R, Gr, Gb and B. The mean of the pixel value of each sub-channel is the sum of the pixel values ​​of the sub-channel divided by the total number of sub-channels. For example, the mean of the pixel value of the R sub-channel is Mean(R)=∑R(i, j) / N(R), where ∑R(i, j) is the sum of the pixel values ​​of the R sub-channel in the dark field frame, and N(R) is the total number of R sub-channels in the dark field frame; the mean of the pixel value of the Gr sub-channel is Mean(Gr)=∑Gr(i, j) / N(Gr), where ∑Gr(i, j) is the sum of the pixel values ​​of the Gr sub-channel in the dark field frame. The sum of the pixel values ​​of the four sub-channels, N(Gr) is the total number of Gr sub-channels in the dark field frame; the mean of the pixel values ​​of the Gb sub-channel is Mean(Gb)=∑Gb(i,j) / N(Gb), where ∑Gb(i,j) is the sum of the pixel values ​​of the Gb sub-channels in the dark field frame, and N(Gb) is the total number of Gb sub-channels in the dark field frame; the mean of the pixel values ​​of the B sub-channel is Mean(B)=∑B(i,j) / N(B), where ∑B(i,j) is the sum of the pixel values ​​of the B sub-channels in the dark field frame, and N(B) is the total number of B sub-channels in the dark field frame. After calculating the mean values ​​of the pixel values ​​of the four sub-channels, Mean(R), Mean(Gr), Mean(Gb) and Mean(B), the black level error noise N of all R pixels in the image sensor is BLE(R) = Mean (R), black level error noise N of all Gr pixels BLE(Gr) = Mean (Gr), black level error noise N of all Gb pixels BLE(Gb) = Mean (Gb), black level error noise N of all B pixels BLE(B) =Mean(B).

[0101] See also Fig.11 , with the dark field frame containing only Fig.11 Take the pixel value of the pixel point shown as an example, N BLE(R) =(C1+C3+C9+C11) / 4; N BLE(Gr) =(C2+C4+C10+C12) / 4; N BLE(Gb) =(C5+C7+C13+C15) / 4; N BLE(B) =(C6+C8+C14+C16) / 4.

[0102] The “preset area of ​​the dark field frame” in this application is the central area of ​​the dark field frame. read and line noise N row is zero mean, and the fixed pattern noise N FPNIt is weaker in the center area, so the black level error noise N is calibrated by calculating the average of the pixel values ​​in the center area. BLE It is very accurate, which can ensure that the subsequent target noise model is also relatively accurate.

[0103] See also Fig.12 In some embodiments, 03: At different ISOs, capture dark field frames and perform modeling of line noise N row , and calibrate different ISO and noise parameters σ row The relationships between

[0104] 031: For the row noise N of a column in the dark field frame row The mean is 0 and the standard deviation is σ row The row noise N of the normal distribution row Model: N row ~N(0,σ row 2 );

[0105] 033: Obtain the average value of each row of pixel values ​​of the dark field frame to obtain a column of mean value matrix; and

[0106] 035: Get the standard deviation σ of the mean matrix row , to obtain the ISO and noise parameter σ row The relationship between.

[0107] Correspondingly, please combine Figure 2 The third calibration module 13 is also used to execute the methods in 031, 033 and 035. Specifically, the third calibration module 13 is also used to determine the row noise N of a certain column in the dark field frame. row The mean is 0 and the standard deviation is σ row The row noise N of the normal distribution row Model: N row ~N(0,σ row 2 ); obtain the average value of each row of pixel values ​​in the dark field frame to obtain a column of mean matrix; and obtain the standard deviation σ of the mean matrix row , to obtain the ISO and noise parameter σ row The relationship between.

[0108] Please combine Fig.13 The left picture shows the dark field frame taken under the shooting conditions: the sensitivity value is ISO1 and the light input is 0. The right picture shows the mean matrix formed by the average value of each row of pixel values ​​of the dark field frame shown in the left picture, where the mean of each row of pixel values ​​is Meani=∑P(m) / N(m), where ∑P(m) is the sum of the pixel values ​​of the mth row, and N(m) is the total number of pixels in the mth row. Fig.13Taking the pixel values ​​of the pixels shown as an example, the mean of the pixel values ​​in the first row is Mean1 = (C1+C2+C3+C4) / 4, the mean of the pixel values ​​in the second row is Mean2 = (C5+C6+C7+C8) / 4, the mean of the pixel values ​​in the third row is Mean3 = (C9+C10+C11+C12) / 4, and the mean of the pixel values ​​in the fourth row is Mean4 = (C13+C14+C15+C16) / 4.

[0109] In obtaining Fig.13 After obtaining the mean matrix shown in the right figure, the standard deviation σ of the mean matrix is ​​calculated row , to obtain the ISO and noise parameter σ row The relationship between.

[0110] See also Fig.14 In some embodiments, 04: At different ISOs, capture dark field frames and perform modeling of fixed pattern noise N FPN ,include:

[0111] 041: Under each ISO, obtain the average pixel value of each pixel point of the dark field frame with a preset number of frames;

[0112] 043: The mean value of the pixel value of each pixel minus the black level error noise N of the corresponding channel under the corresponding ISO BLE , to obtain the fixed pattern noise N of each pixel FPN ;and

[0113] 045: Using fixed pattern noise N at each ISO value and each pixel FPN , perform linear regression on ISO for each pixel and obtain FPN K and FPN Bias , to get the fixed pattern noise N of the pixel FPN Model: N FPN =FPN K *ISO+FPN Bias , where FPN K *ISO is dark current noise, FPN Bias is the column noise.

[0114] Correspondingly, please combine Figure 2 , the third calibration module 13 is also used to execute the methods in 041, 043 and 045. Specifically, the third calibration module 13 is also used to obtain the average pixel value of each pixel point of the dark field frame of a preset number of frames under each ISO; the average pixel value of each pixel point minus the black level error noise N of the corresponding channel under the corresponding ISO BLE , to obtain the fixed pattern noise N of each pixel FPN; and using each ISO value and the fixed pattern noise N of each pixel FPN , perform linear regression on ISO for each pixel and obtain FPN K and FPN Bias , to get the fixed pattern noise N of the pixel FPN Model: N FPN =FPN K *ISO+FPN Bias , where FPN K *ISO is dark current noise, FPN Bias is the column noise.

[0115] See also Fig.15 The preset number of frames can be an empirical value input by a human, and can be changed by a human according to the demand; the preset number of frames can also be an empirical value generated by the third calibration module 13 according to the historical calibration data during the calibration process; no matter how the preset number of frames is formed, it is a known value, for example, the preset number of frames can be 2 frames, 3 frames, 4 frames or more frames. This application only takes the preset number of frames as 2 frames as an example for explanation.

[0116] Get the average value of the pixel value of each pixel point of the preset number of dark field frames. For example, the average value of the pixel value of the pixel point (i, j) of multiple dark field frames is ∑Frame[x / P(i, j)] / x, where P(i, j) is the pixel value of the pixel point (i, j) of any dark field frame, and x is the number of frames, which is a natural number. For example, when the preset number of frames is 2, Fig.15 As shown in the dark field frame 1 and the dark field frame 2, the mean of the pixel value of the pixel point (1, 1) of the dark field frame 1 and the dark field frame 2 is Mean11=(C1+D1) / 2, the mean of the pixel value of the pixel point (1, 2) is Mean12=(C2+D2) / 2, the mean of the pixel value of the pixel point (1, 3) is Mean13=(C3+D3) / 2, the mean of the pixel value of the pixel point (1, 4) is Mean14=(C4+D4) / 2, the mean of the pixel value of the pixel points in the second row is as shown in the figure, namely Mean21=(C5+D5) / 2, Mean22=(C6+D6) / 2, Mean23=(C7+D7) / 2 and Mean24=(C8+D8) / 2, and the mean of the pixel value of the pixel points in the third row is as shown in the figure. Fig.15As shown, they are Mean31=(C9+D9) / 2, Mean32=(C10+D10) / 2, Mean33=(C11+D11) / 2 and Mean34=(C12+D12) / 2 respectively. The mean pixel values ​​of the pixels in the 4th row are shown in the figure, which are Mean41=(C13+D13) / 2, Mean42=(C14+D14) / 2, Mean43=(C15+D15) / 2 and Mean44=(C16+D16) / 2 respectively.

[0117] See also Fig.15 and Fig.16 After obtaining the mean value ∑Frame[x / P(i, j)] / x of the pixel value of each pixel, use ∑Frame[x / P(i, j)] / x to subtract the black level error noise N of the corresponding channel under the corresponding ISO BLE , we can get the fixed pattern noise N of each pixel FPN .like Fig.16 The fixed pattern noise N of the pixel (1, 1) shown is FPN11 =Mean11-N BLE(R) , the fixed pattern noise N of pixel (1, 2) FPN12 =Mean12-N BLE(Gr) , the fixed pattern noise N of pixel (1, 3) FPN13 =Mean13-N BLE(R) , the fixed pattern noise N of pixel (1, 4) FPN14 =Mean14-N BLE(Gr) The fixed pattern noise of other pixels is similar and will not be listed one by one.

[0118] At one ISO value, the same pixel has one fixed pattern noise, and at multiple ISO values, the pixel has multiple fixed pattern noises, such as Fig.17 As shown, for pixel (2, 2), there is fixed pattern noise N at ISO1 FPN221 , at ISO2 there is fixed pattern noise N FPN222 , at ISO3 there is fixed pattern noise N FPN223 , and so on. For pixel (2, 3), there is fixed pattern noise N at ISO1 FPN231 , at ISO2 there is fixed pattern noise N FPN232 , at ISO3 there is fixed pattern noise N FPN233 , and so on. For other pixel points (i, j), we can get an ISO value and the fixed pattern noise N of each pixel point (i, j). FPN A one-to-one correspondence list.

[0119] See also Fig.18 ,according to Fig.17 The relationship list in can be used to perform linear regression on ISO for each pixel and obtain FPN K and FPN Bias , to get the fixed pattern noise N of the pixel FPN Model: N FPN =FPN K *ISO+FPN Bias , where FPN K *ISO is the dark current noise, FPN Bias is the column noise.

[0120] See also Fig.19 In some embodiments, 05: Take dark field frames at different ISOs and perform modeling of read noise N read , and calibrate ISO with shape parameter λ and noise intensity σ TL The relationships between

[0121] 051: Use the long-tailed TL (Tukey Lambda) distribution to model the read noise to obtain the read noise N read Model: N read ~TL(λ(ISO),0,σ TL ), where λ is the shape parameter, σ TL is the noise intensity, 0 is zero mean;

[0122] 053: Subtract the corresponding black level error noise N from the pixel value of each pixel in the dark field frame BLE , the corresponding fixed pattern noise N FPN And the corresponding average value of each row of pixels to obtain the read noise N of each pixel under this ISO read ;and

[0123] 055: Estimate shape parameter λ and noise intensity σ TL .

[0124] Correspondingly, please combine Figure 2 The fourth calibration module 14 is also used to execute the methods in 051, 053 and 055. Specifically, the fourth calibration module 14 is also used to use the TL (Tukey Lambda) distribution of the long-tail distribution to perform read noise modeling to obtain the read noise N read Model: N read ~TL(λ(ISO),0,σ TL ), where λ is the shape parameter, σ TL is the noise intensity, 0 is the zero mean; the pixel value of each pixel in the dark field frame is subtracted from the corresponding black level error noise N BLE, the corresponding fixed pattern noise and the corresponding average value N of each row of pixels FPN , to obtain the readout noise N of each pixel under this ISO read ; and estimate the shape parameter λ and the noise intensity σ TL .

[0125] Due to the line-by-line reading characteristics of CMOS image sensors, each pixel does not share the same amplifier and analog-to-digital converter, which results in readout noise N read It presents a "long-tail" skewed distribution instead of a normal distribution. Although existing methods have considered using a skewed TL distribution instead of a normal distribution, the CMOS image sensor presents different degrees of skewness at different ISOs, which has not been considered. Different from the previous method that uses a fixed shape parameter λ for each camera, the readout noise N in the embodiment of the present application is read The shape parameter λ is found to be related to ISO so that the read noise N read It is very accurate, which ensures that the subsequent target noise model is also relatively accurate.

[0126] See also Figure 1 , the training method of the denoising neural network in the embodiment of the present application includes:

[0127] 07: Get multiple reference images;

[0128] 08: acquiring noise images corresponding to the multiple reference images according to the multiple reference images and the preset target noise model to acquire a set of paired reference-noise images, wherein in each pair of reference-noise images, the noise of the reference image is less than the noise of the noise image, and the target noise model is acquired by the method for acquiring the noise model described in any one of the above embodiments; and

[0129] 09: Train a neural network using a reference-noise image set to obtain a denoising neural network.

[0130] Please combine Fig. 20 , the denoising neural network training method of the embodiment of the present application can be applied to the denoising neural network training device 30. The denoising neural network training device 30 includes a first acquisition unit 31, a second acquisition unit 33 and a training unit 35. The first acquisition unit 31 is used to acquire multiple reference images. The second acquisition unit 33 is used to acquire noise images corresponding to the multiple reference images according to the multiple reference images and a preset target noise model to acquire a pair of reference-noise image sets, in each pair of reference-noise images, the noise of the reference image is less than the noise of the noise image, and the target noise model is acquired by the noise model acquisition method of any of the above-mentioned embodiments. The training unit 35 is used to train the neural network using the reference-noise image set to obtain the denoising neural network.

[0131] See also Fig.21 , 09: Train a neural network using a reference-noise image set to obtain a denoising neural network, including:

[0132] 091: Input the noise image in each pair of reference-noise images into the neural network;

[0133] 093: Processing a noisy image using a neural network to output a denoised image; and

[0134] 095: Train the parameters of the neural network according to the denoised image and the reference image in each pair of reference-noise images until the difference between the denoised image and the reference image is less than a predetermined threshold.

[0135] Please combine Fig. 20 The training unit 35 is also used to input the noise image in each pair of reference-noise images into the neural network; use the neural network to process the noise image to output a denoised image; and train parameters in the neural network according to the denoised image and the reference image in each pair of reference-noise images until the difference between the denoised image and the reference image is less than a predetermined threshold.

[0136] Among them, the difference between the denoised image and the reference image can be represented by a distance vector. The predetermined threshold can be an empirical value obtained before the electronic device leaves the factory, or a set value input when the electronic device is used after leaving the factory, or an empirical value obtained by a processor processing historical data after the electronic device leaves the factory. There is no limitation here.

[0137] Please combine Fig. 22 When the shooting module 17 actually shoots, after the shooting module 17 executes shooting and obtains the Raw image, the Raw image usually has a lot of noise. The electronic device will input the Raw image into the denoising neural network for denoising processing, and after the denoising neural network denoising processing, the denoised image can be output.

[0138] See also Fig.23 The upper left picture is a clear image. The clear image can be obtained by using a shooting module 17 with better firmware, or by using a stronger processing software (for example, superimposing multiple noise images), or by using a shooting module 17 with better firmware and using a stronger processing software. It can be seen from the figure that it has a higher degree of clarity and less noise. The lower left picture is obtained using the other first noise model, which is different from the read noise N in the target noise model of this application. read The upper right picture is obtained by using the other second noise model, which only uses the Poisson-Gaussian model. The lower right picture is obtained by using the target noise model of the present application. By comparison, it can be seen that the lower right picture is closer to the effect of the upper left picture, and the imaging quality is better.

[0139] See also Fig.24 The electronic device of the embodiment of the present application includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the noise model acquisition method of any of the above embodiments, or executes the noise neural network training method of any of the above embodiments.

[0140] The electronic device may be a terminal, and its internal structure diagram may be as follows: Fig.24 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory.

[0141] The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and the external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0142] See also Fig.24 , a computer-readable storage medium of an embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, it implements the method for obtaining the noise model of any of the above-mentioned embodiments, or the method for training the noise neural network of any of the above-mentioned embodiments.

[0143] Those skilled in the art will understand that Fig.24 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0145] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0146] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor 50, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0147] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0148] A person of ordinary skill in the art will appreciate that all or part of the steps carried by the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0149] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for obtaining a noise model, characterized in that: include: Shooting flat-field frames at different ISOs and different amounts of incoming light to model shot noise N P , and calibrate the relationship between ISO and system gain K; At different ISOs, capture dark field frames and perform the following operations: Modeling black level error noise N BLE , to calibrate the black level error noise under different ISO; Modeling row noise N row , and calibrate different ISO and noise parameters σ row the relationship between; Modeling fixed pattern noise N FPN , the fixed pattern noise N FPN including a fixed pattern of dark current noise and a fixed pattern of column noise, wherein the dark current noise is related to ISO; Modeling readout noise N read , and calibrate different ISO with shape parameter λ and noise intensity σ TL the relationship between; and According to the shot noise N P The model, the relationship between the ISO and the system gain K, the black level error noise N BLE The model, the row noise N row The model, the different ISO and noise parameters σ row The relationship between the fixed pattern noise N FPN The model, the read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

2. The method for obtaining a noise model according to claim 1, characterized in that: The modeled shot noise N P , and calibrate the relationship between ISO and system gain K, including: The shot noise is modeled using Poisson distribution P(I) to obtain the shot noise N P Model: (I+N P )~P(I), where I represents the number of photoelectrons proportional to the scene illumination; On each ISO, execute: Step 1: Take two shots of the same uniformly illuminated scene to obtain two flat-field frames; Step 2: Obtaining a mean value Mean of the pixel values ​​of the two flat-field frames; Step 3: obtaining a difference image between two flat-field frames, and obtaining a variance Var of pixel values ​​of the difference image; Step 4: Change the light input amount Q, and repeat steps 1 to 3 under multiple light input amounts Qi to obtain multiple mean Mean and variance Var relationship pairs; Step 5: fitting a Mean-Var curve according to a plurality of mean Mean and variance Var relationship pairs, and taking the slope of the Mean-Var curve as the system gain K under the ISO; and For different ISOs, repeat steps 1 to 5 to obtain the system gain K at different ISOs.

3. The method for obtaining a noise model according to claim 1, characterized in that: At different ISOs, dark field frames are captured and modeled black level error noise N BLE , to calibrate the black level error noise at different ISOs, including: Under different ISOs, the pixel values ​​of each sub-channel in a preset area of ​​the dark field frame are averaged and used as the black level error noise on the corresponding channel under the ISO. The preset area is an area extending outward from the center of the dark field frame to a predetermined range.

4. The method for obtaining a noise model according to claim 1, characterized in that: At different ISOs, dark field frames were taken and modeling of line noise N was performed. row , and calibrate different ISO and noise parameters σ row The relationships between For a row noise N in a certain column of the dark field frame row The mean is 0 and the standard deviation is σ row The row noise N of the normal distribution row Model: N row ~N(0,σ row 2 ); Obtaining an average value of pixel values ​​in each row of the dark field frame to obtain a column of a mean value matrix; and Get the standard deviation σ of the mean matrix row , to obtain the ISO and noise parameter σ row The relationship between.

5. The method for obtaining a noise model according to claim 1, characterized in that: At different ISOs, dark field frames were captured and modeling of fixed pattern noise N was performed. FPN ,include: Under each ISO, obtain the average pixel value of each pixel point of the dark field frames with a preset number of frames; The mean value of the pixel value of each pixel minus the black level error noise N of the corresponding channel under the corresponding ISO BLE , to obtain the fixed pattern noise N of each pixel FPN ;and Using the fixed pattern noise N of each ISO value and each pixel FPN , perform linear regression on ISO for each pixel and obtain FPN K and FPN Bias , to obtain the model of the fixed pattern noise of the pixel: N FPN =FPN K *ISO+FPN Bias , where FPN K *ISO is the dark current noise, FPN Bias is the column noise.

6. The method for obtaining a noise model according to claim 1, characterized in that: At different ISOs, dark field frames were captured and modeled read noise N read , and calibrate ISO with shape parameter λ and noise intensity σ TL The relationships between The read noise is modeled using the long-tailed TL (Tukey Lambda) distribution to obtain the read noise N read Model: N read ~TL(λ(ISO);0,σ TL ), where λ is the shape parameter, σ TL is the noise intensity, 0 is zero mean; The corresponding black level error noise, the corresponding fixed pattern noise and the corresponding average value of each row of pixels are subtracted from the pixel value of each pixel in the dark field frame to obtain the readout noise N of each pixel under the ISO. read ;and Estimate shape parameter λ and noise intensity σ TL .

7. A device for acquiring a noise model, characterized in that: It includes a shooting module, a first calibration module, a second calibration module, a third calibration module, a fourth calibration module, a fifth calibration module and an acquisition module; The shooting module is used to shoot flat-field frames at different sensitivities (ISO) and different amounts of incoming light; the shooting module is also used to shoot dark-field frames at different ISOs; The first calibration module is used to model the shot noise N according to the flat field frame P , and calibrate the relationship between ISO and system gain K; The second calibration module is used to model the black level error noise N according to the dark field frame BLE , to calibrate the black level error noise under different ISO; The third calibration module is used to model the row noise N according to the dark field frame row , and calibrate different ISO and noise parameters σ row the relationship between; The fourth calibration module is used to model the fixed pattern noise N according to the dark field frame FPN , the fixed pattern noise N FPN including a fixed pattern of dark current noise and a fixed pattern of column noise, wherein the dark current noise is related to ISO; The fifth calibration module is used to model the readout noise N according to the dark field frame read , and calibrate different ISO with shape parameter λ and noise intensity σ TL the relationship between; The acquisition module is used to obtain the shot noise N P The model, the relationship between the ISO and the system gain K, the black level error noise N BLE The model, the row noise N row The model, the different ISO and noise parameters σ row The relationship between the fixed pattern noise N FPN The model, the read noise N read Models with different ISO and shape parameters λ and noise intensity σ TL The target noise model is obtained by the relationship between them.

8. A method for training a denoising neural network, characterized in that: include: Obtain multiple reference images; Acquire noise images corresponding to the plurality of reference images according to the plurality of reference images and a preset target noise model to obtain a set of paired reference-noise images, wherein in each pair of reference-noise images, the noise of the reference image is smaller than the noise of the noise image, and the target noise model is acquired by the noise model acquisition method according to any one of claims 1 to 6; and The reference-noise image set is used to train a neural network to obtain a denoising neural network.

9. The training method according to claim 8, characterized in that: The step of using the reference-noise image set to train a neural network to obtain a denoising neural network comprises: Input the noisy image in each reference-noise image pair into the neural network; Processing the noisy image using the neural network to output a denoised image; and Parameters in a neural network are trained according to the denoised image and a reference image in each pair of reference-noise images until a difference between the denoised image and the reference image is less than a predetermined threshold.

10. A training device for a denoising neural network, characterized in that: include: A first acquisition unit, the first acquisition unit is used to acquire a plurality of reference images; a second acquisition unit, the second acquisition unit being used to acquire noise images corresponding to the plurality of reference images according to the plurality of reference images and a preset target noise model, so as to acquire a set of paired reference-noise images, wherein in each pair of reference-noise images, the noise of the reference image is smaller than the noise of the noise image, and the target noise model is acquired by the noise model acquisition method according to any one of claims 1 to 6; and A training unit is used to train a neural network using the reference-noise image set to obtain a denoising neural network.

11. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor executes the method for acquiring a noise model as described in any one of claims 1 to 6, or executes the method for training a noise neural network as described in any one of claims 8 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for acquiring a noise model described in any one of claims 1 to 6 is implemented, or the method for training a noise neural network described in any one of claims 8 to 9 is implemented.

Images