Noise estimation method, image processing method, device and storage medium thereof

Abstract

A noise estimation method and an image processing method and a device and a storage medium thereof, the noise estimation method comprising: performing zoning processing on an image to be estimated synthesized from a long frame image and a short frame image: when the pixel to be estimated is located in the dark area, performing noise estimation based on the long frame image; when the pixel to be estimated is located in the bright area, performing noise estimation based on the short frame image and in combination with the exposure ratio; when the pixel to be estimated is located in the transition area, performing noise estimation based on the long frame image and the short frame image and in combination with preset values ​​such as synthesis parameters, exposure ratio and maximum display value. Therefore, the noise estimation method of the present invention can accurately estimate the noise conditions of each area of ​​a synthetic image with a high dynamic range, and can effectively improve the effect of subsequent noise reduction operations.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a noise estimation method and an image processing method and a device and a storage medium thereof. Background Art

[0002] Accurate noise estimation of images is the basis for many functional modules in image processing to function properly. For example, the noise reduction effects of 2D noise reduction and 3D noise reduction all depend on accurate noise estimation.

[0003] Current technology can already perform relatively accurate noise estimation on unprocessed raw images (RAW images) taken by image sensors, such as noise estimation using Poisson noise model and Gaussian noise model. However, there is still no public report on how to perform noise estimation on high dynamic range images.

[0004] The ability to capture high dynamic images is an important aspect of measuring camera equipment. For example, whether a nighttime autonomous driving camera can capture clear road conditions with headlights is crucial for safe driving. In order to obtain a high dynamic range (HDR) image, one method is to synthesize two RAW images with the same scene but different exposures to obtain an HDR image, such as the method described in the public invention patent CN113905185A.

[0005] However, the noise characteristics of the synthesized HDR image have changed significantly compared to the RAW image. If the noise estimation method of the RAW image is still used, the noise estimation will be inaccurate, resulting in poor noise reduction effect during subsequent image processing such as noise reduction, such as blurred images or failure to effectively reduce noise. Summary of the invention

[0006] The problem solved by the present invention is how to perform noise estimation on a synthetic image with a high dynamic range.

[0007] To solve the above problem, the present invention provides a noise estimation method, comprising:

[0008] Based on a preset exposure ratio, a long frame image and a short frame image are obtained, wherein the ratio of the exposure of the long frame image to the exposure of the short frame image is equal to the exposure ratio; based on the long frame image and the short frame image, in combination with the exposure ratio and preset synthesis parameters, an image to be estimated is synthesized, wherein the image to be estimated has pixel bits to be estimated, the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the long frame image, and the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the short frame image, wherein the image to be estimated includes: a bright area, a dark area and a transition area; based on the long frame image and the short frame image, noise estimation is performed on the image to be estimated, and the step of performing noise estimation on the image to be estimated includes: when the pixel bit to be estimated is located in the dark area, according to the noise variance of the corresponding pixel bit of the long frame, obtaining the noise variance of the pixel bit to be estimated, wherein the corresponding pixel bit of the long frame is the pixel bit in the long frame image corresponding to the pixel bit to be estimated. when the pixel to be estimated is located in the bright area, the noise variance of the pixel to be estimated is obtained according to the noise variance of the pixel corresponding to the short frame in combination with the exposure ratio, wherein the pixel corresponding to the short frame is the pixel corresponding to the pixel to be estimated in the short frame image; when the pixel to be estimated is located in the transition area, based on the noise variance of the selected average brightness value of the long frame and the noise variance of the selected average brightness value of the short frame, in combination with the exposure ratio, the synthesis parameter and the preset maximum display value, according to the average brightness value of the estimated neighborhood in the image to be estimated, the noise variance of the pixel to be estimated is obtained, wherein the pixel to be estimated is located in the estimated neighborhood, the noise variance of the selected average brightness value of the long frame refers to the noise variance of the long frame image when the average brightness value is equal to the first preset value, and the noise variance of the selected average brightness value of the short frame refers to the noise variance of the short frame image when the average brightness value is equal to the second preset value.

[0009] Optionally, in the step of performing noise estimation on the image to be estimated, when the pixel to be estimated is located in the transition area, the noise variance of the pixel to be estimated is obtained based on a quadratic function of an average brightness value of an estimated neighborhood in the image to be estimated.

[0010] Optionally, in the step of obtaining the noise variance of the pixel to be estimated according to the quadratic function of the average brightness value of the estimated neighborhood in the image to be estimated, the vertex of the quadratic function is (offset, var L (μ L = first preset value)), and a data point of the quadratic function is (L max , r 2 exp var s (μ s = second preset value)), where offset represents a synthesis parameter, Lmax Indicates the maximum display value, r exp Indicates exposure ratio, var L (μ L = first preset value) represents the noise variance of the average brightness value selected by the long frame, var s (μ s = second preset value) represents the noise variance of the selected average brightness value of the short frame.

[0011] Optionally, in the step of performing noise estimation on the image to be estimated,

[0012] When the pixel to be estimated is located in the bright area, the noise variance of the pixel to be estimated is:

[0013]

[0014] When the pixel to be estimated is located in the dark area, the noise variance of the pixel to be estimated is:

[0015] var hdr =var L ;

[0016] When the pixel to be estimated is located in the transition area, the noise variance of the pixel to be estimated is:

[0017] var hdr = k·(μ hdr -offset) 2 +b

[0018]

[0019] b=var L (μ L =First preset value

[0020] Among them, var hdr Represents the noise variance of the pixel to be estimated, var s Indicates the noise variance of the pixel corresponding to the short frame, var L represents the noise variance of the pixel corresponding to the long frame, r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value, var s (μ s = second preset value) represents the noise variance of the selected average brightness value of the short frame, var L (μ L =first preset value) represents the noise variance of the selected average brightness value of the long frame.

[0021] Optionally, in the step of synthesizing the image to be estimated, the image to be estimated is:

[0022]

[0023] Wherein, hdr represents the brightness value of the pixel bit of the image to be estimated; S represents the brightness value of the corresponding pixel bit in the short frame image, L represents the brightness value of the corresponding pixel bit in the long frame image, and r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value.

[0024] Optionally, the method further includes: after synthesizing the image to be estimated and before obtaining the noise variance of the pixel to be estimated, obtaining the region to which the pixel to be estimated belongs.

[0025] Optionally, in the step of obtaining the area to which the pixel to be estimated belongs, the area to which the pixel to be estimated belongs is obtained based on the average brightness of all pixel bits in the estimated neighborhood, combined with the maximum display value and the synthesis parameter, wherein the estimated neighborhood includes the pixel to be estimated.

[0026] Optionally, in the step of obtaining the area to which the pixel to be estimated belongs, when the average brightness of all pixel bits in the estimated neighborhood is greater than or equal to the maximum display value, the pixel to be estimated is located in the bright area; when the average brightness of all pixel bits in the estimated neighborhood is less than the maximum display value and greater than the synthesis parameter, the pixel to be estimated is located in the transition area; when the average brightness of all pixel bits in the estimated neighborhood is less than or equal to the synthesis parameter, the pixel to be estimated is located in the dark area.

[0027] Optionally, the method further includes: after obtaining the long frame image and the short frame image, obtaining the noise variance of the long frame image; and obtaining the noise variance of the short frame image.

[0028] Optionally, in the step of obtaining the noise variance of the long frame image, the noise variance is obtained based on at least one of a Poisson noise model and a Gaussian noise model; in the step of obtaining the noise variance of the short frame image, the noise variance is obtained based on at least one of a Poisson noise model and a Gaussian noise model.

[0029] Optionally, the first preset value is equal to the synthesis parameter; and the second preset value is equal to the ratio of the maximum display value to the exposure ratio.

[0030] Correspondingly, the present invention also provides a noise estimation device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the noise estimation method of the present invention.

[0031] and a storage medium, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the noise estimation method of the present invention are executed.

[0032] In addition, the present invention also provides an image processing method, comprising: performing noise estimation and obtaining a noise estimation result, wherein in the step of performing noise estimation, noise estimation is performed by the noise estimation method of the present invention.

[0033] Optionally, the method further includes: performing noise reduction processing according to the noise estimation result.

[0034] Correspondingly, the present invention also provides an image processing device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the image processing method of the present invention.

[0035] and a storage medium, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the image processing method of the present invention are executed.

[0036] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0037] In the technical solution of the present invention, the image to be estimated synthesized by the long frame image and the short frame image is partitioned: when the pixel to be estimated is located in the dark area, noise estimation is performed based on the long frame image; when the pixel to be estimated is located in the bright area, noise estimation is performed based on the short frame image and in combination with the exposure ratio; when the pixel to be estimated is located in the transition area, noise estimation is performed based on the long frame image and the short frame image and in combination with preset values ​​such as synthesis parameters, exposure ratio and maximum display value. Therefore, for a synthetic image with a high dynamic range, the noise estimation method of the present invention can accurately estimate the noise conditions in each area and can effectively improve the effect of subsequent noise reduction operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of an embodiment of a noise estimation method of the present invention;

[0039] Figure 2 yes Figure 1 A schematic diagram of a long frame image obtained in an embodiment of the noise estimation method shown;

[0040] Figure 3 yes Figure 1 A schematic diagram of a short frame image obtained in an embodiment of the noise estimation method shown;

[0041] Figure 4 yes Figure 1 A schematic diagram of an image to be estimated synthesized by an embodiment of the noise estimation method shown;

[0042] Figure 5 yes Figure 4 A schematic diagram of noise estimation results for all regions of the image to be estimated is shown;

[0043] Figure 6 yes Figure 4 A schematic diagram of noise estimation results of a dark area of ​​an image to be estimated is shown;

[0044] Figure 7 yes Figure 4 A schematic diagram of noise estimation results of a bright area of ​​an image to be estimated is shown;

[0045] Figure 8 yes Figure 4 The figure shows a schematic diagram of the noise estimation result in the transition area of ​​the image to be estimated. DETAILED DESCRIPTION

[0046] As can be seen from the background technology, the prior art lacks a method for estimating noise of a synthetic image with a high dynamic range.

[0047] To solve the technical problem, the present invention provides a noise estimation method and an image processing method and a device and a storage medium thereof. The noise estimation method of the present invention can accurately estimate the noise conditions of each area of ​​a high dynamic range synthetic image, and can effectively improve the effect of subsequent noise reduction operations.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] refer to Figure 1 , shows a flow chart of an embodiment of a noise estimation method of the present invention.

[0050] First, step S110 is performed to determine the exposure ratio r based on the preset exposure ratio r. exp , obtaining a long frame image and a short frame image, wherein the ratio of the exposure amount of the long frame image to the exposure amount of the short frame image is equal to the exposure ratio r exp .

[0051] The long frame image refers to an image with a large exposure, and the short frame image refers to an image with a small exposure. Therefore, the exposure ratio r exp represents the ratio of the exposure of the long frame image to the exposure of the short frame image, which is a positive number greater than 1, that is, r exp >1. Specifically, the exposure amount can be adjusted by at least one of adjusting the exposure time and changing the optical aperture size to obtain the long frame image and the short frame image.

[0052] It should be noted that, in the step of obtaining the long frame image and the short frame image, the long frame image and the short frame image are obtained by a linear image sensor, that is, for the long frame image and the short frame image, the digital signal increases linearly with the increase in the number of received photons.

[0053] It should be noted that one exposure cycle includes one long frame image and one short frame image. In one exposure cycle, the exposure ratio r exp For fixed and unchanging.

[0054] In some embodiments of the present invention, the long frame image and the short frame image are both original images (RAW images). The pixel value L of the pixel bit in the long frame image is greater than or equal to 0 and less than or equal to the maximum value L of the RAW image data. max , that is, 0≤L≤L max , the pixel value S of the pixel bit in the short frame image is greater than or equal to 0 and less than or equal to the maximum value of the RAW image data, that is, 0≤S≤L max Among them, the maximum value of RAW image data is L max Refers to the maximum possible value of RAW image data, L max =2 dbits -1, dbits indicates the RAW image data bit width. max It is called the maximum display value.

[0055] It should be noted that the scenes, image sizes, and data bit widths of the long frame image and the short frame image are all the same.

[0056] Figure 2 and Figure 3 They are shown respectively Figure 1 The long frame image and the short frame image obtained in the embodiment shown. The long frame image and the short frame image are both grayscale images containing noise. Specifically, the long frame image and the short frame image are both 10-bit images, that is, the RAW image data bit width dbits of the long frame image and the short frame image is equal to 10. Therefore, in the long frame image and the short frame image, the maximum value L of the RAW image data max Both are 1023, that is, L max =2 dbits -1=1023.

[0057] also, Figure 1 In the illustrated embodiment, in the step S110 of obtaining the long frame image and the short frame image, both the long frame image and the short frame image are images of 1000 rows and 4000 columns.

[0058] in addition, Figure 1In the embodiment shown, in the step S110 of obtaining the long frame image and the short frame image, the preset exposure ratio r exp is 16, that is, r exp =16, that is, the exposure amount of the long frame image is 16 times that of the short frame image, which can be achieved, for example, by setting the exposure time of the long frame image to 16 times that of the short frame image.

[0059] like Figure 2 and Figure 3 In the illustrated embodiment, in the row direction, the long frame image and the short frame image are equally divided into 40 grayscales. The reason for using grayscale images in some embodiments of the present invention is that each grayscale is equivalent to superimposing noise on a uniform average brightness value, which facilitates the statistics of the actual noise situation on the grayscale corresponding to each average brightness value in the image, so that it can be compared with the noise estimation result to measure the accuracy of the noise estimation method.

[0060] Specific as Figure 2 As shown, multiple gray levels in the bright area of ​​the long frame image are overexposed (i.e. Figure 2 The overexposure causes the loss of this part of the image information in the long frame image. Figure 3 As shown, the grayscale signal-to-noise ratio of the dark area of ​​the short frame image is relatively low (i.e., Figure 3 The image quality of the short frame image is lower than that of the same area in the long frame image.

[0061] It should be noted that if Figure 1 As shown, in some embodiments of the present invention, after executing step S110 to obtain the long frame image and the short frame image, executing step S111 to obtain the noise variance var of the long frame image L ; S112, obtain the noise variance var of the short frame image S .

[0062] Specifically, execute step S111 to obtain the noise variance var of the long frame image. L In the step, the noise variance var is obtained based on at least one of the Poisson noise model and the Gaussian noise model. L Execute step S112 to obtain the noise variance var of the short frame image S In the step, the noise variance var is obtained based on at least one of the Poisson noise model and the Gaussian noise model. S .

[0063] It should be noted that Figure 1In the embodiment shown, the image noise of the long frame image and the short frame image is composed of Poisson noise and Gaussian noise. Specifically, the global system gain of the long frame image and the short frame image is equal, both K 全局 =5DN / e - DN is a dimensionless unit representing the grayscale value of a digital image, e - Represents unit charge, K 全局 Indicates the grayscale value of the unit charge conversion output.

[0064] Therefore, according to the Poisson noise model, the Poisson noise standard deviation of the long frame image is:

[0065]

[0066] The Poisson noise standard deviation of the short frame image is:

[0067]

[0068] Among them, λ L represents the expected pixel value of the pixel position corresponding to the pixel position to be estimated in the long frame image, that is, the pixel value of the pixel position when it is assumed that there is no noise; due to λ L is not actually obtainable, so μ L To approximate, μ L represents the average brightness value of the estimated neighborhood in the long frame image; S represents the expected pixel value of the pixel position corresponding to the pixel position to be estimated in the short frame image, that is, the pixel value of the pixel position when assuming that there is no noise; due to λ S is not actually obtainable, so μ S To approximate, μ S Represents the average brightness value of the estimated neighborhood in the short frame image.

[0069] In addition to Poisson noise, the image noise of the long frame image also includes: a mean of 0 and a standard deviation of σ GL =3e - Gaussian white noise; the image noise of the short frame image also includes: the mean is 0, the standard deviation is σ GS =3e - Gaussian white noise. - Represents unit charge.

[0070] The noise of RAW images captured by most linear image sensors satisfies the Poisson noise model and the Gaussian noise model. For linear image sensors, their global system gain K 全局 is an easily measurable parameter, σ GL and σ GSIt can be calculated by the characteristic parameters of the sensor such as read noise, dark current noise and temperature, μ L and μ S It is also easy to obtain from the image, so the noise variance var of each pixel of the long frame image and the short frame image (the long frame image and the short frame image are both RAW images) is L and var S All are obtainable.

[0071] Continue to refer Figure 1 , combined with reference Figure 4 After obtaining the long frame image and the short frame image, step S120 is performed to obtain the long frame image and the short frame image in combination with the exposure ratio r exp and a preset synthesis parameter offset, synthesizing an image to be estimated, wherein the image to be estimated has pixel bits to be estimated, the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the long frame image, and the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the short frame image, wherein the image to be estimated includes: a bright area, a dark area and a transition area.

[0072] Specifically, in some embodiments of the present invention, according to the long frame image and the short frame image, combined with the exposure ratio r exp And the preset synthesis parameter offset, the method of synthesizing the image to be estimated is:

[0073]

[0074] hdr=r exp ×S×α+L×(1-α) where α represents the grayscale matrix with the same size as the long frame image, 0≤α≤1; clip(y,0,1) means clamping the values ​​of all elements of the matrix y within the range of 0 to 1; offset represents the synthesis parameter; S represents the brightness value of the corresponding pixel bit in the short frame image, L represents the brightness value of the corresponding pixel bit in the long frame image, and L max Indicates the maximum display value.

[0075] The larger the brightness value of the pixel in the long frame image, the larger the pixel value of the corresponding pixel in α; the smaller the brightness value of the pixel in the long frame image, the smaller the pixel value of the corresponding pixel in α. It can be seen that in the image to be estimated, the higher the weight of the short frame image is for the pixel corresponding to the larger brightness value of the pixel in the long frame image; the higher the weight of the long frame image is for the pixel corresponding to the smaller brightness value of the pixel in the long frame image; therefore, the synthesized image to be estimated is a high dynamic image, which has both bright field information and dark field information in the scene.

[0076] Therefore, in the step S120 of synthesizing the image to be estimated, the image to be estimated is:

[0077]

[0078] Wherein, hdr represents the brightness value of the pixel bit of the image to be estimated; S represents the brightness value of the corresponding pixel bit in the short frame image, L represents the brightness value of the corresponding pixel bit in the long frame image, and r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value.

[0079] The image to be estimated obtained by synthesis includes three areas: bright area, dark area and transition area. hdr ≥L max The area is the bright area of ​​the image to be estimated, the long frame image is overexposed in the bright area, and the short frame image provides all the information of the bright area of ​​the image to be estimated; 0≤μ hdr The area where offset<μ is the dark area of ​​the image to be estimated, and the long frame image provides all the information of the dark area of ​​the image to be estimated; hdr <L max The area is the transition area of ​​the image to be estimated, and the long frame image and the short frame image together provide information of the transition area of ​​the image to be estimated. hdr Represents the average brightness value of the estimated neighborhood in the image to be estimated.

[0080] It should be noted that the estimated neighborhood refers to the neighborhood used for noise estimation of the image to be estimated, and the pixel to be estimated is located within the estimated neighborhood. The estimated neighborhood can be a square array of 3×3, 5×5 or 7×7 pixels, or other centrally symmetrical arrays.

[0081] like Figure 4 As shown, in the image to be estimated synthesized by the long frame image and the short frame image, the dark area has a higher signal-to-noise ratio of the long frame image, and the bright area is not overexposed, thereby achieving complementary advantages of the long frame image and the short frame image.

[0082] Continue to refer Figure 1 After obtaining the image to be estimated, step S130 is executed to perform noise estimation on the image to be estimated according to the long frame image and the short frame image to obtain the noise variance of the pixel position to be estimated.

[0083] Since the image to be estimated is divided into a bright area, a dark area and a transition area, and the composition of the image to be estimated in different areas is different; therefore, in some embodiments of the present invention, the noise estimation method also includes: executing step S120, after synthesizing the image to be estimated, executing step S130, before obtaining the noise variance of the pixel position to be estimated, executing step S140, to obtain the area to which the pixel position to be estimated belongs.

[0084] Specifically, in some embodiments, in the step of executing step S140, in the step of obtaining the region to which the pixel to be estimated belongs, the average brightness of all pixels in the estimated neighborhood is combined with the maximum display value L max and the synthesis parameter offset to obtain the area to which the pixel to be estimated belongs, wherein the estimated neighborhood includes the pixel to be estimated.

[0085] It should be noted that the pixel position to be estimated is the central pixel position of the estimation neighborhood.

[0086] Specific as Figure 1 In the embodiment shown, in step S140, in the step of obtaining the region to which the pixel to be estimated belongs, the average brightness of all the pixels in the estimated neighborhood is greater than or equal to the maximum display value L max , that is, μ hdr ≥L max When the pixel to be estimated is located in the bright area; the average brightness of all pixels in the estimated neighborhood is less than the maximum display value L max And is greater than the synthesis parameter offset, that is, offset<μ hdr <L max , the pixel to be estimated is located in the transition zone; the average brightness of all pixels in the estimated neighborhood is less than or equal to the synthesis parameter offset, that is, 0≤μ hdr ≤offset, the pixel to be estimated is located in the dark area.

[0087] After the region to which the pixel to be estimated belongs is obtained, the noise variance of the pixel to be estimated is obtained based on the region to which the pixel to be estimated belongs.

[0088] Since the short frame image provides all the information of the bright area of ​​the image to be estimated, when the pixel to be estimated is located in the bright area, step S132 is executed to calculate the noise variance var of the pixel corresponding to the short frame. s , combined with the exposure ratio r exp , obtain the noise variance var of the pixel to be estimated hdr , wherein the short frame corresponding pixel bits are pixel bits in the short frame image corresponding to the pixel bits to be estimated.

[0089] Therefore, when the pixel to be estimated is located in the bright area, the noise variance of the pixel to be estimated is:

[0090]

[0091] Among them, var hdr represents the noise variance of the pixel to be estimated in the image to be estimated; var s represents the noise variance of the pixel corresponding to the short frame, r exp Indicates exposure ratio.

[0092] Since the long frame image provides all the information of the dark area of ​​the image to be estimated, when the pixel position to be estimated is located in the dark area, step S131 is executed. When the pixel position to be estimated is located in the dark area, the noise variance of the pixel position to be estimated is obtained based on the noise variance of the pixel position corresponding to the long frame, wherein the pixel position corresponding to the long frame is the pixel position in the long frame image corresponding to the pixel position to be estimated.

[0093] Therefore, when the pixel to be estimated is located in the dark area, the noise variance of the pixel to be estimated is:

[0094] var hdr =var L

[0095] Among them, var hdr represents the noise variance of the pixel to be estimated in the image to be estimated; var L Indicates the noise variance of the pixel bits corresponding to the long frame.

[0096] The long frame image and the short frame image together provide image information of the transition area of ​​the image to be estimated. Therefore, when the pixel to be estimated is located in the transition area, step S133 is performed to calculate the noise variance of the selected average brightness value of the long frame and the selected average brightness value of the short frame based on the exposure ratio r exp , the synthesis parameter offset and the maximum display value L max , according to the average brightness value of the estimated neighborhood in the image to be estimated, the noise variance of the pixel to be estimated is obtained, wherein the pixel to be estimated is located in the estimated neighborhood, and the noise variance of the average brightness value selected by the long frame refers to the average brightness value μ L The noise variance var of the long frame image when it is equal to the first preset value L (μ L = first preset value), the noise variance of the average brightness value selected by the short frame refers to the average brightness value μ S The noise variance var of the short frame image when it is equal to the second preset value S (μ S= second preset value).

[0097] It should be noted that, in some embodiments of the present invention, the first preset value is equal to the synthesis parameter offset; the second preset value is equal to the maximum display value L max With exposure ratio r exp Therefore, the noise variance of the selected average brightness value of the long frame refers to the average brightness value μ L The noise variance var of the long frame image when it is equal to the synthesis parameter offset, that is, the first preset value L (μ L = offset), the noise variance of the selected average brightness value of the short frame refers to the average brightness value μ S The noise variance of the short frame image when it is equal to the second preset value

[0098] In some embodiments of the present invention, when the pixel to be estimated is located in the transition area, the noise variance of the pixel to be estimated is obtained according to a quadratic function of an average brightness value of an estimated neighborhood in the image to be estimated, and the quadratic function formula is:

[0099]

[0100] Among them, (x0, y0) represents the vertex of the quadratic function, and (x1, y1) represents another data point on the quadratic function except the vertex.

[0101] Specifically, in the step of obtaining the noise variance of the pixel to be estimated according to the quadratic function of the average brightness value of the estimated neighborhood in the image to be estimated, the vertex of the quadratic function is (offset, var L (μ L = first preset value)), and a data point of the quadratic function is Among them, offset represents the synthesis parameter, L max Indicates the maximum display value, var L (μ L = first preset value) represents the noise variance of the average brightness value selected by the long frame, var S (μ S = second preset value) represents the noise variance of the average brightness value selected by the short frame, μ L Indicates the average brightness value of the estimated neighborhood in the long frame image, μ S Represents the average brightness value of the estimated neighborhood in the short frame image.

[0102] In some embodiments, the first preset value is equal to the synthesis parameter offset; the second preset value is equal to the maximum display value L max With exposure ratio rexp The ratio of, therefore, the vertex of the quadratic function is (offset, var L (μ L = offset), and the quadratic function passes through the point

[0103] Therefore, when the pixel to be estimated is located in the transition area, the noise variance of the pixel to be estimated is:

[0104] var hdr = k·(μ hdr -offset) 2 +b

[0105]

[0106] b=var L (μ L =First preset value

[0107] Among them, var hdr Represents the noise variance of the pixel to be estimated, var S Indicates the noise variance of the pixel corresponding to the short frame, var L represents the noise variance of the pixel corresponding to the long frame, r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value, var S (μ S = second preset value) represents the noise variance of the selected average brightness value of the short frame, var L (μ L = first preset value) represents the noise variance of the selected average brightness value of the long frame, μ L Indicates the average brightness value of the estimated neighborhood in the long frame image, μ S Represents the average brightness value of the estimated neighborhood in the short frame image.

[0108] Among them, the exposure ratio r exp , synthesis parameter offset and maximum display value L max are all fixed parameters; the noise variance var of the short frame image s And the noise variance var of the long frame image L It can be calculated by Poisson noise model and Gaussian noise model according to the characteristics of image sensor. exp , synthesis parameter offset and maximum display value L max When determining, the noise variance var of the average brightness value selected by the short frame S (μ S= second preset value) and the noise variance var of the selected average brightness value of the long frame L (μ L = first preset value) is also a fixed value.

[0109] It can be seen that when the pixel to be estimated is located in the transition zone, in the noise variance of the pixel to be estimated, k and b are only related to the image sensor and the camera parameters, and have nothing to do with the image information itself. Therefore, for a certain image sensor and camera parameters, the noise variance var of the pixel to be estimated is hdr The average brightness value μ of the estimated neighborhood hdr The correspondence relationship only needs to be calculated once, without frame-by-frame or pixel-by-pixel calculation, which can save hardware resources and is especially suitable for applications with high real-time requirements.

[0110] It should be noted that after obtaining the noise variance of the pixel to be estimated, the noise variances of all the pixel to be estimated in the image to be estimated are obtained, so as to obtain the noise estimation result of the image to be estimated.

[0111] Combined with reference Figures 5 to 8 , showing Figure 1 The noise estimation result obtained by the noise estimation embodiment is shown in black circles, where Figure 5 yes Figure 4 The noise estimation results of the entire area of ​​the image to be estimated are shown. Figure 6 yes Figure 4 The noise estimation result of the dark area of ​​the image to be estimated is shown in FIG. Figure 7 yes Figure 4 The noise estimation result of the bright area of ​​the image to be estimated is shown in FIG. Figure 8 yes Figure 4 The noise estimation result of the transition area of ​​the image to be estimated is shown.

[0112] It should be noted that Figures 5 to 8 In the figure, the horizontal axis represents the average brightness value of the estimated neighborhood in the image to be estimated (unit: DN); the vertical axis represents the noise variance (unit: DN 2 ).

[0113] It should be noted that Figures 5 to 8 It also shows Figure 4 The actual statistical result of the noise variance of each gray scale of the image to be estimated is shown in black solid line. Since the gray scale image is equivalent to superimposing noise on the uniform average brightness value, the actual noise variance of each gray scale can be accurately calculated from the image.

[0114] It can be seen from this that Figure 1The noise estimation result obtained by the embodiment of the noise estimation method shown is almost completely consistent with the actual statistical result. Therefore, the noise estimation method has high accuracy and can effectively estimate the noise situation of the synthesized image to be estimated.

[0115] Correspondingly, the present invention also provides a noise estimation device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the noise estimation method of the present invention.

[0116] and a storage medium, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the noise estimation method of the present invention are executed.

[0117] Since the processor of the noise estimation device is used to execute the steps of the noise estimation method of the present invention; and the computer instructions stored in the storage medium execute the steps of the noise estimation method of the present invention when running, the specific technical solutions of the noise estimation device and the storage medium refer to the embodiments of the noise estimation method mentioned above, and the present invention will not be repeated here.

[0118] In addition, the present invention also provides an image processing method, comprising: performing noise estimation and obtaining a noise estimation result, wherein in the step of performing noise estimation, noise estimation is performed by the noise estimation method of the present invention.

[0119] In some embodiments of the present invention, noise reduction processing is performed according to the noise estimation result. The noise estimation method can accurately estimate the noise of a synthetic image with a high dynamic range, and the noise reduction processing performs noise reduction according to the noise estimation result, so the noise reduction effect is good, the noise points can be effectively reduced, and the image quality can be improved.

[0120] Correspondingly, the present invention also provides an image processing device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the image processing method of the present invention.

[0121] A storage medium is provided, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the image processing method of the present invention are executed.

[0122] Since the processor of the image processing device is used to execute the steps of the image processing method of the present invention; the computer instructions stored in the storage medium execute the steps of the image processing method of the present invention when running, the specific technical solutions of the image processing device and the storage medium refer to the embodiments of the aforementioned image processing method, and the present invention will not be repeated here.

[0123] In summary, the image to be estimated synthesized by the long frame image and the short frame image is partitioned: when the pixel to be estimated is located in the dark area, noise estimation is performed based on the long frame image; when the pixel to be estimated is located in the bright area, noise estimation is performed based on the short frame image and in combination with the exposure ratio; when the pixel to be estimated is located in the transition area, noise estimation is performed based on the long frame image and the short frame image and in combination with preset values ​​such as synthesis parameters, exposure ratio and maximum display value. Therefore, the noise estimation method of the present invention can accurately estimate the noise conditions of each area of ​​the synthesized image with a high dynamic range, and can effectively improve the effect of subsequent noise reduction operations.

[0124] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A noise estimation method, characterized in that: include: Based on a preset exposure ratio, obtaining a long frame image and a short frame image, wherein a ratio of an exposure amount of the long frame image to an exposure amount of the short frame image is equal to the exposure ratio; According to the long-frame image and the short-frame image, in combination with the exposure ratio and the preset synthesis parameter, an image to be estimated is synthesized, the image to be estimated has pixel bits to be estimated, the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the long-frame image, and the pixel bits in the image to be estimated correspond one-to-one with the pixel bits in the short-frame image, wherein the image to be estimated includes: a bright area, a dark area and a transition area, an area where the average brightness of all pixel bits in the estimated neighborhood is greater than or equal to the maximum display value is the bright area of ​​the image to be estimated, an area where the average brightness of all pixel bits in the estimated neighborhood is less than the maximum display value and greater than the synthesis parameter is the transition area of ​​the image to be estimated, and an area where the average brightness of all pixel bits in the estimated neighborhood is less than or equal to the synthesis parameter is the dark area of ​​the image to be estimated; According to the long frame image and the short frame image, noise estimation is performed on the image to be estimated. The step of performing noise estimation on the image to be estimated includes: When the pixel to be estimated is located in the dark area, the noise variance of the pixel to be estimated is obtained according to the noise variance of the pixel corresponding to the long frame, wherein the pixel corresponding to the long frame is the pixel corresponding to the pixel to be estimated in the long frame image; When the pixel to be estimated is located in the bright area, the noise variance of the pixel to be estimated is obtained according to the noise variance of the pixel corresponding to the short frame in combination with the exposure ratio, wherein the pixel corresponding to the short frame is the pixel corresponding to the pixel to be estimated in the short frame image; When the pixel to be estimated is located in the transition zone, based on the noise variance of the selected average brightness value of the long frame and the noise variance of the selected average brightness value of the short frame, in combination with the exposure ratio, the synthesis parameter and the preset maximum display value, according to the average brightness value of the estimated neighborhood in the image to be estimated, the noise variance of the pixel to be estimated is obtained, wherein the pixel to be estimated is located in the estimated neighborhood, the noise variance of the selected average brightness value of the long frame refers to the noise variance of the long frame image when the average brightness value is equal to a first preset value, and the noise variance of the selected average brightness value of the short frame refers to the noise variance of the short frame image when the average brightness value is equal to a second preset value, and in the step of performing noise estimation on the image to be estimated, when the pixel to be estimated is located in the transition zone, the noise variance of the pixel to be estimated is obtained according to the quadratic function of the average brightness value of the estimated neighborhood in the image to be estimated.

2. The noise estimation method according to claim 1, characterized in that: In the step of obtaining the noise variance of the pixel to be estimated according to the quadratic function of the average brightness value of the estimated neighborhood in the image to be estimated, the vertex of the quadratic function is (offset, var L (μ L = first preset value)), and a data point of the quadratic function is (L max , r 2 exp var s (μ s = second preset value)), where offset represents a synthesis parameter, L max Indicates the maximum display value, r exp Indicates exposure ratio, var L (μ L = first preset value) represents the noise variance of the average brightness value selected by the long frame, var s (μ s = second preset value) represents the noise variance of the selected average brightness value of the short frame.

3. The noise estimation method according to any one of claims 1 to 2, characterized in that: In the step of performing noise estimation on the image to be estimated, When the pixel to be estimated is located in the bright area, the noise variance of the pixel to be estimated is: When the pixel to be estimated is located in the dark area, the noise variance of the pixel to be estimated is: where hdr =where L ; When the pixel to be estimated is located in the transition area, the noise variance of the pixel to be estimated is: Var hdr =k (μ hdr -offset) 2 +b b=var L (μ L =First preset value Among them, var hdr Represents the noise variance of the pixel to be estimated, var s Indicates the noise variance of the pixel corresponding to the short frame, var L represents the noise variance of the pixel corresponding to the long frame, r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value, var s (μ s = second preset value) represents the noise variance of the selected average brightness value of the short frame, var L (μ L =first preset value) represents the noise variance of the selected average brightness value of the long frame.

4. The noise estimation method according to claim 1, characterized in that: In the step of synthesizing the image to be estimated, the image to be estimated is: Wherein, hdr represents the brightness value of the pixel bit of the image to be estimated; S represents the brightness value of the corresponding pixel bit in the short frame image, L represents the brightness value of the corresponding pixel bit in the long frame image, and r exp represents the exposure ratio, offset represents the synthesis parameter, L max Indicates the maximum display value.

5. The noise estimation method according to claim 1 or 4, characterized in that: Also includes: After synthesizing the image to be estimated and before obtaining the noise variance of the pixel to be estimated, the region to which the pixel to be estimated belongs is obtained.

6. The noise estimation method according to claim 1, characterized in that: Also includes: After obtaining the long frame image and the short frame image, obtaining the noise variance of the long frame image; The noise variance of the short frame image is obtained.

7. The noise estimation method according to claim 6, characterized in that: In the step of obtaining the noise variance of the long frame image, the noise variance is obtained based on at least one of a Poisson noise model and a Gaussian noise model; In the step of obtaining the noise variance of the short frame image, the noise variance is obtained based on at least one of a Poisson noise model and a Gaussian noise model.

8. The noise estimation method according to claim 1, characterized in that: The first preset value is equal to the synthesis parameter; the second preset value is equal to the ratio of the maximum display value to the exposure ratio.

9. A noise estimation device, comprising: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the processor performs the steps of the noise estimation method according to any one of claims 1 to 8.

10. A storage medium, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, characterized in that: When the computer instructions are executed, the steps of the noise estimation method according to any one of claims 1 to 8 are executed.

11. An image processing method, characterized in that: include: Noise estimation is performed to obtain a noise estimation result, wherein in the step of performing noise estimation, noise estimation is performed by the noise estimation method according to any one of claims 1 to 8.

12. The image processing method according to claim 11, characterized in that: Also includes: Noise reduction processing is performed according to the noise estimation result.

13. An image processing device, comprising: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the processor performs the steps of the image processing method according to any one of claims 11 to 12.

14. A storage medium, wherein the storage medium is a non-volatile storage medium or a non-transient storage medium, and a computer instruction is stored thereon, wherein: When the computer instructions are executed, the steps of the image processing method according to any one of claims 11 to 12 are executed.

Images