High dynamic range image noise reduction method and device

Abstract

The invention provides an image denoising method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the variance stabilization transformation of a to-be-denoised image according to a preset noise model parameter, and obtaining a first intermediate image; performing noise reduction processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image; wherein the signal-to-noise ratio fluctuation curve is used for representing difference information of noise variances corresponding to different exposure gains of the to-be-denoised image; and performing variance stabilization inverse transformation on the second intermediate image according to the noise model parameters to obtain a denoised image. The image noise reduction method is used for improving the noise reduction effect of the high dynamic range image.

Description

High dynamic range image noise reduction method and device Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an image noise reduction method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of intelligent driving technology, CMOS (Complementary Metal Oxide Semiconductor, CMOS for short) image sensors for in-vehicle application scenarios have emerged. The high dynamic range images captured by CMOS image sensors have embedded information of multiple exposures, and have high sensitivity and high dynamic range. The dynamic range of images captured by traditional image sensors is usually around 70DB, while the dynamic range of images captured by CMOS image sensors can reach more than 100DB, or even 180DB (exceeding the human eye's ability to perceive the environment). The advantage of CMOS image sensors is that they can accurately restore the shape and color of traffic lights in highlighted areas, and can also retain details in dark areas under harsh lighting conditions. In order to restore the high dynamic range information in the images captured by the CMOS image sensor, it is necessary to perform noise reduction processing on the image information output by the CMOS image sensor in the image processing module to improve the signal-to-noise ratio.

[0003] Currently, when performing noise reduction on high dynamic range images, linear noise models are usually used directly. However, since the noise characteristics in high dynamic range images do not follow the linear model, the noise reduction is uneven and the noise reduction effect is poor.

[0004] Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image denoising method, apparatus, electronic device and computer-readable storage medium to improve the effect of denoising high dynamic range images.

[0006] In a first aspect, the present application provides an image denoising method, comprising: performing a variance-stabilized transformation on an image to be denoised according to preset noise model parameters to obtain a first intermediate image; performing denoising processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image; wherein the signal-to-noise ratio fluctuation curve is used to characterize difference information of noise variance corresponding to the image to be denoised at different exposure gains; and performing a variance-stabilized inverse transformation on the second intermediate image according to the noise model parameters to obtain a denoised image.

[0007] In an embodiment of the present application, through the above-mentioned method, when denoising a high dynamic range image, the high dynamic range image is denoised based on the signal-to-noise ratio fluctuation curve and the noise reduction algorithm that characterize the difference information of the noise variance corresponding to the image to be denoised at different exposure gains. During the noise reduction process, it is ensured that the estimation of the noise variance distribution is accurate at each brightness (i.e., at different exposure gains), thereby improving the noise reduction effect. In addition, compared to the noise reduction method of splitting the high dynamic range image into multiple frames of low dynamic range images, and then denoising the multiple frames of low dynamic range images separately and then reconstructing them into high dynamic range images, the image denoising method provided in the embodiment of the present application directly denoises the high dynamic range image, avoiding the resource consumption caused by processing multiple frames of low dynamic range images.

[0008] In an optional embodiment, before performing a variance stabilization transform on the image to be denoised according to preset noise model parameters to obtain a first intermediate image, the method further includes: obtaining multiple groups of sample images captured by an image sensor at different exposure gains; wherein the image sensor is an image sensor that captures the image to be denoised, and each group of sample images corresponds to an exposure gain; and determining noise model parameters based on the multiple groups of sample images and a preset noise model.

[0009] In an embodiment of the present application, an image sensor that captures the image to be denoised is used to capture multiple groups of sample images at different exposure gains, and noise model parameters are determined by fitting the multiple groups of sample images to avoid deviations in the noise model parameters due to different image sensor models, thereby improving the effect of subsequent denoising of the image to be denoised.

[0010] In an optional embodiment, the multiple groups of sample images are obtained by photographing a gray card in a preset environment using the image sensor at different exposure gains.

[0011] In an optional embodiment, the determining of noise model parameters based on the multiple groups of sample images and a preset noise model includes: denoising each group of sample images separately to obtain each group of sample images after denoising; for each group of sample images, determining the noise variance of the group of sample images based on the group of sample images and their corresponding denoised sample images, and determining the first parameter and second parameter corresponding to the group of sample images based on the group of sample images and their corresponding noise variance; determining the first hyperparameter and second hyperparameter of the noise model based on the sensitivity of each group of sample images and the first parameter and second parameter corresponding to each group of sample images; and determining the noise model parameters based on the baseline sensitivity of the image to be denoised, the first hyperparameter and the second hyperparameter.

[0012] In an embodiment of the present application, firstly, the first and second parameters corresponding to the plurality of groups of sample images are determined using each group of sample images and their corresponding noise variances. Then, the first and second hyperparameters of the noise model are determined based on the sensitivity of each group of sample images and their corresponding first and second parameters. The first and second hyperparameters are equivalent to determining the functional relationship between the sensitivity and the first and second parameters. Furthermore, when an image sensor is used to capture the image to be denoised, the noise model parameters corresponding to the image to be denoised are determined in combination with the baseline sensitivity corresponding to the image to be denoised and the first and second hyperparameters. In this manner, the determined noise model parameters are more closely matched to the image to be denoised.

[0013] In an optional embodiment, the reference sensitivity of the image to be denoised is the sensitivity corresponding to the longest exposure frame in the image to be denoised.

[0014] In this embodiment of the present application, a high dynamic range image contains multiple independent exposures at different exposure gains. The longest exposure frame is the one with the longest exposure time and has the best signal-to-noise ratio. By using the sensitivity corresponding to the longest exposure frame in the image to be denoised as the reference sensitivity and then determining the noise model parameters based on the reference sensitivity, the noise reduction effect during subsequent image denoising is improved.

[0015] In an optional embodiment, before performing noise reduction processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image, the method further includes: obtaining exposure information in the image to be denoised; wherein the exposure information includes exposure gains and exposure times corresponding to each of a plurality of independent exposures; determining respective exposure ratios based on the exposure gains and exposure times corresponding to each of the independent exposures; and determining the signal-to-noise ratio fluctuation curve based on the exposure ratios corresponding to each of the independent exposures.

[0016] In this embodiment of the present application, the exposure ratio corresponding to each independent exposure in the image to be denoised is analyzed to determine a signal-to-noise ratio fluctuation curve. When the signal-to-noise ratio fluctuation curve is subsequently combined with a preset noise reduction algorithm to perform noise reduction on the image to be denoised, an accurate estimate of the noise variance distribution is effectively guaranteed at different exposure gains. This allows the denoised image to maintain detail at all brightness levels, improving the noise reduction effect.

[0017] In an optional embodiment, the exposure ratios of the independent exposures are determined based on the exposure gains and exposure times corresponding to the independent exposures, including: i =(exp i ×gain i ) / (EXP×GAIN) determines the exposure ratio corresponding to each independent exposure; where Ratio iis the exposure ratio corresponding to the i-th independent exposure, exp i is the exposure duration corresponding to the i-th independent exposure, gain i is the exposure gain corresponding to the i-th independent exposure, EXP is the exposure duration of the benchmark exposure, and GAIN is the exposure gain corresponding to the benchmark exposure.

[0018] In an optional embodiment, the preset signal-to-noise ratio fluctuation curve and the preset noise reduction algorithm are used to perform noise reduction processing on the first intermediate image to obtain the second intermediate image, including: determining the similarity between each pixel point in the first intermediate image and its adjacent pixel points; correcting the similarity according to the signal-to-noise ratio fluctuation curve to obtain the corrected similarity of each pixel point; and performing noise reduction on each pixel point according to the corrected similarity of each pixel point and the preset noise reduction algorithm to obtain the second intermediate image.

[0019] In a second aspect, the present application provides an image denoising device, comprising: a first determination module, configured to perform a variance-stabilised transform on an image to be denoised according to preset noise model parameters to obtain a first intermediate image; a second determination module, configured to perform denoising processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image; wherein the signal-to-noise ratio fluctuation curve is used to characterise difference information of noise variance corresponding to the image to be denoised at different exposure gains; and a third determination module, configured to perform a variance-stabilised inverse transform on the second intermediate image according to the noise model parameters to obtain a denoised image.

[0020] In an optional embodiment, the device also includes: a fourth determination module, used to obtain multiple groups of sample images captured by the image sensor under different exposure gains; wherein the image sensor is an image sensor that captures the image to be denoised, and each group of sample images corresponds to an exposure gain; and determine the noise model parameters based on the multiple groups of sample images and a preset noise model.

[0021] In an optional embodiment, the multiple groups of sample images are obtained by photographing a gray card in a preset environment using the image sensor at different exposure gains.

[0022] In an optional embodiment, the fourth determination module is specifically used to perform noise reduction on each group of sample images separately to obtain each group of sample images after noise reduction; for each group of sample images, the noise variance of the group of sample images is determined based on the group of sample images and their corresponding noise reduced sample images, and the first parameters and second parameters corresponding to the group of sample images are determined based on the group of sample images and their corresponding noise variance; the first hyperparameters and second hyperparameters of the noise model are determined based on the sensitivity of each group of sample images and the first parameters and second parameters corresponding to each group of sample images; the noise model parameters are determined based on the baseline sensitivity of the image to be denoised, the first hyperparameters and the second hyperparameters.

[0023] In an optional embodiment, the reference sensitivity of the image to be denoised is the sensitivity corresponding to the longest exposure frame in the image to be denoised.

[0024] In an optional embodiment, the device further includes: a fifth determination module, configured to obtain exposure information in the image to be denoised; wherein the exposure information includes exposure gains and exposure times corresponding to multiple independent exposures; determining respective exposure ratios based on the exposure gains and exposure times corresponding to each independent exposure; and determining the signal-to-noise ratio fluctuation curve based on the exposure ratios corresponding to each independent exposure.

[0025] In an optional implementation manner, the fifth determination module is specifically configured to calculate the value of Ratio according to the formula: i =(exp i × gain i ) / (EXP×GAIN) determines the exposure ratio corresponding to each independent exposure; where Ratio i is the exposure ratio corresponding to the i-th independent exposure, exp i is the exposure duration corresponding to the i-th independent exposure, gain i is the exposure gain corresponding to the i-th independent exposure, EXP is the exposure duration of the benchmark exposure, and GAIN is the exposure gain corresponding to the benchmark exposure.

[0026] In an optional embodiment, the second determination module is specifically used to determine the similarity between each pixel point in the first intermediate image and its adjacent pixel points; correct the similarity according to the signal-to-noise ratio fluctuation curve to obtain the corrected similarity of each pixel point; and perform noise reduction on each pixel point based on the corrected similarity of each pixel point and the preset noise reduction algorithm to obtain the second intermediate image.

[0027] In a third aspect, the present application provides an electronic device comprising: a processor and a memory; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method described in any one of the aforementioned embodiments.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are read and executed by a computer, the method as described in any one of the aforementioned embodiments is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0030] FIG1 is a flow chart of an image noise reduction method provided by an embodiment of the present application;

[0031] FIG2 is a flow chart of determining noise model parameters according to an embodiment of the present application;

[0032] FIG3 is a flowchart of a specific implementation of step A2 provided in an embodiment of the present application;

[0033] FIG4 is a flow chart of determining a signal-to-noise ratio fluctuation curve according to an embodiment of the present application;

[0034] FIG5 is a flowchart of determining a second intermediate image according to an embodiment of the present application;

[0035] 6a-6d are schematic diagrams of noise distribution determined using the image noise reduction method provided by an embodiment of the present application;

[0036] FIG7 is a structural block diagram of an image noise reduction device provided in an embodiment of the present application;

[0037] FIG8 is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0039] The purpose of the embodiments of the present application is to provide an image denoising method, apparatus, electronic device and computer-readable storage medium to improve the effect of denoising high dynamic range images.

[0040] Please refer to FIG1 , which is a flowchart of an image noise reduction method provided in an embodiment of the present application. The image noise reduction method may include the following contents:

[0041] Step 101: performing a variance stabilization transform on the image to be denoised according to preset noise model parameters to obtain a first intermediate image.

[0042] Step 102: Perform noise reduction processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image.

[0043] Step 103: Perform a variance-stabilized inverse transform on the second intermediate image according to the noise model parameters to obtain a denoised image.

[0044] In an embodiment of the present application, the image to be denoised may be a high dynamic range image. The signal-to-noise ratio fluctuation curve characterizes the difference information of the noise variance corresponding to the image to be denoised at different exposure gains. Since the high dynamic range image range embeds brightness information at multiple different exposure gains, noise at different exposure gains will be introduced. If only the preset noise reduction algorithm is used for noise reduction, the problem of uneven noise reduction will occur. Therefore, the first intermediate image is subjected to noise reduction processing, and the first intermediate image is subjected to noise reduction processing together with the signal-to-noise ratio fluctuation curve and the noise reduction algorithm, thereby improving the accuracy of noise reduction. Finally, the second intermediate image obtained after noise reduction is subjected to a variance-stabilized inverse transform, and the second intermediate image is restored from the variance-stabilized nonlinear domain to the linear domain to obtain a noise-reduced image.

[0045] Through the above method, when denoising a high dynamic range image, the high dynamic range image is denoised based on the signal-to-noise ratio fluctuation curve and the noise reduction algorithm that characterize the difference information of the noise variance corresponding to the image to be denoised at different exposure gains. During the noise reduction process, it is ensured that the estimation of the noise variance distribution is accurate at each brightness (i.e., at different exposure gains), thereby improving the noise reduction effect. In addition, compared to the noise reduction method of splitting the high dynamic range image into multiple frames of low dynamic range images, and then denoising the multiple frames of low dynamic range images separately and then reconstructing them into high dynamic range images, the image denoising method provided in the embodiment of the present application directly denoises the high dynamic range image, avoiding the resource consumption caused by processing multiple frames of low dynamic range images.

[0046] The above steps are described in detail below.

[0047] Step 101: performing a variance stabilization transform on the image to be denoised according to preset noise model parameters to obtain a first intermediate image.

[0048] In an embodiment of the present application, a variance-stabilized transformation is first performed on the image to be denoised according to preset noise model parameters, and the image to be denoised is converted from a linear domain to a variance-stabilized nonlinear domain to obtain a first intermediate image for subsequent image denoising processing.

[0049] Specifically, the preset noise model may be a Poisson-Gaussian noise model. Based on the noise model parameters in the Poisson-Gaussian noise model, a variance-stabilized transformation is performed on the denoised image to determine the first intermediate image. The variance-stabilized transformation of the denoised image may be determined by the following formula:

[0050] Where y is the value of each pixel in the first intermediate image, v is the value of each pixel in the image to be denoised, and a and b are noise model parameters.

[0051] It can be understood that the image denoising method provided in the embodiment of the present application can be applied to an image denoising device, which is pre-configured with noise model parameters. When the image to be denoised is input into the image denoising device, the image denoising device directly performs the above-mentioned processing on the image to be denoised according to the noise model parameters to obtain a first intermediate image.

[0052] The following describes how to determine the noise model parameters.

[0053] Referring to FIG. 2 , before step 101 , the image noise reduction method provided in the embodiment of the present application may include the following steps:

[0054] A1: Acquire multiple sets of sample images captured by the image sensor at different exposure gains.

[0055] A2: Determine noise model parameters based on multiple groups of sample images and a preset noise model.

[0056] In the embodiment of the present application, in order to improve the accuracy of the noise model parameters, the image sensor for acquiring the multiple groups of sample images at different exposure gains and the image sensor for acquiring the image to be denoised are the same image sensor.

[0057] Specifically, the multiple groups of sample images are obtained by shooting a gray card in a preset environment using an image sensor with different exposure gains.

[0058] The following describes a method for obtaining multiple groups of sample images with a specific example.

[0059] (1) In a standard laboratory, place a uniform gray plate with a length of 1.5 meters and a width of 1 meter. The image sensor's viewing plane is parallel to the gray plate, ensuring that the entire field of view is filled with gray plates and that the field of view covers 80% of the gray plate area.

[0060] (2) Place a flat light source 1-2 meters away from the plate, and the flat light source is parallel to the gray plate.

[0061] (3) Turn on the automatic exposure module in the image sensor and adjust the brightness of the flat panel light source so that the exposure gain of the automatic exposure module is equal to 1.

[0062] (4) Continuously shoot 20-50 frames of gray card images to obtain a set of sample images with an exposure gain of one.

[0063] (5) Return to step (3), and by adjusting the brightness of the flat panel light source, capture sample images with different exposure gains, such as exposure gains of 2x, 4x, 8x, 16x, etc., thereby obtaining multiple sets of sample images.

[0064] It should be noted that in step (3) above, the brightness of the flat panel light source is adjusted, and the automatic exposure module, under the action of the automatic exposure control (AEC) algorithm, makes the exposure gain of the automatic exposure module 1. In addition, the automatic exposure control algorithm is configured to ensure that the brightness range of the sample image output by the image sensor is moderate. For example, when the image sensor uses 8 bits to represent brightness, the automatic exposure control algorithm is configured to ensure that the brightness range of the sample image output by the image sensor is approximately 100, 128, and 130.

[0065] The above example is only an example provided in the embodiment of the present application. The embodiment of the present application does not specifically limit the size of the gray plate and the field of view angle coverage.

[0066] After obtaining multiple groups of sample images, noise model parameters are determined according to the multiple groups of sample images and a preset noise model.

[0067] As an optional implementation, referring to FIG3 , step A2 may include the following steps:

[0068] A21: Perform noise reduction on each group of sample images to obtain noise-reduced sample images.

[0069] A22: For each group of sample images, determine the noise variance of the group of sample images based on the group of sample images and the corresponding denoised sample images, and determine the first parameter and the second parameter corresponding to the group of sample images based on the group of sample images and the corresponding noise variance.

[0070] A23: Determine first hyperparameters and second hyperparameters of the noise model according to the sensitivity of each group of sample images and the first parameters and second parameters corresponding to each group of sample images.

[0071] A24: Determine noise model parameters according to the baseline sensitivity of the image to be denoised, the first hyperparameter, and the second hyperparameter.

[0072] In the embodiment of the present application, noise reduction is first performed on a set of sample image data at each exposure gain. Specifically, frame matching and temporal superposition are performed to obtain each set of sample images after noise reduction. The noise variance is then determined by taking the difference between the sample image and each set of sample images after noise reduction.

[0073] Taking the noise model as the Poisson-Gaussian noise model as an example, the Poisson-Gaussian noise model can be expressed as:

[0074] var 2 =a*luma+b

[0075] Among them, var 2 is the noise variance, luma is the brightness information of the sample image, a is the first parameter, and b is the second parameter. By determining the noise variance based on the difference between each set of sample images and their corresponding denoised sample images, multiple sets of noise variances can be obtained. Substituting each set of noise variance and the brightness information of the sample image into the above formula, multiple sets of first and second parameters can be solved.

[0076] Each set of sample images corresponds to a sensitivity. The sensitivity is linearly related to the first parameter a, that is, it satisfies the following relationship:

[0077] a=k×ISO

[0078] Among them, k is the first hyperparameter and ISO is the sensitivity.

[0079] There is a linear relationship between the sensitivity and the square of the second parameter b, that is, the following relationship is satisfied:

[0080] b=p×ISO×ISO

[0081] Among them, p is the second hyperparameter and ISO is the sensitivity.

[0082] Therefore, the first hyperparameter and the second hyperparameter can be determined by substituting the sensitivity of each group of sample images and the first parameter and the second parameter corresponding to each group of sample images into the above two relational expressions.

[0083] After determining the first hyperparameter and the second hyperparameter, when it is necessary to determine the noise model parameters a and b of the image to be denoised, the baseline sensitivity of the image to be denoised, the first hyperparameter and the second hyperparameter are substituted into the above two formulas to calculate the noise model parameters a and b of the image to be denoised.

[0084] In some embodiments, the reference sensitivity is the sensitivity corresponding to the longest exposure frame in the image to be denoised. A high dynamic range image contains multiple independent exposures with different exposure gains. The longest exposure frame is the one with the longest exposure time and has the best signal-to-noise ratio. By using the sensitivity corresponding to the longest exposure frame in the image to be denoised as the reference sensitivity and determining the noise model parameters based on the reference sensitivity, the noise reduction effect during subsequent image denoising can be improved.

[0085] Step 102: Perform noise reduction processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image.

[0086] In the embodiment of the present application, the preset noise reduction algorithm may be a noise reduction algorithm that follows Gaussian distribution, such as a non-local mean filter algorithm, a median filter algorithm, a Gaussian filter algorithm, etc. This application does not make any specific limitation on this.

[0087] The signal-to-noise ratio fluctuation curve is determined according to the characteristics of the image sensor that acquires the image to be denoised. In some embodiments, the signal-to-noise ratio fluctuation curve may be pre-configured in the image denoising device for a specific image sensor.

[0088] In other embodiments, referring to FIG. 4 , before step 102 , the signal-to-noise ratio fluctuation curve may be determined in the following manner.

[0089] B1: Obtain exposure information of the image to be denoised.

[0090] B2: Determine the exposure ratio of each independent exposure based on the exposure gain and exposure time corresponding to each independent exposure.

[0091] B3: Determine the signal-to-noise ratio fluctuation curve based on the exposure ratio corresponding to each independent exposure.

[0092] In this embodiment of the present application, the image to be denoised contains multiple independent exposures with different exposure gains. The exposure information for the image to be denoised includes the exposure gain and exposure duration corresponding to each of the multiple independent exposures. The exposure gain and exposure duration corresponding to each independent exposure are pre-configured and determined by the image sensor before acquiring the image to be denoised. When determining the signal-to-noise ratio fluctuation curve, the configuration parameters of the image sensor can be directly obtained to determine the exposure gain and exposure duration corresponding to each independent exposure in the image to be denoised.

[0093] Specifically, the exposure ratio corresponding to each independent exposure can be determined by the following formula: Ratio i =(exp i ×gain i ) / (EXP×GAIN)

[0094] Among them, Ratio i is the exposure ratio corresponding to the i-th independent exposure, exp i is the exposure duration corresponding to the i-th independent exposure, gain i is the exposure gain corresponding to the i-th independent exposure, EXP is the exposure duration of the benchmark exposure, and GAIN is the exposure gain corresponding to the benchmark exposure.

[0095] In some embodiments, as described above, the longest exposure frame has the best signal-to-noise ratio. Therefore, the exposure duration corresponding to the longest exposure frame is used as the exposure duration of the baseline exposure, and the exposure gain corresponding to the longest exposure frame is used as the exposure gain corresponding to the baseline exposure.

[0096] In the above step B3, after the exposure ratio corresponding to each independent exposure is determined, the square root of the exposure ratio corresponding to each independent exposure is calculated, and the signal-to-noise ratio fluctuation curve is determined according to the square root of the exposure ratio of each independent exposure.

[0097] Specifically, the exposure ratio corresponding to the first independent exposure is Ratio1, and the value of the first independent exposure on the signal-to-noise ratio fluctuation curve is F1: Similarly, the exposure ratio corresponding to the second independent exposure is Ratio2, and the value of the second independent exposure on the signal-to-noise ratio fluctuation curve is F2: Similarly, the value of each independent exposure on the signal-to-noise ratio fluctuation curve is determined, thereby determining the signal-to-noise ratio fluctuation curve.

[0098] Further, referring to FIG5 , the above step 102 may include the following steps:

[0099] C1: Determine the similarity between each pixel in the first intermediate image and its adjacent pixels.

[0100] C2: Correct the similarity according to the signal-to-noise ratio fluctuation curve to obtain the corrected similarity of each pixel.

[0101] C3: Perform noise reduction on each pixel according to the corrected similarity of each pixel and a preset noise reduction algorithm to obtain the second intermediate image.

[0102] It is understood that the processing process is identical for each pixel in the first intermediate image. For ease of illustration, the specific processing of steps C1-C3 is described below using a pixel A in the first intermediate image and a non-local means filtering algorithm as the default noise reduction algorithm.

[0103] In step C1, multiple pixels surrounding pixel point A are determined as a pixel block, and a similarity D between the pixel block corresponding to pixel point A and the adjacent pixel blocks is calculated. Similarity D can be the average of the absolute value differences between the pixel block corresponding to pixel point A and the adjacent pixel blocks, or the Euclidean distance between the pixel block corresponding to pixel point A and the adjacent pixel blocks.

[0104] For example, consider an 8x8 pixel block surrounding pixel A as its representation. The pixel values ​​of all pixels within this block form a 64-dimensional vector. In a 16x16 window centered on pixel A, in addition to the 8x8 pixel block surrounding pixel A, there are 80 adjacent pixel blocks. Each pixel block can be represented as a 64-dimensional vector qi, where i ranges from 1 to 80, corresponding to each of these 80 adjacent pixel blocks.

[0105] When the mean of the absolute value difference is used as the similarity between the pixel block corresponding to pixel point A and multiple adjacent pixel blocks, the similarity D can be expressed as: Wherein, N is the number of adjacent pixel blocks adjacent to pixel point A.

[0106] When the Euclidean distance is used as the similarity between the pixel block corresponding to the pixel point A and multiple adjacent pixel blocks, the similarity D can be expressed as: Wherein, N is the number of adjacent pixel blocks adjacent to pixel point A.

[0107] In step C2, the value F of the independent exposure corresponding to pixel A on the signal-to-noise ratio fluctuation curve is determined based on the exposure gain corresponding to the pixel. Then, the corrected similarity D′ is determined according to the following formula:

[0108] D′=D×F.

[0109] In step C3, after determining the similarity D' corresponding to the modified pixel A, noise reduction is performed on each pixel using D' and the non-local means filtering algorithm to obtain the second intermediate image. It will be appreciated that the specific process of noise reduction using the non-local means filtering algorithm will not be further described here.

[0110] Step 103: Perform a variance-stabilized inverse transform on the second intermediate image according to the noise model parameters to obtain a denoised image.

[0111] In the embodiment of the present application, after steps 101-102, the second intermediate image obtained is in a nonlinear domain. Therefore, the second intermediate image in the nonlinear domain is converted back to a linear domain according to the noise model parameters to obtain a denoised image.

[0112] Specifically, the variance-stabilized inverse transform of the second intermediate image can be determined by the following formula:

[0113] Where y′ is the value of each pixel in the second intermediate image, v′ is the value of each pixel in the denoised image, and a and b are noise model parameters.

[0114] Please refer to Figures 6a-6d. Figure 6a shows the linear image noise model curve for the image to be denoised, Figure 6b shows the signal-to-noise ratio fluctuation curve, Figure 6c shows the actual noise distribution of the image to be denoised, and Figure 6d shows the noise distribution obtained by combining the linear image noise model curve for the image to be denoised corresponding to Figure 6a and the signal-to-noise ratio fluctuation curve for the image to be denoised corresponding to Figure 6b. By comparing Figures 6c and 6d, it can be seen that the noise distribution obtained by combining the linear image noise model curve and the signal-to-noise ratio fluctuation curve for the image to be denoised is very close to the actual noise distribution of the image to be denoised, indicating that the image denoising method provided in the embodiments of the present application can improve the noise reduction effect.

[0115] Based on the same inventive concept, please refer to FIG. 7 , which is a structural block diagram of an image noise reduction device provided in an embodiment of the present application. The image noise reduction device 700 may include:

[0116] A first determining module 701 is configured to perform a variance stabilization transform on the image to be denoised according to preset noise model parameters to obtain a first intermediate image;

[0117] A second determining module 702 is configured to perform noise reduction processing on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset noise reduction algorithm to obtain a second intermediate image, wherein the signal-to-noise ratio fluctuation curve is used to represent information about difference in noise variance of the image to be denoised at different exposure gains;

[0118] The third determining module 703 performs a variance-stabilized inverse transform on the second intermediate image according to the noise model parameters to obtain a denoised image.

[0119] In an optional embodiment, the device also includes: a fourth determination module, used to obtain multiple groups of sample images captured by the image sensor under different exposure gains; wherein the image sensor is an image sensor that captures the image to be denoised, and each group of sample images corresponds to an exposure gain; and determine the noise model parameters based on the multiple groups of sample images and a preset noise model.

[0120] In an optional embodiment, the multiple groups of sample images are obtained by photographing a gray card in a preset environment using the image sensor at different exposure gains.

[0121] In an optional embodiment, the fourth determination module is specifically used to perform noise reduction on each group of sample images separately to obtain each group of sample images after noise reduction; for each group of sample images, the noise variance of the group of sample images is determined based on the group of sample images and their corresponding noise reduced sample images, and the first parameters and second parameters corresponding to the group of sample images are determined based on the group of sample images and their corresponding noise variance; the first hyperparameters and second hyperparameters of the noise model are determined based on the sensitivity of each group of sample images and the first parameters and second parameters corresponding to each group of sample images; the noise model parameters are determined based on the baseline sensitivity of the image to be denoised, the first hyperparameters and the second hyperparameters.

[0122] In an optional embodiment, the reference sensitivity of the image to be denoised is the sensitivity corresponding to the longest exposure frame in the image to be denoised.

[0123] In an optional embodiment, the device further includes: a fifth determination module, configured to obtain exposure information in the image to be denoised; wherein the exposure information includes exposure gains and exposure times corresponding to multiple independent exposures; determining respective exposure ratios based on the exposure gains and exposure times corresponding to each independent exposure; and determining the signal-to-noise ratio fluctuation curve based on the exposure ratios corresponding to each independent exposure.

[0124] [Corrected 22.09.2023 according to Rule 91] In an optional embodiment, the fifth determination module is specifically configured to determine the value of the ratio according to the formula: i =(exp i ×gain i ) / (EXP×GAIN) determines the exposure ratio corresponding to each independent exposure; where Ratio i is the exposure ratio corresponding to the i-th independent exposure, exp i is the exposure duration corresponding to the i-th independent exposure, gain i is the exposure gain corresponding to the i-th independent exposure, EXP is the exposure duration of the benchmark exposure, and GAIN is the exposure gain corresponding to the benchmark exposure.

[0126] In an optional embodiment, the second determination module 702 is specifically used to determine the similarity between each pixel point in the first intermediate image and its adjacent pixel points; correct the similarity according to the signal-to-noise ratio fluctuation curve to obtain the corrected similarity of each pixel point; and perform noise reduction on each pixel point based on the corrected similarity of each pixel point and the preset noise reduction algorithm to obtain the second intermediate image.

[0127] The image denoising device 700 corresponds to the aforementioned image denoising method, and each functional module corresponds to each step of the aforementioned image denoising method. Therefore, the implementation of each functional module refers to the implementation of the image denoising method in the aforementioned embodiment and will not be repeated here.

[0128] Referring to FIG. 8 , an embodiment of the present application further provides an electronic device 800 , which can serve as an executor of the aforementioned image dynamic range compression method, and includes: a processor 810 and a memory 820 .

[0129] The memory 820 stores instructions that can be executed by the processor 810 . The instructions are executed by the processor 810 so that the processor 810 can perform the image noise reduction method in the aforementioned embodiment.

[0130] The processor 810 and the memory 820 may be connected via a communication bus, or may be connected via some communication modules, such as a wireless communication module, a Bluetooth communication module, a Wi-Fi communication module, a 2G (second generation mobile communication technology), a 3G (third generation mobile communication technology), a 4G (fourth generation mobile communication technology), and a 5G (fifth generation mobile communication technology) communication module.

[0131] Processor 810 can be an integrated circuit chip with signal processing capabilities. Processor 810 can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a digital signal processor, an application-specific integrated circuit, an off-the-shelf programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0132] The memory 820 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0133] It is understandable that the electronic device 800 may also include more common modules required by itself, which are not introduced one by one in the embodiments of the present application.

[0134] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the steps of the image noise reduction method in the above embodiment are executed.

[0135] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0138] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0139] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0140] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image denoising method, characterized in that: include: Performing a variance stabilization transformation on the image to be denoised according to preset noise model parameters to obtain a first intermediate image; Performing denoising on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset denoising algorithm to obtain a second intermediate image; wherein the signal-to-noise ratio fluctuation curve is used to characterize difference information of noise variance corresponding to the image to be denoised at different exposure gains; A variance-stabilized inverse transform is performed on the second intermediate image according to the noise model parameters to obtain a denoised image.

2. The method according to claim 1, characterized in that Before performing variance stabilization transformation on the image to be denoised according to preset noise model parameters to obtain the first intermediate image, the method further includes: Acquire multiple groups of sample images acquired by an image sensor under different exposure gains; wherein the image sensor is an image sensor that acquires the image to be denoised, and each group of sample images corresponds to an exposure gain; Noise model parameters are determined according to the multiple groups of sample images and a preset noise model.

3. The method according to claim 2, characterized in that The multiple groups of sample images are obtained by photographing a gray plate in a preset environment using the image sensor with different exposure gains.

4. The method according to claim 2, characterized in that: The determining of noise model parameters according to the plurality of groups of sample images and a preset noise model includes: Denoising each group of sample images respectively to obtain each group of sample images after denoising; For each group of sample images, determine the noise variance of the group of sample images according to the group of sample images and the corresponding sample images after noise reduction, and determine the first parameter and the second parameter corresponding to the group of sample images according to the group of sample images and the corresponding noise variance; Determine a first hyperparameter and a second hyperparameter of the noise model according to the sensitivity of each group of sample images and the first parameter and the second parameter corresponding to each group of sample images; The noise model parameter is determined according to the reference sensitivity of the image to be denoised, the first hyperparameter and the second hyperparameter.

5. The method according to claim 4, characterized in that The reference sensitivity of the image to be denoised is the sensitivity corresponding to the longest exposure frame in the image to be denoised.

6. The method according to claim 1, characterized in that Before performing noise reduction processing on the first intermediate image based on the preset signal-to-noise ratio fluctuation curve and the preset noise reduction algorithm to obtain the second intermediate image, the method further includes: Acquire exposure information in the image to be denoised; wherein the exposure information includes exposure gains and exposure durations corresponding to a plurality of independent exposures; Determine the exposure ratios of the independent exposures according to the exposure gains and exposure durations corresponding to the independent exposures; The signal-to-noise ratio fluctuation curve is determined according to the exposure ratios corresponding to the respective independent exposures.

7. The method according to claim 6, characterized in that Determining the exposure ratios according to the exposure gains and exposure durations corresponding to the independent exposures includes: According to the formula: Ratio i =(exp i ×gain i ) / (EXP×GAIN) determines the exposure ratio corresponding to each independent exposure; Among them, Ratio i is the exposure ratio corresponding to the ith independent exposure, exp i is the exposure duration corresponding to the ith independent exposure, gain i is the exposure gain corresponding to the i-th independent exposure, EXP is the exposure duration of the benchmark exposure, and GAIN is the exposure gain corresponding to the benchmark exposure.

8. The method according to claim 1, characterized in that The method of performing noise reduction processing on the first intermediate image based on the preset signal-to-noise ratio fluctuation curve and the preset noise reduction algorithm to obtain the second intermediate image includes: Determine the similarity between each pixel in the first intermediate image and its adjacent pixels; Correcting the similarity according to the signal-to-noise ratio fluctuation curve to obtain a corrected similarity of each pixel; According to the corrected similarity of each pixel and the preset noise reduction algorithm, noise reduction is performed on each pixel to obtain the second intermediate image.

9. An image noise reduction device, characterized in that: The device comprises: A first determination module is used to perform a variance stabilization transformation on the image to be denoised according to preset noise model parameters to obtain a first intermediate image; A second determination module is configured to perform denoising on the first intermediate image based on a preset signal-to-noise ratio fluctuation curve and a preset denoising algorithm to obtain a second intermediate image; wherein the signal-to-noise ratio fluctuation curve is used to characterize difference information of noise variance corresponding to the image to be denoised at different exposure gains; The third determination module performs a variance-stabilized inverse transform on the second intermediate image according to the noise model parameters to obtain a denoised image.

10. An electronic device, characterized in that: include: Processor, memory; The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a computer, the method according to any one of claims 1 to 8 is executed.

Images