Image noise level estimation method, image denoising method and medium

Abstract

The invention provides an image noise level estimation method, an image denoising method and a medium. The estimation method comprises the following steps: acquiring RGBW data output by a multi-line color sensor; based on the TDI delay amount of the multi-line color sensor, performing delay alignment on different rows of data to obtain a TDI superposition result; and estimating a noise level of the current position based on the brightness threshold, and outputting a guide image of the current position. On the basis of the estimation method, a TDI superposition result is used as a denoised input image based on a guide image, so that a denoised image corresponding to the input image is obtained. According to the characteristics of RGBW data, in combination with the influence of selection of the guide images on the denoising effect under different illuminance conditions, the guide image corresponding to each pixel is determined, so that each pixel is denoised according to texture information of reasonable illuminance, and the denoising effect is improved. On the basis, the invention further provides a corresponding hardware implementation mode so as to improve the calculation efficiency.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for estimating image noise levels, an image denoising method, and a medium. Background Technology

[0002] During image acquisition, image sensors often capture images with noticeable noise due to various reasons, reducing image quality and affecting user viewing and subsequent applications. Common filters such as mean filtering and Gaussian filtering, due to their isotropic nature, smooth edges in the image while removing noise, which is detrimental to feature extraction. Guided filtering and joint bilateral filtering rely on the texture information of the guide image, but the guide image itself is affected by noise. Especially in low-light conditions, due to the low signal-to-noise ratio of the image itself, using the image itself as the guide image cannot achieve good noise reduction results.

[0003] Especially for denoising multi-line color sensors, conventional techniques involve converting the RGBW data output by the sensor between RGB and HSI color spaces to ensure that the average brightness and color remain unchanged after denoising. However, the denoising process using color space conversion is complex and not conducive to hardware implementation.

[0004] Chinese patent CN 103745446 B discloses an image guided filtering method and system that accelerates filtering speed by feeding different values ​​into different streams for calculation. It also configures multiple GPU kernel functions for averaging, which, by being configured into different GPU streams, speeds up the entire guided filtering process, effectively reducing video processing latency and improving real-time performance. This patent differs from the inventive concept of this solution.

[0005] Chinese patents CN 118764721 A, CN 112104847 A, and CN 117441338 A propose different methods for image reconstruction based on RGBW data, all of which address different technical problems than the solution presented here. Summary of the Invention

[0006] This invention proposes a method for estimating image noise levels, an image denoising method, and a medium, thereby solving at least one of the aforementioned technical problems.

[0007] To achieve the above objectives, the present invention proposes the following technical solution: A method for estimating image noise levels includes: Acquire RGBW data output from the multi-line color sensor; Based on the TDI delay of the multi-line color sensor, the data of different rows are delayed and aligned; the pixel data of the R, G, B and W rows corresponding to the same position are accumulated to obtain the TDI superposition result; where the TDI delay is an integer. If the pixel value of row W at the current position is less than the brightness threshold, then the noise level at the current position is low illumination, and the pixel value of row W at the current position is output as the guide image; otherwise, the noise level at the current position is high illumination, and the average value of the pixel data of rows R, G, and B at the same position is used as the guide image.

[0008] This application also proposes an image denoising method, including: Using the estimation method described above, TDI overlay results and a guide image are obtained; the TDI overlay results are used as the input image for denoising. Set up a local window and calculate the variance of the guide image within the local window and the covariance between the guide image and the input image; Calculate the first coefficient of the local window based on the variance and covariance within the local window; calculate the second coefficient of the current window based on the first coefficient at the current position within the window, the mean pixel value of the guiding image, and the mean pixel value of the input image. The denoised image of the input image within the local window is calculated using the local mean of the first and second coefficients.

[0009] Furthermore, it also includes: within a local window, based on the color channel corresponding to the guiding image at any pixel position, such that when calculating the first coefficient and the second coefficient at that pixel position, the color channel corresponding to each pixel value in the neighborhood of that pixel position is the same as the color channel corresponding to the guiding image.

[0010] This application also proposes an image denoising method, including: Using the estimation method described above, guide images are acquired and cached row by row; the guide image is the mean of the pixel data of the R, G, and B rows corresponding to the same position, and the guide image is used as the input image for denoising; Based on the arrival order of the guide images, the local first coefficient and local second coefficient of each local window are calculated sequentially. Based on the linear model, the local denoised image of the local window is calculated. The denoising process of the guide images in each local window is completed sequentially to obtain the denoised image.

[0011] Furthermore, the local first coefficient includes: The square of each pixel in the guide image is calculated row by row. The sum of the squares of the pixels in each local window is calculated sequentially and used as the local sum of squares for that local window. The sum of the squares of each pixel in that local window is also calculated and used as the local sum. The variance of the local window is the difference between the local sum of squares and the square of the local sum. The variance of the local window is truncated at the high bit to obtain the newly assigned variance, which is used to construct a lookup table for the first coefficient corresponding to each pixel. Calculate the sum of the first coefficients within this local window, and use it as the local first coefficient.

[0012] Furthermore, the local second coefficient includes: Based on the local sum and the first coefficient of the current local window, calculate the second coefficient corresponding to each pixel; calculate the sum of the second coefficients within the local window, and use it as the local second coefficient.

[0013] Furthermore, the linear model includes: the local denoised image of the local window = the guiding image of the local window × the local first coefficient + the local second coefficient.

[0014] Furthermore, the newly assigned variance includes: if the variance of the local window is less than (2 M If -1), then the variance remains unchanged; otherwise, the variance of the local window is assigned a value of (2). M -1); where M is the high-level cutoff bit width.

[0015] Furthermore, the first coefficient a(x,y) is calculated as follows: Where var_I(x,y) is the variance of the newly assigned value; ε is the regularization coefficient, 0<ε<1; N is the bit width of the guiding image I(x,y).

[0016] On the other hand, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the method described above.

[0017] The beneficial effects of the embodiments of the present invention are as follows: This embodiment proposes a method for estimating image noise levels. Taking into account the spatially discrete arrangement of the RGBW channels, this embodiment utilizes the high responsivity of the W channel and uses a brightness threshold to estimate the noise level at each location, thereby improving the denoising effect.

[0018] This embodiment also fully considers the impact of W channel data saturation on the denoising results. Based on the estimated noise level at each location, in low-light conditions, the W channel with higher brightness is used as the guide image, which is equivalent to having a reference image with a high signal-to-noise ratio. In high-light conditions, the W channel is saturated, and if this is used as the guide image, it will lead to blurred edges and fail to achieve a good denoising effect. Therefore, (R+G+B) / 3 is used as the guide image.

[0019] In this embodiment, a guide image corresponding to each pixel position is selected based on the estimated noise level, so that each pixel position is denoised with texture information of reasonable illumination, thereby improving the denoising effect.

[0020] This embodiment proposes an improved image denoising method based on the idea of ​​guided filtering.

[0021] In the image denoising method proposed in this embodiment, the local linear relationship model can ensure that when there is an edge on the guide image of the local window, the local denoised image of the local window still retains the edge unchanged, thus ensuring the accuracy of the final denoising result.

[0022] According to the calculation formula of the first coefficient a(x,y), in this embodiment, the smoothness of the guide image within the local window is evaluated by the variance of the local window. That is, the smaller the variance of the local window, the smoother the guide image of the local window is. At this time, the local denoised image of the local window is mainly determined by the local second coefficient. According to the calculation formula of the second coefficient b(x,y), the smaller the first coefficient a(x,y), the more the second coefficient b(x,y) tends to the pixel mean of the local window.

[0023] If the variance of the local window tends to 1, it indicates that there is a drastic change in the guiding image of the local window. The local denoised image of the local window is mainly determined by the guiding image. As for the image edges, the details in the guiding image are largely preserved and are not overly smoothed, reducing the pixel data loss caused by linear model fitting.

[0024] In this embodiment, the pixel data of the guide image is confirmed based on the judgment relationship between the brightness W channel and the brightness threshold. The TDI superposition result is used as the input image for denoising. Based on the pixel data of the guide image, the input image is denoised to obtain the final denoised image.

[0025] This embodiment proposes an image denoising method that uses the average pixel values ​​of the R, G, and B channels in the guide image as the input image for denoising. This method is suitable for hardware implementation and improves the computational efficiency of image denoising.

[0026] This embodiment proposes an image denoising method that omits division operations to adapt to hardware implementation. In the variance var_I(x,y) calculation formula, local sums of squares and local sums are directly used to align the variance calculation formula via product. Finally, the corresponding division operation is achieved by right-shifting the pixel data.

[0027] To facilitate hardware implementation, this embodiment reduces the quantization level of variance var_I(x,y) to reduce the space of the subsequent lookup table. By using the mapping method of the lookup table, the introduction of division operations is avoided, thereby reducing the resource consumption inside the FPGA.

[0028] This embodiment employs an M-bit high-order truncation method for the variance var_I(x,y) to reduce the quantization levels of the variance. Furthermore, based on the aforementioned high-order truncation, this embodiment also proposes quantizing the variance of the current local window to construct a low-bit-width lookup table. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the image noise level estimation method in this invention; Figure 2 This is a schematic diagram illustrating the principle of TDI delay in this invention; Figure 3 This is a schematic diagram of pixel scanning in each row when the rate ratio in the TDI delay is 1:1 in this invention; Figure 4 This is a schematic diagram of pixel scanning in each row when the rate ratio in the TDI delay is 1:2 in this invention; Figure 5 This is a schematic diagram of the method flow of Embodiment 1 of the present invention; Figure 6 This is a flowchart illustrating the image denoising method in Embodiment 1 of the present invention; Figure 7 This is a flowchart illustrating the image denoising method in Embodiment 2 of the present invention; Figure 8 This is a data graph of the first coefficient 'a' quantized when different variance maximum values ​​are truncated in this invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0031] like Figure 1 As shown, this application proposes a method for estimating image noise levels, including: Acquire RGBW data output from a multi-line color sensor.

[0032] In this application, the multi-line color sensor has at least one R channel, G channel, B channel, and W channel, and the number of R channels, G channels, B channels, and W channels is the same.

[0033] That is, the image format output by the multi-line color sensor is RGBW×n, where n is the number of channels of any color pixel in the multi-line color sensor, and n≥1.

[0034] Based on the delay of the multi-line color sensor, the data of different rows are delayed and aligned; the pixel data of the R, G, B and W rows corresponding to the same position are accumulated to obtain the TDI superposition result; where the TDI delay is an integer.

[0035] If there are RGBW pixel data from multiple channels, first superimpose pixel data of the same color, and then superimpose pixel data of different colors.

[0036] Alternatively, based on the arrangement order of the pixel channels, the R, G, B, and W channels, which are arranged first, are used as a group of processing units for superposition, and the superposition of the RGBW data of each subsequent group is carried out in turn. Finally, each group of pixel data is superimposed.

[0037] This embodiment takes into account the characteristics of RGBW data and determines the relationship between the W row pixel value at each position and the brightness threshold to estimate the noise level at the current position.

[0038] Based on a brightness threshold, the noise level at the current location is estimated, and a guidance image for the current location is output, including: If the pixel value of row W at the current position is less than the brightness threshold, then the noise level at the current position is low illumination, and the pixel value of row W at the current position is output as the guide image; otherwise, the noise level at the current position is high illumination, and the average value of the pixel data of rows R, G, and B at the same position is used as the guide image.

[0039] TDI latency refers to the delay time between the output of one row of pixel data and the output of the next row of pixel data.

[0040] The TDI integration process is as follows: In a multi-line color sensor, during continuous exposure, the grayscale value P of the M-th row pixel after the t-th exposure is obtained along the direction of motion of the object being photographed. t Align it with the grayscale value obtained after the (t+α+1)th exposure of the (M+1)th row pixel.

[0041] Where α is the number of pixels between the RGBW components, and α is an integer.

[0042] In a multi-line color sensor, the pixels in the R, G, B, and W rows are aligned in the manner described above, and the R, G, B, and W components of one row of pixels are output.

[0043] When R, G, B, and W are arranged in order, the R row, G row, B row, and W row are set sequentially along the direction of motion of the object being photographed; the delay of the R row pixels relative to the G row pixels, the delay of the G row pixels relative to the B row pixels, and the delay of the B row pixels relative to the W row pixels are D, where D is an integer.

[0044] In this application, the TDI delay is D, where D is an integer.

[0045] like Figure 2 As shown, during four consecutive scan times, a portion A of the object being photographed is captured by pixel0, pixel1, pixel2, and pixel3 respectively. However, at the same time, such as when pixel3 captures A, pixels0, pixel1, pixel2, and pixel3 output the pixel data of four parts, D, C, B, and A respectively. Direct integration at this time is obviously incorrect.

[0046] The usual approach is to delay the pixel data acquired by pixel0 by one exposure cycle and then add it to the pixel data acquired by pixel1. This is because after pixel0 captures A, pixel1 will capture A in the next exposure. Therefore, delaying the pixel data acquired by pixel0 by one exposure cycle and then adding it to the pixel data acquired by pixel1 is exactly how the pixel signal corresponding to A is accumulated. In this case, the TDI delay amount D is said to be 1.

[0047] The relationship between the line frequency of the multi-line color sensor output image and the motion rate of the photographed object directly affects the delay superposition relationship of the RGBW components. In typical user scenarios, the line frequency and the object speed satisfy a 1:1 relationship (i.e., the speed ratio is 1:1). At this time, depending on the arrangement of the RGBW components in the multi-line color sensor, the typical TDI delay D is 1 or 2, corresponding to the cases where the vertical position of the RGBW pixels is arranged with a 0 pixel interval and a 1 pixel interval, respectively.

[0048] like Figure 3 As shown, when the RGBW components of a multi-line color sensor have a 1-pixel interval (i.e., the number of pixels between the RGBW components is 1), the rate ratio is 1:1, and the TDI delay is 2. That is, the R component can be aligned with the G component by delaying 2 exposure cycles, and the G component can be aligned with the B component by delaying 2 exposure cycles, and the B component can be aligned with the W component by delaying 2 exposure cycles. After alignment, the RGBW components simultaneously output the marker point at the same position of the photographed object, so there is no chromatic aberration.

[0049] like Figure 4 As shown, when the RGBW components of a multi-line color sensor have a 1-pixel interval (i.e., the number of pixels between the RGBW components is 1), the rate ratio is 1:2, and the TDI delay is 1. That is, the R component can be aligned with the G component by delaying it by 1 exposure cycle, and the G component can be aligned with the B component by delaying it by 1 exposure cycle. The B component can be aligned with the W component by delaying it by 1 exposure cycle. After alignment, the RGBW components simultaneously output the marker point at the same position of the photographed object, so there is no chromatic aberration.

[0050] Where TDI delay D = (number of pixels between RGBW components + 1) × rate ratio.

[0051] The rate ratio is the line frequency of the multi-line color sensor output image and the motion rate of the object being photographed.

[0052] As described above, this embodiment presents two cases (with different rate ratios) of TDI delay when the vertical positions of RGBW pixels are arranged with a 1-pixel interval. Preferably, the delay case where the vertical positions of RGBW pixels are arranged with a 0-pixel interval can also be set.

[0053] As described above, this embodiment provides two cases where the ratio of the output line frequency of the multi-line color sensor to the object speed is 1:1 and 1:2; preferably, other ratios can also be set to meet the needs of practical applications.

[0054] In this embodiment, the TDI delay amount D is an integer, meaning that the alignment of RGBW components can be achieved by adjusting the exposure period between different channels without any chromatic aberration.

[0055] To address the spatially discrete arrangement of the RGBW channels in the TDI overlay results, the high responsivity of the W channel is utilized to denoise the RGBW data, improving the denoising effect. However, when using the W channel pixel data as the basis for denoising, the saturation issue of the W channel data must be fully considered. If the W channel pixel values ​​are saturated, it is easy to determine that there is no noise at that location, leading to noise-induced edge blurring. In this case, the average pixel values ​​of the R, G, and B channels are used as a guide image, whose noise level is higher than that of the original RGB channel pixel values, thus improving the denoising level.

[0056] This application employs guided filtering on the input image p i The principle of noise reduction is to apply the noise reduction process to a local window w. k In the process, the guiding image I and the output image q (denoised image) have a local linear relationship, that is, q i =a k I i +b k , i∈w k .

[0057] In the formula, k represents the index of the local window, i represents the pixel index within the local window, and a k b k These are the coefficients of the linear transformation.

[0058] The linear relationship between the output image and the guide image within the local window is used to ensure that the output image retains the edge when there is an edge in the guide image; at the same time, the output image is as similar to the input image as possible to reduce information loss caused by filtering and noise reduction. Example 1

[0059] like Figure 5 As shown, this embodiment uses the above estimation method to obtain the TDI overlay result and the guide image I. The TDI overlay result is used as the input image for denoising processing to obtain the final denoised image.

[0060] like Figure 6 As shown, this embodiment uses the TDI overlay result as the input image p for denoising and proposes a denoising method, as detailed below: Set up a local window and calculate the variance of the guide image and the covariance between the guide image and the input image within the local window.

[0061] Preferably, in this embodiment, the size of the local window is fixed at 3×3.

[0062] In this embodiment, the guide image can be the average pixel value of the R, G, and B channels, or it can be the pixel value of the C channel.

[0063] Within a local window, based on the color channel corresponding to the guiding image at any pixel position, when calculating the first and second coefficients at that pixel position, the color channel corresponding to each pixel value in the neighborhood of that pixel position is the same as the color channel corresponding to the guiding image, ensuring the consistency of the calculation logic.

[0064] If the guiding image corresponding to any pixel position within the current local window is a pixel value of the C channel, then during the calculation of the first coefficient and the second coefficient corresponding to that pixel position, the neighborhood of that pixel is the pixel value of the C channel, in order to ensure the consistency of the calculation logic.

[0065] Similarly, if the guiding image corresponding to any pixel position within the current local window is the average pixel value of the R, G, and B channels, then during the calculation of the first and second coefficients corresponding to that pixel position, the neighborhood of that pixel is the average pixel value of the R, G, and B channels, to ensure the consistency of the calculation logic.

[0066] Based on the formula for calculating variance, this embodiment calculates the square of the mean pixel value of the guide image within the local window and the mean of the squared pixel values ​​of the guide image, respectively, to calculate the variance of the guide image.

[0067] Preferably, mean filtering can be used to calculate the mean pixel value of the guide image within the local window, mean_I, and the mean square pixel value of the guide image, mean_II.

[0068] The variance of the guided image within the local window is var_I = mean_II - mean_I × mean_I.

[0069] Based on the formula for calculating covariance, this embodiment calculates the mean pixel value of the input image (mean_p) and the mean pixel product of the guide image and the input image (mean_Ip) to calculate the covariance between the guide image and the input image.

[0070] The covariance between the guide image and the input image within the local window is cov_Ip = mean_Ip - mean_I × mean_p.

[0071] Calculate the first coefficient 'a' of the local window based on the variance and covariance within the local window.

[0072] The second coefficient b of the current window is calculated based on the first coefficient at the current position within the window, the average pixel value of the guiding image, and the average pixel value of the input image.

[0073] The first and second coefficients of each local window can be calculated according to the above process. However, each pixel is contained in multiple windows. For each pixel, multiple first and second coefficients can be calculated. Then, the mean of the multiple first coefficients mean_a and the mean of the multiple second coefficients mean_b of each pixel in any local window are used as the linear coefficients for calculating the denoised image q in that local window.

[0074] Using the local mean of the first and second coefficients, calculate the denoised image q = mean_a × I + mean_b of the input image within the local window. Example 2

[0075] like Figure 7 As shown, to facilitate hardware implementation of the noise reduction process, this application proposes an image denoising method, including: Based on the estimation method described above, the obtained boot image is used. The hardware device in this embodiment receives and buffers the boot image line by line.

[0076] The guide image is the average of the pixel data of rows R, G, and B corresponding to the same position, and is used as the input image for denoising.

[0077] In this embodiment, a local window is set up. According to the arrival order of the guide images, the denoising results corresponding to the guide images in the local window are calculated sequentially to obtain a complete denoised image.

[0078] Acquire and cache the guide images line by line; according to the arrival order of the guide images, calculate the local first coefficient and local second coefficient of each local window in sequence, and calculate the local denoised image of the local window based on the linear model; complete the denoising process of the guide images in each local window in sequence to obtain the denoised image.

[0079] Local first coefficients include: The square of each pixel in the guide image is calculated row by row. The sum of the squares of the pixels in each local window is calculated sequentially and used as the local sum of squares for that local window. The sum of the squares of each pixel in that local window is also calculated and used as the local sum. The variance of the local window is the difference between the local sum of squares and the square of the local sum. The variance of the local window is truncated at the high bit to obtain the newly assigned variance, which is used to construct a lookup table for the first coefficient corresponding to each pixel. Calculate the sum of the first coefficients within this local window, and use it as the local first coefficient.

[0080] Based on the local sum and the first coefficient of the current local window, calculate the second coefficient corresponding to each pixel; calculate the sum of the second coefficients within the local window, and use it as the local second coefficient.

[0081] The linear model includes: the local denoised image of the local window = the guiding image of the local window × the local first coefficient + the local second coefficient.

[0082] In this embodiment, the maximum value of the guiding image I(x,y) is 2. N -1, x ranges from 0 to H-1, y ranges from 0 to W-1; H is the number of rows, W is the number of columns, and N is the data width (bit).

[0083] Starting from the first row of I(x,y), calculate the square of each pixel II(x,y) in the order they arrive, as follows:

[0084] The bit width of II(x,y) is 2N bits.

[0085] Preferably, in this embodiment, the size of the local window is fixed at 3×3.

[0086] Starting from the 3rd row of the guide image I(x,y), within a local window, calculate the local summation sum_I(x,y) of the guide image I(x,y). The calculation formula is as follows:

[0087] In the formula, i represents the row coordinate of the image in the guide image, and j represents the ordinate of the image in the guide image. The bit width of sum_I(x,y) is (N+4) bits.

[0088] Starting from the third row of the guiding image I(x,y), within a local window, calculate the local sum of squares of the squares II(x,y) of each pixel, denoted as sum_II(x,y), using the following formula:

[0089] The bit width of sum_II(x,y) is (2N+4) bits.

[0090] In this embodiment, the variance var_I(x,y) is calculated using the local summation and the local sum of squares.

[0091] In the variance var_I(x,y) calculation formula, directly using the local sum of squares will amplify the variance calculation result by 9×9=81 times. Furthermore, for ease of hardware implementation, this embodiment does not include division for the mean calculation of variance var_I(x,y). Therefore, to align with the var_I(x,y) calculation formula, sum_II(x,y) needs to be multiplied by 9 (the local window size in this embodiment is 3×3). The variance calculation formula is as follows:

[0092] In the formula, ">>16" means right shift by 16 bits, that is, to achieve a division operation of 1 / 81 by right shifting by 16 bits.

[0093] To facilitate hardware implementation, this embodiment reduces the quantization level of variance var_I(x,y) to decrease the space required for subsequent lookup tables. Furthermore, this embodiment reduces the quantization level of variance by truncating the high-order bits of variance var_I(x,y) by M bits.

[0094] If the variance of the local window is less than (2 M If -1), then the variance remains unchanged; otherwise, the variance of the local window is assigned a value of (2). M -1); where M is the high-level cutoff bit width.

[0095] The variance calculation result is greater than 2. M The value of 2 is directly assigned to var_I(x,y) which is -1. M -1, as follows:

[0096] High-bit truncation essentially compensates for the low-variance pixel quantization precision by reducing the quantization precision of high-variance pixels. For example... Figure 8 As shown, this embodiment experimentally verified the effect of different high-order truncations (different values ​​of M) on the first coefficient, to show that the impact of the high-order truncation of variance proposed in this embodiment on the calculation accuracy is within a controllable range and will not produce quantization error.

[0097] The experimental data provided in this embodiment uses an 8-bit image as an example. The horizontal axis represents the lookup table index, and the vertical axis represents the first coefficient a.

[0098] When M=14 bits, the maximum variance corresponds to an 8-bit image, meaning no high-bit truncation is performed. Since the mapping relationship from variance to coefficient a in the lookup table is non-linear, when the variance is small (lookup table index < 10), the range of the first coefficient a is from 0 to 0.9. This means that the actual quantization bit width of coefficient a with values ​​from 0 to 0.9 is too low, introducing quantization error. For coefficient a with values ​​from 0.9 to 1.0, there is obviously overquantization.

[0099] When M=12 bits, the trend of the calculated first coefficient a is consistent with that when M=14 bits, and the first coefficient a in the final flattening stage is very close to 1.0. Therefore, high-bit truncation will not cause quantization error, while taking into account both low-variance and high-variance pixels.

[0100] When M=10 bits, the trend of the calculated first coefficient a is consistent with that when M=14 bits, but the maximum value of the first coefficient a is only 0.94. This will cause significant errors in the calculation of floating-point numbers for the edges of drastic changes in the image.

[0101] Based on the experimental data above, it can be seen that when M is reduced to 12 bits or even 10 bits, the quantization bit width of the coefficient a corresponding to smaller variance is significantly improved. Considering that the actual acquired image usually has a small proportion of pixels with very large local variance, it is reasonable to improve the quantization bit width of smaller variance by high-bit truncation.

[0102] In this embodiment, a lookup table is used to map var_I(x,y) to the first coefficient a(x,y), i.e., a(x,y) = LUT(var_I(x,y)). This lookup table mapping method avoids introducing division operations and reduces resource consumption within the FPGA.

[0103] Preferably, to further reduce the quantization levels of variance, based on the above-mentioned high-bit truncation, this embodiment also proposes to quantize the bit width of the newly assigned variance to the lookup table quantization level L, so as to construct a low-bit-width lookup table with a lookup table size of 2. L .

[0104] The quantization of var_I(x,y) to L bits is calculated as follows:

[0105] In the formula, L represents the lookup table quantization level, and in this embodiment, L=N. This means right-shifting the newly assigned variance by (ML) bits to achieve bit-width quantization of the newly assigned variance.

[0106] The formula for calculating the first coefficient a(x,y) in the table is as follows:

[0107] In the formula, ε is a regularization parameter, which takes a value of 0 to 1 and represents the smoothness of the image.

[0108] The denominator of the first coefficient a(x,y) calculation formula is set to ε to prevent the calculation result of a(x,y) from being too large. Since it is impossible for FPGA implementation to normalize the pixels to floating-point numbers, considering the integer values ​​of image bit width conversion and the square operation of variance, the calculation formula is set to ε×(2N-1)×(2N-1).

[0109] In this embodiment, the lookup table size is fixed at 2. L When a(x,y) is used in floating-point operations, the value of a(x,y) is between 0 and 1. When constructing a lookup table, it needs to be quantized to Q bits to ensure data precision, that is, the bit width of a(x,y) is Q bits.

[0110] With a local window size of 3×3, calculate the local first coefficient sum_a(x,y) of a(x,y) using the following formula:

[0111] Taking into account image bit width conversion, the formula for calculating the second coefficient is:

[0112] In the formula, ">>16" means right shift by 16 bits, that is, the division operation of 1 / 9 is achieved by right shifting by 16 bits; L is the quantization level of the lookup table, and in this embodiment L=N; sum_I(x,y) is the local sum of the local window.

[0113] With a local window size of 3×3, calculate the local second coefficient sum_b(x,y) of b(x,y) using the following formula:

[0114] Based on a linear model, the local denoised image of the local window is calculated; the linear model includes: local denoised image of the local window = guiding image of the local window × local first coefficient + local second coefficient.

[0115] The local linear relationship model proposed in this embodiment can ensure that when there is an edge on the guide image of the local window, the local denoised image of the local window still retains the edge unchanged.

[0116] According to the calculation formula of the first coefficient a(x,y), in this embodiment, the smoothness of the guide image within the local window is evaluated by the variance of the local window. That is, the smaller the variance of the local window, the smoother the guide image of the local window is. At this time, the local denoised image of the local window is mainly determined by the local second coefficient. According to the calculation formula of the second coefficient b(x,y), the smaller the first coefficient a(x,y), the more the second coefficient b(x,y) tends to the pixel mean of the local window.

[0117] If the variance of the local window tends to 1, it indicates that there is a drastic change in the guiding image of the local window. The local denoised image of the local window is mainly determined by the guiding image. As for the image edges, the details in the guiding image are largely preserved and are not overly smoothed, reducing the pixel data loss caused by linear model fitting.

[0118] Considering the image bit width conversion, the formula for calculating the local denoised image of this local window is:

[0119] In the formula, That is, the division operation of 1 / 9 is achieved by right shifting by 16 bits; since the bit width of a(x,y) is Q bits and the bit width of b(x,y) is (N+Q+4) bits, right shifting by Q bits means normalizing the first coefficient a(x,y) and the second coefficient b(x,y), so it is merged into right shifting by (Q+16) bits.

[0120] In this embodiment, according to the above formula for calculating q(x,y), the value is first right-shifted by (Q+15) bits, and then rounded. The calculation method is as follows:

[0121] The maximum bit width of the denoised image q(x,y) is (N+Q+16) bits, and the final bit width is N bits, which is consistent with the bit width (N bits) of the guiding image I(x,y).

[0122] Based on the same inventive concept, this application also proposes a computer-readable storage medium including a computer program that, when executed by a processor, implements the method described above.

[0123] This application estimates the noise level at each pixel location based on the characteristics of RGBW data to identify the guiding image for each pixel location. Building upon the estimation method, this application uses the TDI overlay result as the input image for denoising based on the guiding image to obtain the corresponding denoised image. Based on the characteristics of RGBW data and considering the impact of guiding image selection on denoising performance under different illumination conditions, this application determines the guiding image corresponding to each pixel, ensuring that each pixel receives texture information with appropriate illumination for denoising, thus improving the denoising effect. Furthermore, this application proposes a corresponding hardware implementation to improve computational efficiency.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating image noise level, characterized in that, include: Acquire RGBW data output from the multi-line color sensor; Based on the TDI delay of the multi-line color sensor, data from different rows are aligned with the delay. Accumulate the pixel data of rows R, G, B, and W corresponding to the same position to obtain the TDI overlay result; where the TDI delay is an integer. If the pixel value of row W at the current position is less than the brightness threshold, then the noise level at the current position is low illumination, and the pixel value of row W at the current position is output as the guide image; otherwise, the noise level at the current position is high illumination, and the average value of the pixel data of rows R, G, and B at the same position is used as the guide image.

2. An image denoising method, characterized in that, include: Using the estimation method described in claim 1, TDI overlay results and a guide image are obtained; the TDI overlay results are used as the input image for denoising. Set up a local window and calculate the variance of the guide image within the local window and the covariance between the guide image and the input image; Calculate the first coefficient of the local window based on the variance and covariance within the local window; calculate the second coefficient of the current window based on the first coefficient at the current position within the window, the mean pixel value of the guiding image, and the mean pixel value of the input image. The denoised image of the input image within the local window is calculated using the local mean of the first and second coefficients.

3. The image denoising method according to claim 2, characterized in that, Also includes: Within a local window, based on the color channel corresponding to the guiding image at any pixel position, the color channel corresponding to each pixel value in the neighborhood of that pixel position is the same as the color channel corresponding to the guiding image when calculating the first coefficient and the second coefficient at that pixel position.

4. An image denoising method, characterized in that, include: The estimation method described in claim 1 is used to acquire and cache the guide image line by line; The guide image is the mean of the pixel data of the R, G, and B rows corresponding to the same position, and the guide image is used as the input image for denoising. Based on the arrival order of the guide images, the local first coefficient and local second coefficient of each local window are calculated sequentially. Based on the linear model, the local denoised image of the local window is calculated. The denoising process of the guide images in each local window is completed sequentially to obtain the denoised image.

5. The image denoising method according to claim 4, characterized in that, Local first coefficients include: The square of each pixel in the guide image is calculated row by row. The sum of the squares of the pixels in each local window is calculated sequentially and used as the local sum of squares for that local window. The sum of the squares of each pixel in that local window is also calculated and used as the local sum. The variance of the local window is the difference between the local sum of squares and the square of the local sum. The variance of the local window is truncated at the high bit to obtain the newly assigned variance, which is used to construct a lookup table for the first coefficient corresponding to each pixel. Calculate the sum of the first coefficients within this local window, and use it as the local first coefficient.

6. The image denoising method according to claim 5, characterized in that, Local second coefficients include: Based on the local sum and the first coefficient of the current local window, calculate the second coefficient corresponding to each pixel; calculate the sum of the second coefficients within the local window, and use it as the local second coefficient.

7. The image denoising method according to claim 4, characterized in that, The linear model includes: the local denoised image of the local window = the guiding image of the local window × the local first coefficient + the local second coefficient.

8. The image denoising method according to claim 5, characterized in that, The newly assigned variance includes: if the variance of the local window is less than (2 M If -1), then the variance remains unchanged; otherwise, the variance of the local window is assigned a value of (2). M -1); where M is the high-level cutoff bit width.

9. The image denoising method according to claim 5, characterized in that, The first coefficient a(x,y) is calculated as follows: ; Where var_I(x,y) is the variance of the newly assigned value; ε is the regularization coefficient, 0<ε<1; N is the bit width of the guiding image I(x,y).

10. A computer-readable storage medium comprising a computer program that, when executed by a processor, implements the image noise level estimation method of claim 1, and / or the image denoising method of any one of claims 2-9.

Images