Method and image processing system for reducing noise in digital images

Abstract

The present application relates to a method and an image processing system for reducing noise in a digital image. Embodiments of the present invention disclose a system and a method for reducing noise in a digital image. The method includes calculating a spatial weight of each pixel s in a current frame, wherein the spatial weight is calculated based on the sum of diffusion values ​​associated with the pixel s; and selectively filtering the intensity I of the pixel s spatially using the spatial weight.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to image processing, and more particularly, to noise reduction and contrast enhancement of images. Background Art

[0002] Figure Figure 1 A high-level block diagram of a camera 10 according to the prior art is shown. As can be seen, the camera 10 includes a lens 12 for focusing light onto an image sensor 14. The image sensor 14 may be a charge coupled device (CCD) or an active pixel (CMOS) sensor that is capable of converting incident light into pixels that are fed into a downstream image signal processing pipeline 16. Typically, the signal processing pipeline performs operations such as demosaicing, noise reduction, auto exposure, auto focus, auto white balance, and image sharpening on the pixels before output. Summary of the invention

[0003] According to a first aspect of the present invention, there is provided a method for reducing noise in a digital image, comprising:

[0004] calculating a spatial weight for each pixel s in the current frame, wherein the spatial weight is calculated based on a sum of diffusion values ​​associated with the pixel s; and

[0005] The intensity I of pixel s is selectively spatially filtered using the spatial weights.

[0006] According to a second aspect of the present invention, a system for implementing the above method is provided.

[0007] Other aspects of the invention will become apparent from the following written description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A high level block diagram of a prior art camera is shown.

[0009] Figure 2 A high-level block diagram of a camera with a modified image processing pipeline for 3D noise reduction and contrast enhancement according to one embodiment of the present invention is shown.

[0010] Figure 3 A flow chart showing operations performed to spatially filter pixels according to one embodiment of the present invention.

[0011] Figure 4 A schematic diagram of the current and previous frames is shown.

[0012] Figure 5 A representation of the Laplacian operator for edge detection according to one embodiment of the present invention is shown.

[0013] Figure 6 A representation of the Sobel operator for edge detection according to one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0014] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention.

[0015] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not of other embodiments.

[0016] In addition, although for the purpose of illustration, the following description contains many details, it will be understood by any person skilled in the art that many changes and / or modifications to the details are within the scope of the present invention. Similarly, although many features of the present invention are described according to each other or in combination with each other, it will be understood by those skilled in the art that many of these features can be provided independently of other features. Therefore, this description of the present invention is proposed without losing the generality of the present invention and without imposing limitations on the present invention.

[0017] Camera pipeline with 3D noise reduction and contrast enhancement

[0018] According to one embodiment of the present invention, and with reference to the accompanying drawings Figure 2 The camera pipeline includes a lens 12 and an image sensor 14, which are similar to those already described with respect to Figure 1 1. The image processing pipeline 22 is similar to the components described in the prior art camera pipeline 10 of FIG. 1. Therefore, the same reference numerals are used to represent these components. However, unlike the prior art camera pipeline 10, the output of the image sensor 14 is fed to a modified image processing pipeline 22, which includes a 3D noise reduction block 24 and a contrast enhancement block 26.

[0019] As will be described, 3D noise reduction block 24 performs spatial filtering in two dimensions and temporal filtering in one dimension, and thus the noise reduction technique performed by block 24 is referred to herein as 3D noise reduction.

[0020] 2D spatial filtering

[0021] refer to Figure 3In one embodiment, the 3D noise reduction block 24 may include a spatial filtering block 30 that is configured to receive the input frame 28 and, thereby, apply a spatial filtering technique on a frame-by-frame basis, as will now be described. The technique begins by receiving a current frame at block 32 and immediately saving a copy of the current frame to a temporal buffer at block 34. The saved copy of the current frame is used in the temporal filtering block that will be described later. Next, for each pixel in the current frame, the difference between the current pixel and its neighboring pixels is calculated. At block 36, the difference between the current pixel and its neighboring pixels is calculated. To illustrate how the difference is calculated, consider the accompanying drawings. Figure 4 46 is depicted in FIG. 4. For illustration purposes, the current frame shows only nine pixels P(x,y). In practice, one of ordinary skill in the art will appreciate that the current frame may include many more pixels based on the resolution of the image. Consider pixel P(1,1) located at the center of the frame. Arrows starting from pixel P(1,1) indicate each of its neighboring pixels. In this case, calculating the difference between the current pixel and its neighboring pixels simply means calculating the following differences:

[0022] P(1,1)-P(0,0)

[0023] P(1,1)-P(1,0)

[0024] P(1,1)-P(2,0)

[0025] P(1,1)-P(0,1)

[0026] P(1,1)-P(2,1)

[0027] P(1,1)-P(0,3)

[0028] P(1,1)-P(1,3)

[0029] P(1,1)-P(2,3).

[0030] If the current pixel has no neighbors, such as the case of pixel P(0,0), pixel padding may be performed. In one embodiment, the particular padding technique used involves simply taking the value of the current pixel as the value of the padding pixel.

[0031] Again, refer to the attached drawings Figure 3 , at block 38, the sum of the differences calculated in the previous step is calculated. The sum of the calculated differences is used at block 40 to selectively apply spatial filtering to the current pixel as follows.

[0032] If the sum is greater than the threshold, the current pixel is not changed, in other words, no spatial filtering is applied to the current pixel. However, if the sum is less than the threshold, the current pixel is replaced with a value corresponding to the difference sum calculated at block 38.

[0033] In the above technique, the spatial filter block 30 calculates a new pixel value in the spatial domain for each pixel "s" in the frame time t, wherein the neighboring pixels of the pixel are included in the sum of the diffusion values. In the described example, all neighboring pixels of the current pixel (A, B, C, D, E, F and G, see frame 40) are used in the calculation. In some embodiments, not all neighboring pixels of a pixel are used in the calculation. For example, only four pixels B, D, E and F may be used.

[0034] To implement filtering, first filter the current pixel with the surrounding pixels in the same frame, as follows:

[0035] T_weight_space = ∑ψ(I(t,p)–I(t,s));

[0036] Where I is the pixel intensity; and (t,s) is the current pixel in the current frame. Therefore, the above calculation adds up the "differences" on all "p", that is, the surrounding pixels in the same frame above. "s" is the center pixel P(1,1), and "p" is the neighboring pixel;

[0037] The spatially filtered samples are output according to the following equation:

[0038] spatial_filter_sample=I(t,s)+λ*T_weight_space;

[0039] Among them, "spatial_filtered_sample" is the output of spatial filtering of the current pixel.

[0040] In some embodiments, λ may be 0.603.

[0041] As described above, spatial filtering is selective because the current pixel value changes only in certain circumstances. In one embodiment, spatial filtering of the current pixel is performed only when the sum of the differences is below a threshold rho, as shown below.

[0042] For a constant "rho" proportional to the noise variance:

[0043] ψ(x)=(1–(x / rho)^2)^2; if |x|<=rho;

[0044] ψ(x)=0;if |x|>rho.

[0045] In one embodiment, rho can be set to 3 or 4. The above calculations are performed separately for each channel in the color space. For example, for the RGB color space, the above calculations are performed separately for each of the R, G, and B pixels.

[0046] According to the above spatial filtering, if the sum of the differences is greater than rho, the current pixel remains unchanged (ψ(x) = 0), otherwise, adaptive diffusion is applied to the current pixel according to the formula ψ(x) = (1–(x / rho)^2)^2.

[0047] Time Filtering

[0048] Again, refer to the attached drawings Figure 3 , temporal filtering is applied at block 42. In one embodiment, for temporal filtering, each pixel in the new current frame received at block 32 is compared with the corresponding co-located pixel from the previous frame (t-1) stored in the temporal buffer at block 34. In particular, I(t,s), the intensity of each pixel of the current frame is compared with the intensity of the co-located pixel I(t-1,s) at time (t-1) of the previous frame. For example, referring to Figure 4 , the current frame is denoted by reference numeral 46, and the previous frame is denoted by reference numeral 44; and the same co-located pixels have the same (x, y) coordinates. Temporal filtering can then be performed according to the following formula:

[0049] T_weight_time = ψ(I(t-1,s)–I(t,s)).

[0050] The output of temporal filtering is the temporal filtered pixel calculated according to the following formula:

[0051] Time_filter_sample=spatial_filter_sample+λ*T_weight_time.

[0052] This new pixel "s" and the same address past frame pixel "p" are used for the above-mentioned space-time calculation. The calculation results are stored in the past frame memory (time buffer). It is used as the input for subsequent spatial contrast enhancement, and if contrast enhancement is not required, it can be sent out as the output of the time-space filter.

[0053] Contrast Enhancement

[0054] In one embodiment, for each pixel s in the temporal buffer, the adaptive diffusion in the spatial domain is calculated as follows:

[0055] T_weight(s)=∑ψ(I(t,p)–I(t,s));

[0056] I(t+1,s)=I(t,s)+λ*T_weight(s); if |T_weight(s)|>0; this means the current sample is filtered; and

[0057] I(t+1,s)=I(t,s)+k*H(s); if |T_weight(s)| is equal to 0; this means that this sample is on the edge;

[0058] Among them, H(s) is the contrast enhancement function described below, weighted by the constant k.

[0059] Contrast enhancement function

[0060] In one embodiment, the contrast enhancement function can utilize edge detectors in the form of the Laplacian operator and the Sobel operator. The Laplacian operator is a second-order derivative operator and is therefore very sensitive to random noise. For this reason, it is best to modulate the output of the Laplacian operator with a more structured first-order Sobel operator. The Laplacian operator also provides a flag for contrast enhancement.

[0061] The Sobel is defined as the sum of the absolute values of the horizontal and vertical gradient operators:

[0062] Sobel = abs(Gx(s)) + abs(Gy(s));

[0063] In one embodiment, small Sobel values can be set to zero, and large Sobel values can be declared as follows:

[0064] If (Sobel < c1) Sobel = 0; where c1 is a constant; and

[0065] If (Sobel > c2) Sobel = c2; where c2 is a constant.

[0066] In some embodiments, c1 can be 32 and c2 can be 128.

[0067] The contrast enhancement value H(s) can be calculated as follows:

[0068] H(s) = Laplacian(s) * Sobel(s).

[0069] The above contrast enhancement technique is executed by the contrast notification block 26 shown in the accompanying drawings. Figure 1 as shown.

[0070] Figure 5 and Figure 6 shows representations of the Laplacian operator and the Sobel operator for edge detection, respectively, according to an embodiment of the present invention.

[0071] As those skilled in the art will understand, aspects of the present invention can be embodied as a system, a method, or a computer program product. Therefore, aspects of the present invention can take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as "circuits", "modules", or "systems". In addition, aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media, having computer-readable program code thereon.

[0072] The title, background, brief description of the drawings, abstract, and drawings are hereby incorporated into the present disclosure and are provided as illustrative examples of the present disclosure, not as limiting descriptions. It should be understood when submitted that they will not be used to limit the scope or meaning of the claims. In addition, in the detailed description, it can be seen that the description provides illustrative examples and that various features are combined together in various implementations in order to simplify the present disclosure. The disclosed method should not be interpreted as reflecting the intention that the claimed subject matter requires more features than expressly stated in each claim. On the contrary, as reflected in the claims, the inventive subject matter lies in less than all the features of a single disclosed configuration or operation. The claims are thereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

[0073] The claims are not intended to be limited to the aspects described herein, but are to be accorded the full scope consistent with the language of the claims and including all legal equivalents. Nevertheless, no claim is intended to encompass subject matter that does not satisfy applicable patent law requirements, nor should it be interpreted in such a manner.

Claims

1. A method for reducing noise in a digital image, comprising: Generate a current frame through an image sensor; receiving the current frame from the image sensor in an image processing pipeline; Calculating a spatial weight of each pixel s in the current frame in a spatial filtering block of the image processing pipeline, wherein the spatial weight is calculated based on a sum of diffusion values ​​associated with the pixel s; As a step within the image processing pipeline, selectively spatially filtering the intensity I of a pixel s using the spatial weight; and The diffusion value is calculated according to the following formula: T_weight_space = ∑ψ(I(t,p)–I(t,s)); Where I represents pixel intensity; (t, s) specifies each pixel s in the current frame; t is the frame time, p is the neighboring pixel adjacent to pixel s; and wherein selectively spatially filtering the intensity I of a pixel s comprises selecting a constant value rho such that: ψ(x)=(1–(x / rho)^2)^2; if |x|<=rho; ψ(x)=0;if |x|>rho.

2. The method according to claim 1, wherein: The diffusion value includes the intensity difference between pixel s and neighboring pixels of pixel s in the spatial domain.

3. The method according to claim 1, wherein: The intensity I of pixel s is spatially filtered according to the following formula: spatial_filter_sample=I(t,s)+λ*T_weight_space; Here, λ is a constant.

4. The method according to claim 1, wherein: rho is chosen to be proportional to the noise variance in the current frame.

5. The method of claim 4, further comprising selectively temporally filtering the intensity based on a temporal weight calculated between the intensity I of the pixel s in the current frame and the value of the pixel s from the immediately previous frame.

6. The method according to claim 5, wherein: The time weight is calculated according to the following formula: T_weight_time = ψ(I(t-1,s)–I(t,s)); where I(t,s) is the intensity of pixel s in the current frame; and I(t-1,s) is the intensity of pixel s from the immediately previous frame.

7. The method of claim 6 further comprising outputting a temporally filtered pixel calculated according to the following formula: Time_filter_sample = space_filter_sample + λ*T_weight_time; in, λ is a constant.

8. The method of claim 7, further comprising enhancing the contrast in the current frame by applying a contrast enhancement function H(s) to each pixel s, wherein The values ​​change due to spatial filtering.

9. The method according to claim 8, wherein: The contrast enhancement function includes an edge detection algorithm.

10. An image processing system, comprising: lens; at least one image sensor coupled to the lens, wherein the at least one image sensor is configured to generate a current frame; and Image processing pipeline, including: A spatial filtering block configured to receive the current frame from the at least one image sensor: Calculating a spatial weight of each pixel s in the current frame, wherein the spatial weight is calculated based on a sum of diffusion values ​​associated with the pixel s; selectively spatially filtering the intensity I of pixel s using the spatial weight; and The diffusion value is calculated according to the following formula: T_weight_space = ∑ψ(I(t,p)–I(t,s)); Where I represents pixel intensity; (t, s) specifies each pixel s in the current frame; t is the frame time, p is the neighboring pixel adjacent to pixel s; and wherein selectively spatially filtering the intensity I of a pixel s comprises selecting a constant value rho such that: ψ(x)=(1–(x / rho)^2)^2; if |x|<=rho; ψ(x)=0;if |x|>rho.

11. The image processing system according to claim 10, wherein: The diffusion value includes the intensity difference between pixel s and neighboring pixels of pixel s in the spatial domain.

12. The image processing system according to claim 10, wherein: The intensity I of pixel s is spatially filtered according to the following formula: spatial_filter_sample=I(t,s)+λ*T_weight_space; Here, λ is a constant.

13. The image processing system according to claim 10, wherein: rho is chosen to be proportional to the noise variance in the current frame.

14. The image processing system according to claim 13, wherein: The method further comprises selectively filtering the intensity I of the pixel s in the current frame based on a temporal weight calculated between the intensity I and the value of the pixel s from the immediately previous frame.

15. The image processing system according to claim 14, wherein: The time weight is calculated according to the following formula: T_weight_time = ψ(I(t-1,s)–I(t,s)); where I(t,s) is the intensity of pixel s in the current frame; and I(t-1,s) is the intensity of pixel s from the immediately previous frame.

16. The image processing system of claim 15, further comprising outputting a temporally filtered pixel calculated according to the following formula: Time_filter_sample = space_filter_sample + λ*T_weight_time; in, λ is a constant.

17. The image processing system according to claim 10, further comprising contrast enhancement, wherein the contrast enhancement is configured to apply a contrast enhancement function H(s) to each pixel s, wherein The values ​​change due to spatial filtering.

Images