Method of improved nlm filter based on gradient edge detection

Abstract

The application provides a method of improved NLM filter based on gradient edge detection, which improves the perception ability of NLM filter to image texture and edge by combining with the gradient edge detection method, and improves the ability of the filter to retain more effective image information. According to the principle of NLM filter, the filtering strength is related to delta and h, so combining edge detection to set different filtering strengths in flat, texture and edge areas can improve the perception ability of the filter to mutation and achieve the edge preservation effect. The adaptive adjustment of noise variance according to the quantitative gradient calculation result can improve the perception ability of NLM filter to image texture and edge.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for improving NLM filters based on gradient edge detection. Background Technology

[0002] Classic spatial edge-preserving filters in image denoising include bilateral filtering (BF), guided filtering (GF), and non-local means (NLM) filtering. The task of edge-preserving filters is to separate noise from valid signals, suppress noise, and retain valid signals. NLM filters leverage the stronger spatial structural similarity between image textures and edges than noise, employing block matching to filter neighboring interfering pixels. Under the same noise intensity, NLM filters outperform bilateral filtering and guided filtering in preserving valid image signals, exhibiting stronger edge-preserving performance. However, in high-noise scenarios, where noise density and intensity are high, the accuracy of block matching decreases significantly, the filter's edge-preserving ability degrades, and valid image textures and edges may be lost or blurred.

[0003] Among existing technologies, the Canny edge detection method is a popular edge detection method, often applied in image segmentation, image enhancement, and recognition. It is a multi-stage computation method with strong robustness. The core steps of the Canny edge detection method include...

[0004] (1) Pre-noise reduction reduces noise interference to subsequent processing;

[0005] (2) Multi-directional gradient calculation;

[0006] (3) Non-maximum suppression filters redundant points;

[0007] (4) Double thresholds determine the final edge point.

[0008] Nonlocal mean filter (NLM) is a classic image denoising method. Based on the idea of ​​block matching, it calculates the similarity between neighboring pixels and the center pixel within the filtering window, converts this similarity into filtering weights, and finally outputs the weighted sum of all pixels within the window and the filtering weights.

[0009] The NLM filter has two windows, the search window w s and matching window w b The search window size is generally larger than the matching window size. The weighted sum of the pixels within the search window and their weights is the output of the center point. The matching window is used to calculate the similarity between neighboring blocks and the center block.

[0010] Both bilateral filters, used as local filters, detect abrupt changes (texture and edges) in an image by calculating weights that combine the positions of neighboring pixels with pixel similarity information, where pixel similarity is measured by individual pixel values. In contrast, the NLM filter removes positional information, using block matching instead of point matching. Block matching captures points with similar neighborhood structures and weights them accordingly. Compared to noise, the effective texture and edge spatial structure of an image have stronger similarity; therefore, block matching is more effective at filtering noise than point matching. Thus, the NLM filter is superior to the bilateral filter in preserving edges.

[0011] In scenarios with low noise levels, the NLM filter performs relatively well in block matching. However, as noise intensity increases, the block matching accuracy drops significantly due to noise interference, and the filter's ability to detect edges also decreases. At this point, effective textures and edges cannot be well preserved, and the output image tends to be blurry. Increasing the matching window size can improve block matching accuracy, but the improvement is not significant compared to the increased computational cost.

[0012] Furthermore, the terminology used in this art includes:

[0013] ISP: Image Signal Processing. Its main task is to process the raw signals captured by the image sensor into digital images of a specific pattern for subsequent display or other processing. It includes many processing modules to improve image quality, such as color conversion, white balance, gamma correction, depigmentation, and noise reduction.

[0014] Noise Reduction: Image noise reduction is an important part of the ISP processing flow. Since noise is inevitably generated during the imaging process, the effective information of the image will be contaminated by noise and affect the image quality. An appropriate noise reduction module can effectively separate noise and effective signal, reduce the noise component, and retain the effective signal as much as possible, thereby improving the image output quality.

[0015] Spatial domain filters: Filters are used to remove redundant signals, i.e., noise, from images. Filters can be divided into frequency domain and spatial domain. The frequency domain is convenient for processing signals in different frequency bands, while the spatial domain focuses on the pixel information and spatial structure information of the image.

[0016] Anisotropic filters: From a linear perspective, filters can be divided into isotropic and anisotropic. Isotropic filters are rotationally invariant and do not specifically process abrupt changes in image content (texture or edges). For example, Gaussian filtering's weights are only related to the position of neighboring pixels, regardless of whether the pixel is texture or an edge. Therefore, Gaussian filtering often results in overly blurry images. In contrast, anisotropic filters are sensitive to abrupt changes in the image, distinguishing the noise reduction intensity between texture edges and flat areas, thus preserving texture details well.

[0017] Edge detection: Extracting locations in an image where information (brightness or color) changes drastically, typically corresponding to object outlines or region boundaries. Combining edge information with subsequent processing is an important technique in image processing.

[0018] Image gradient: describes the degree of change in image information. Smaller changes are relatively flat, while more dramatic changes are generally seen in textures or edges. The direction of the image gradient describes the direction of change in image information. Summary of the Invention

[0019] To address the aforementioned issues, this application aims to improve the NLM filter's ability to perceive image texture and edges by combining gradient edge detection with a more effective method, thereby enhancing the filter's ability to retain more effective image information. As described above, the filtering strength is related to δ and h. Therefore, by combining edge detection, different filtering strengths are set for flat, textured, and edge regions. The filtering strength is adaptively achieved by the flatness Flat(p), which describes the degree of flatness. The flatter the surface, the slower W decays, resulting in a stronger filter. Conversely, in textured edge regions, increased unevenness leads to faster W decay and a weaker filter, preserving more original details. This method improves the filter's ability to perceive abrupt changes, achieving edge preservation.

[0020] Specifically, this invention provides a method for an improved NLM filter based on gradient edge detection, wherein the filter intensity is adaptively achieved by the flatness Flat(p), including:

[0021] S1 uses the gradient edge detection method to calculate the gradient values ​​of the image in four directions: horizontal, 45 degrees, vertical, and 135 degrees. A differential filter, assuming to be 3x3, is used and defined as follows:

[0022]

[0023] Setting all non-zero weights to 1 or -1 is equivalent to smoothing the input, which improves the noise resistance of the edge detection method.

[0024] S2, let the absolute values ​​of the gradients in the four directions be G_0, G_45, G_90, and G135, respectively. Then the gradient magnitude at pixel p is calculated using the following formula:

[0025] G(p)=max(G_0+G_90,G_45,G_135)

[0026] Set two gradient thresholds G_tl(p) and G_th(p);

[0027] G_tl(p) = func(in(p))

[0028] G_th(p) = G_tl(p) * scale

[0029] Where G_tl(p) represents the lower threshold of the gradient, which is a function of the pixel value or brightness value at p; G_th(p) is the upper threshold of the gradient, which is linearly related to G_tl(p); and scale is an empirical value, which takes values ​​[3,5].

[0030] G_tl(p) is defined as a function of pixel or brightness, so that the threshold is an adaptive threshold and has global characteristics;

[0031] S3, Next, the flat region and the edge region are determined by thresholding, and the gradient magnitude is quantized to a specified range to describe the flatness of the image;

[0032]

[0033] Where f_l1 and f_h1 are the lower and upper limits of quantization for flat regions, respectively, with f_l1 set to 0.5 and f_h1 set to 1.0; regions with gradient magnitude G(p) greater than G_th(p) are considered edge regions, and their flatness Flat(p) is set to 0; since a gradient greater than the upper threshold indicates that this is a valid texture, the flatness is minimized, so flatness = 0; f_l2 and f_h2 are the lower and upper limits of quantization for texture region edge regions, respectively, with f_l2 set to 0 and f_h1 set to 0.5; the flatness range is set to [0,1], with larger values ​​indicating greater flatness; flatness and gradient magnitude are inversely proportional, with larger gradient magnitudes indicating less flatness;

[0034] A value greater than G_th(p) indicates that the texture is valid, i.e., the texture edge area, with flatness = 0;

[0035] A value less than or equal to G_tl(p) indicates relative flatness, with a flatness range of [0.5, 1.0].

[0036] Flatness values ​​between the two thresholds indicate relative unevenness and can be considered as potentially textured edge regions. The flatness range is [0, 0.5].

[0037] f_l2 = f_h1.

[0038] S4, based on the filtering result of the NLM filter at pixel p, i.e., the result of flatness Flat(p), then:

[0039] (1) Calculate the weights w(p,q) of the filter kernel based on the flatness Flat(p);

[0040] (2) Calculate the final filtering result out(p) using the weights w(p,q);

[0041] The calculation is as follows:

[0042]

[0043] Among them, w s (p,r) represents a search window w centered at p with radius r. s ; in(q) represents the pixel value at neighboring pixel q, w(p,q) is the corresponding filter weight; c(p) is the sum of the filter weights; w b (p,f) represents a matching window centered at p with radius f, and the corresponding matching window at q is w. b (q,f), d 2 Indicates w b (p,f) and w b Euclidean distance between pixels within (q,f); δ 2 Let p represent the noise variance, and h be δ. 2 The two parameters together determine the noise reduction strength of the filter;

[0044] w(p,q) is a power function of e. The larger δ and h are, the smoother the curve changes, the greater the contribution of neighboring pixels to the output, and the smoother the output image. In other words, the larger δ and h are, the greater the noise reduction strength of the filter.

[0045] S5, when p is located in a flat region, the similarity between neighboring pixels and the center pixel block is less than 2δ. 2 With the same weight, the block similarity is greater than 2δ. 2 The weight of neighboring points decreases exponentially as block similarity increases;

[0046] When p is located in the edge region, the flatness Flat(p) is 0. The weight of neighboring pixels decreases exponentially as block similarity increases;

[0047] Compared to flat regions, edge regions exhibit faster weight decay and thus offer better edge preservation.

[0048] In step S3, the flatness range is set to [0,1], and the larger the value, the greater the flatness. Flatness and gradient magnitude are inversely proportional, and the larger the gradient magnitude, the less flat the surface.

[0049] A value greater than G_th(p) indicates that the texture is valid, i.e., the texture edge area, with flatness = 0;

[0050] A value less than or equal to G_tl(p) indicates relative flatness, with a flatness range of [0.5, 1.0].

[0051] Flatness values ​​between the two thresholds indicate relative unevenness and can be considered as potentially textured edge regions. The flatness range is [0, 0.5].

[0052] f_l2 = f_h1.

[0053] The gradient edge detection method includes the Canny edge detection method.

[0054] The method is applied to the ISP noise reduction module of an image processing chip, which can improve the quality of the ISP output image.

[0055] Therefore, this application has the following advantages:

[0056] By combining gradient edge detection, the NLM filter's ability to perceive image texture and edges is improved. Flat(p) is adaptively adjusted to enhance the filter's edge preservation capability. Attached Figure Description

[0057] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0058] Figure 1 This is a schematic diagram of the overall process of this method. Detailed Implementation

[0059] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0060] This application proposes an improved NLM filter method based on gradient edge detection, which can be applied to the ISP noise reduction module of an image processing chip to improve the quality of the ISP output image. Figure 1 As shown, it includes:

[0061] S1, the gradient edge detection method uses an edge detection operator to calculate the gradient values ​​in four directions: horizontal, 45 degrees, vertical, and 135 degrees. A differential filter is used: symmetrical weights have opposite signs, and generally, the absolute value of the weight is larger closer to the center. Taking a 3x3 image as an example, the definition is as follows:

[0062]

[0063] Setting all non-zero weights to 1 or -1 is equivalent to smoothing the input, which improves the noise resistance of the edge detection method.

[0064] S2, let the absolute values ​​of the gradients in the four directions be G_0, G_45, G_90, and G135, respectively. Then the gradient magnitude at pixel p is calculated using the following formula:

[0065] G(p)=max(G_0+G_90,G_45,G_135)

[0066] Set two gradient thresholds G_tl(p) and G_th(p);

[0067] G_tl(p) = func(in(p))

[0068] G_th(p) = G_tl(p) * scale

[0069] Wherein, G_tl(p) represents the threshold under the gradient, which is a function of the pixel value or brightness value at p; G_th(p) is the threshold on the gradient, which is linearly related to G_tl(p); scale is an empirical value, which is generally taken as [3,5]; G_tl(p) is defined as a function of pixel or brightness, so the threshold is an adaptive threshold and has global characteristics;

[0070] S3, Next, the flat region and the edge region are determined by thresholding, and the gradient magnitude is quantized to a specified range to describe the flatness of the image;

[0071]

[0072] Where f_l1 and f_h1 are the lower and upper limits of quantization for the flat region, with f_l1 set to 0.5 and f_h1 set to 1.0; regions with gradient magnitude G(p) greater than G_th(p) are considered edge regions, and their flatness Flat(p) is set to 0; f_l2 and f_h2 are the lower and upper limits of quantization for the texture region edge region, with f_l2 set to 0 and f_h1 set to 0.5;

[0073] S4, calculated based on the filtering result of the NLM filter at pixel p, is as follows:

[0074]

[0075] Among them, w s (p,r) represents a search window w centered at p with radius r. s ; in(q) represents the pixel value at neighboring pixel q, w(p,q) is the corresponding filter weight; c(p) is the sum of the filter weights; w b (p,f) represents a matching window centered at p with radius f, and the corresponding matching window at q is w. b (q,f), d 2 Indicates w b (p,f) and w b The Euclidean distance between pixels within (q,f); w(p,q) is a power function of e. The larger δ and h are, the smoother the curve changes, the greater the contribution of neighboring pixels to the output, and the smoother the output image. In other words, the larger δ and h are, the greater the noise reduction strength of the filter. It can be determined according to the actual needs of the scene.

[0076] S5, when p is located in a flat region, the similarity between neighboring pixels and the center pixel block is less than 2δ. 2 With the same weight, the block similarity is greater than 2δ. 2 The weight of neighboring points decreases exponentially as block similarity increases;

[0077] When p is located in the edge region, the flatness Flat(p) is 0. The weight of neighboring pixels decreases exponentially as block similarity increases.

[0078] Compared to flat regions, edge regions exhibit faster weight decay and thus offer better edge preservation.

[0079] In summary, this method adaptively adjusts the noise variance based on the quantization gradient calculation results to improve the NLM filter's ability to perceive image texture and edges, where δ 2 It is δ in the above formula W(p,q) 2 That is, the filtering strength is adaptively achieved by the flatness Flat(p).

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for improving NLM filters based on gradient edge detection, characterized in that, The filtering intensity in the method is adaptively achieved by the flatness Flat(p), including: S1 uses the gradient edge detection method to calculate the gradient values ​​of the image in four directions: horizontal, 45 degrees, vertical, and 135 degrees. A differential filter, assuming to be 3x3, is used and defined as follows: Setting all non-zero weights to 1 or -1 is equivalent to smoothing the input, which improves the noise resistance of the edge detection method. S2, let the absolute values ​​of the gradients in the four directions be G_0, G_45, G_90, and G135, respectively. Then the gradient magnitude at pixel p is calculated using the following formula: G(p)=max(G_0+G_90,G_45,G_135) Set two gradient thresholds G_tl(p) and G_th(p); G_tl(p) = func(in(p)) G_th(p) = G_tl(p) * scale Where G_tl(p) represents the lower threshold of the gradient, which is a function of the pixel value or brightness value at p; G_th(p) is the upper threshold of the gradient, which is linearly related to G_tl(p); and scale is an empirical value, which takes values ​​[3,5]. G_tl(p) is defined as a function of pixel or brightness, so that the threshold is an adaptive threshold and has global characteristics; S3, Next, the flat region and the edge region are determined by thresholding, and the gradient magnitude is quantized to a specified range to describe the flatness of the image; Where f_l1 and f_h1 are the lower and upper limits of quantization for flat regions, respectively, with f_l1 set to 0.5 and f_h1 set to 1.0; regions where the gradient magnitude G(p) is greater than G_th(p) are edge regions, and the flatness Flat(p) is directly set to 0; because a gradient greater than the upper threshold indicates that this is definitely a valid texture, and the flatness is minimal, so flatness = 0; f_l2 and f_h2 are the lower and upper limits of quantization for texture region edge regions, respectively, with f_l2 set to 0 and f_h1 set to 0.5; S4, based on the filtering result of the NLM filter at pixel p, i.e., the result of flatness Flat(p), then: (1) Calculate the weights w(p,q) of the filter kernel based on the flatness Flat(p); (2) Calculate the final filtering result out(p) using the weights w(p,q); The calculation is as follows: Among them, w s (p,r) represents a search window w centered at p with radius r. s ; in(q) represents the pixel value at neighboring pixel q, w(p,q) is the corresponding filter weight; c(p) is the sum of the filter weights; w b (p,f) represents a matching window centered at p with radius f, and the corresponding matching window at q is w. b (q,f), d 2 Indicates w b (p,f) and w b Euclidean distance between pixels within (q,f); δ 2 Let p represent the noise variance, and h be δ. 2 The two parameters together determine the noise reduction strength of the filter; w(p,q) is a power function of e. The larger δ and h are, the smoother the curve changes, the greater the contribution of neighboring pixels to the output, and the smoother the output image. In other words, the larger δ and h are, the greater the noise reduction strength of the filter. S5, when p is located in a flat region, the similarity between neighboring pixels and the center pixel block is less than 2δ. 2 With the same weight, the block similarity is greater than 2δ. 2 The weight of neighboring points decreases exponentially as block similarity increases; When p is located in the edge region, the flatness Flat(p) is 0. The weight of neighboring pixels decreases exponentially as block similarity increases; Compared to flat regions, edge regions exhibit faster weight decay and thus offer better edge preservation.

2. The method for improving the NLM filter based on gradient edge detection according to claim 1, characterized in that, In step S3, the flatness range is set to [0,1], and the larger the value, the greater the flatness. Flatness and gradient magnitude are inversely proportional, and the larger the gradient magnitude, the less flat the surface. A value greater than G_th(p) indicates that the texture is valid, i.e., the texture edge area, with flatness = 0; A value less than or equal to G_tl(p) indicates relative flatness, with a flatness range of [0.5, 1.0]. Flatness values ​​between the two thresholds indicate relative unevenness and can be considered as potentially textured edge regions. The flatness range is [0, 0.5]. f_l2 = f_h1.

3. The method for improving the NLM filter based on gradient edge detection according to claim 1, characterized in that, The gradient edge detection method includes the Canny edge detection method.

4. The method for improving the NLM filter based on gradient edge detection according to claim 1, characterized in that, The method is applied to the ISP noise reduction module of an image processing chip, which can improve the quality of the ISP output image.

Images