An edge-aware guided filtering method combining superpixels and window offsets

Abstract

The present invention provides an edge-aware guided filtering method that fuses superpixels and window offsets, including steps S1, using bilateral filtering to preprocess an input image and then using the SLIC algorithm to perform superpixel segmentation to obtain a superpixel region; S2, calculating the RTV value in different local windows of each pixel point of the input image, selecting the minimum RTV value for window offset, and obtaining a local optimal filtering window based on the RTV value; S3, selecting the overlapping part of the offset window and the superpixel region where the current pixel point is located as the filtering region; S4, combining the previous filtering result with the superpixel region to perform iterative guided filtering, and fusing filtering results of different scales to obtain the final filtering output; S5, proposing an evaluation index for comparing the similarities and differences between the reference filtered image and the input image and the characteristics of the filtered image itself. The present application improves the local filtering window by fusing superpixels and window offsets, improving the edge retention capability of the guided filtering method while achieving a smoothing effect on detail information.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an edge-aware guided filtering method that integrates superpixels and window offsets. Background Art

[0002] Images are affected by various factors during the imaging, transmission, and compression processes, and noise will inevitably be generated, thus affecting the quality of the image. In order to effectively reduce the interference caused by noise, artifacts, etc. and improve the quality of the image, it is very important to filter the image in the image preprocessing stage. Edge-preserving filtering technology plays a very important role in computer vision and image processing applications, such as image denoising, image detail enhancement, structure transfer, image defogging, image fusion, etc. So far, a large number of filters with edge-preserving filtering characteristics have been proposed and developed. Representative methods mainly include bilateral filtering and its related improved methods, weighted least squares filtering, domain transform filtering, guided filtering, etc. Among them, bilateral filtering and guided filtering have been widely used because of their simplicity and high efficiency.

[0003] Although bilateral filtering is simple and easy to use, it cannot maintain the consistency of the pixel gradient direction at the edge, so it may be affected by the gradient reversal artifact near the edge, especially when it is used for image detail enhancement. In order to overcome this shortcoming of bilateral filtering, He et al. proposed guided filtering in 2010. This method starts from a local linear model, considers the relevant information of the guidance image, and calculates the filter output. Guided filtering not only has the edge preservation characteristics of bilateral filtering, but also can avoid the generation of gradient reversal artifacts. Therefore, it is widely used in edge-aware smoothing, detail enhancement, image feathering, high dynamic range compression, image fusion, image denoising, image defogging and other fields. At the same time, inspired by its guidance strategy, several edge preservation methods have been proposed one after another, and guided filtering has also received more and more attention in the field of image processing.

[0004] Guided filtering not only has the same processing speed as bilateral filtering, but also can avoid gradient reversal artifacts. However, the regularization factor in guided filtering is fixed. Therefore, when the pixel is located at the edge structure, blurring will inevitably occur. In 2015, Li et al. proposed weighted guided filtering based on guided filtering, using edge-aware weights to improve the problem of the same penalty factor in guided filtering. Different regularization penalty factors are set for different pixels to achieve better edge preservation. In the same year, inspired by edge perception and gradient gradient, Kou et al. further improved weighted guided filtering by combining clear first-order edge-aware constraints, and proposed a gradient domain guided filtering with stronger edge preservation ability than guided filtering and weighted guided filtering. New edge-aware weights are defined by the standard deviation of the neighborhood. Later, Long et al. proposed an improved weighted guided filtering algorithm based on the LoG edge operator. By calculating the LoG amplitude response in a local window, the penalty factor can be adaptively perceived in different regions, further highlighting the difference between edge pixels and flat area pixels. In 2016, Xie et al. proposed an improved guided filtering algorithm that integrates gradient information. The gradient information of the guidance image is used to determine the edge position of the image, and the weights are designed in combination with the exponential function to control the smoothing strength in different image regions to avoid the halo phenomenon caused by excessive blur near the edge of the image. In 2017, Lu et al. proposed a dual guided filter, which decomposes the image into edge structure and texture details for dual guidance, while achieving structural perception and texture perception, removing texture while better retaining the structural edge. In 2018, Ham et al. proposed a more robust filtering algorithm that considers the structural information of the guidance image and the input image at the same time, achieving a more robust effect against outliers and good edge retention. In 2019, Yin et al. proposed a side window guided filter, which improves the edge retention performance of the guided filter by decomposing each pixel rule window into eight windows: up, down, left, right, northeast, northwest, southeast, and southwest. In 2020, Sun et al. used the steering kernel to adaptively perceive the gradient direction and proposed an improved guided filter that fully utilizes the edge direction information. In the same year, Ochotorena et al. proposed an anisotropic guided filtering algorithm, which uses weighted averaging to achieve maximum diffusion while retaining significant edges in the image. It achieves anisotropic filtering by optimizing weights based on local neighborhood differences, and still has the computational efficiency of guided filtering. In 2021, Deng et al. used local interpolation characteristics and combined with generalized gamma distribution parameter estimation to integrate image smoothing and sharpening operations through guided filtering. Only one parameter value needs to be adjusted to achieve image smoothing and sharpening operations.

[0005] Guided filtering may cause halo artifacts near the edge, although some subsequent improvement methods have improved this problem to a certain extent, making it more capable of retaining edges. However, the inventors of this application have found through research that due to the isotropy of mean filtering, all pixels in the local window are given the same weight, ignoring the feature information of the edge of the image, resulting in blurred edges. Considering the important role of edge structure information in filtering processing, how to innovatively make full use of edge feature information for improvement is a significant task. Summary of the invention

[0006] In order to solve the technical problem that the existing guided filtering assigns the same weight to all pixels in the local window due to the isotropy of mean filtering and ignores the feature information of the image edge, thus causing edge blur, the present invention provides an edge-aware guided filtering method that fuses superpixels and window offsets. By fusing superpixels and window offsets, the local filtering window is improved, the edge retention capability of the guided filtering method is improved, and a smoothing effect on detail information is achieved.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] An edge-aware guided filtering method combining superpixels and window offsets comprises the following steps:

[0009] S1, use bilateral filtering to preprocess the input image, and then use the SLIC algorithm to perform superpixel segmentation on the preprocessed image to obtain a more accurate superpixel area;

[0010] S2, calculating the RTV value in different local windows of each pixel of the input image, selecting the minimum RTV value for window shift, and obtaining the local optimal filtering window based on the RTV value;

[0011] S3, selecting the overlapping part of the offset window and the superpixel area where the current pixel is located as the filtering area, and the pixels contained in the filtering area are highly similar to the current processing pixel;

[0012] S4, retaining the structure edge while smoothing the detail information through local windows of different scales, combining the previous filtering result with the superpixel area to perform iterative guided filtering, and finally fusing the filtering results of different scales to obtain the final filtering output;

[0013] S5. Based on the results of image filtering, an evaluation index is proposed for comparing the similarities and differences between the reference filtered image and the input image, as well as the characteristics of the filtered image itself, to reflect the evaluation of the image filtering effect.

[0014] Further, the step S2 specifically includes:

[0015] S21. Define the information including noise and texture as the spatial oscillation of the image signal. Assuming that the information including noise and texture has a smaller amplitude than the adjacent structure edge, the chromaticity range Δ(ω) in the following formula (1) is k ) is used to measure whether the local window contains a structural edge; then, the local window with the smallest chromaticity range is selected by window offset. However, if the chromaticity range of the local window containing information such as noise and texture is the same as or close to the nearby edge structure, the RTV(ω) in the following formula (2) is finally used. k ) to measure the best local window:

[0016] Δ(ω k )=I max (ω k )-I min (ω k ) Formula (1)

[0017]

[0018]

[0019] Among them, I max (ω k ) and I min (ω k ) represent the maximum and minimum values ​​of the pixel intensity in the local window, Δ(ω k ) represents the local window ω k Within the chromaticity range, represents the gradient of the input image I at pixel j, ε represents a very small positive number, Represents the square of the x-direction gradient of the input image I at pixel j, Represents the square of the y-direction gradient of the input image I at pixel j;

[0020] S22. For each pixel in the input image, select a window with the smallest RTV value for offset to obtain a local optimal filtering window.

[0021] Further, the step S4 specifically includes:

[0022] S41, firstly, a filtering area obtained by a small-scale local window is subjected to self-guided filtering on the input image to obtain a filtering result;

[0023] S42, then use the last filtering result and the superpixel area obtained in step S1 to perform guided filtering with a larger filtering window size, and obtain guided filtering coefficients under filtering windows of different scales:

[0024]

[0025]

[0026] in, and Represents the coefficient of pixel k when the scale is s, and is the guide image G in the final filtering area The mean and variance of is the filter area The number of pixels, λ represents a very small positive number, Indicates that the input image I is in the filter area The mean of :

[0027]

[0028] S43, fusing the guided filtering results of different local window sizes, and calculating the final output image by the following formula:

[0029]

[0030]

[0031]

[0032] in, and represents the mean value of the coefficient of pixel k at different scales, n represents the number of scale filter windows, q k Represents the final filtered output value of pixel k.

[0033] Further, the step S5 specifically includes:

[0034] S51, multiple images are selected and collected from the existing publicly available texture image data set, and the canny operator with a very small threshold is used to perform edge detection on the input image to obtain all edge information of the structure and details;

[0035] S52, separating the image structure edge from the edge information through manual labeling, thereby obtaining the image structure edge and the detail information edge respectively;

[0036] S53, calculate the gradients in the x and y directions at each pixel of the output image and the filtered image, respectively, measure the gradients of the filtered image and the input image at the structure edge and detail edge, and finally integrate them into a unified evaluation index E Q :

[0037]

[0038]

[0039]

[0040] Among them, S and T represent the structure edge and detail information pixel sets respectively, I x (i) and I y (i) is the gradient of pixel i in the original image, Q x (j) and Q y (j) is the gradient of pixel j in the filtered image, Q x (i) and Q y (i) is the gradient of the filtered image pixel i, and the filtered image pixel j is the neighborhood of i, and N(i) represents the 3*3 neighborhood centered on the pixel i.

[0041] Compared with the prior art, the edge-aware guided filtering method for fusing superpixels and window offsets provided by the present invention has the following advantages:

[0042] 1. In the selection of local filtering area, the present invention adopts the overlapping part of window offset and superpixel area as filtering area, considering that when processing structural edge pixels, the regular window centered on it will contain pixels with large differences. By using the idea of ​​window offset and fusing superpixel area, the pixels contained in the filtering area during local filtering are highly consistent with the currently processed pixels, thereby enhancing the ability of guided filtering to retain structural edges.

[0043] 2. When performing guided filtering, a subtle blocking effect may occur when using a local window of a single scale. The present invention adopts a multi-scale iterative method, by increasing the size of the local filtering window, using the last filtering result and the superpixel area to update the filtering output, and finally fusing the filtering outputs of different scales to obtain the output image. This not only well preserves the image structure edge information, but also smoothes the image detail information;

[0044] 3. When evaluating the image filtering effect, the present invention obtains an evaluation index that can better reflect the image filtering effect by comparing the similarities and differences between the filtered image and the original input image as well as the characteristics of the filtered image itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the edge-aware guided filtering method for fusing superpixels and window offsets provided in an embodiment of the present invention.

[0046] Figure 2 This is a visualization diagram of the overlapping part of the window offset and the superpixel area as the final filtering area provided by an embodiment of the present invention.

[0047] Figure 3 This is a structural edge and detail edge result map obtained by using the Canny operator and through manual screening provided by an embodiment of the present invention.

[0048] Figure 4 The filtering result diagrams are obtained by respectively using guided filtering (b), weighted guided filtering (c), gradient domain guided filtering (d) and the method proposed by the present invention (e) on the Man image provided by the embodiments of the present invention.

[0049] Figure 5 This is a filtering result diagram obtained by respectively using joint bilateral filtering (b), rolling guided filtering (c), side window filtering (d), smoothing-sharpening filter (e) and the method proposed by the present invention (f) on the Flower image provided by the embodiment of the present invention.

[0050] Figure 6 It is a mean comparison chart obtained by using the evaluation index proposed by the present invention to collect 20 image data sets provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0052] Please refer to Figure 1 As shown, the present invention provides an edge-aware guided filtering method that fuses superpixels and window offsets, comprising the following steps:

[0053] S1. Input a color image. In order to obtain a more accurate super-pixel region, the input image is first preprocessed by using the existing bilateral filtering, and then the preprocessed image is segmented into super-pixels by using the existing SLIC (Simple Linear Iterative Clustering) algorithm to obtain a more accurate super-pixel region.

[0054] S2, calculating the RTV (relative total variation) value of each pixel in different local windows of the input image, selecting the smallest RTV value for window shift, and obtaining the local optimal filtering window based on the RTV value, specifically including the following steps:

[0055] S21. Considering the noise, texture and other information in the local area, the information including noise and texture is defined as the spatial oscillation of the image signal. Assuming that the information including noise and texture has a smaller amplitude than the adjacent structure edge, the chromaticity range Δ(ω) in the following formula (1) can be simply k ) to measure whether the local window contains a structural edge; then, the local window with the smallest chromaticity range is selected by window offset. However, if the chromaticity range of the local window containing information such as noise and texture is the same as or close to the nearby edge structure, the window offset is calculated by the chromaticity range Δ(ω k) will not be able to accurately select the best local window, so the present invention finally uses the RTV(ω k ) to measure the best local window:

[0056] Δ(ω k )=I max (ω k )-I min (ω k ) Formula (1)

[0057]

[0058]

[0059] Among them, I max (ω k ) and I min (ω k ) represent the maximum and minimum values ​​of the pixel intensity in the local window, Δ(ω k ) represents the local window ω k Within the chromaticity range, represents the gradient of the input image I at pixel j, and ε represents a small positive number such as 10 -15 To prevent the denominator from being 0, Represents the square of the x-direction gradient of the input image I at pixel j, Represents the square of the y-direction gradient of the input image I at pixel j;

[0060] S22. For each pixel in the input image, select a window with the smallest RTV value for offset to obtain a local optimal filtering window.

[0061] S3, the local window after the shift is still regular, and when the pixel is located in the corner area, it will still contain pixels with large differences. Considering the superpixel characteristics, the overlapping part of the shifted window and the superpixel area where the current pixel is located is selected as the filtering area. The pixels contained in the filtering area are highly similar to the current processing pixel. That is, based on step S2, the local window corresponding to the minimum RTV value of each pixel is found, such as Figure 2 As shown in (b), the superpixel area generated in step S1 is then combined to obtain the overlapping part as the final filtering area, as shown in Figure 2 As shown in (c), the influence of difference pixels is avoided, thereby eliminating edge artifacts and achieving edge-aware guided filtering.

[0062] S4, retaining the structure edge while smoothing the detail information through local windows of different scales, combining the previous filtering result with the superpixel area to perform iterative guided filtering, and finally fusing the filtering results of different scales to obtain the final filtering output, which specifically includes the following steps:

[0063] S41, in the filtering area with smaller pixel difference obtained in step S3, firstly, the filtering area obtained by the small-scale local window is subjected to self-guided filtering with the input image to obtain a filtering result, that is, firstly, the small-scale filtering window is used to perform guided filtering to better retain the image structure edge information;

[0064] S42, only guiding filtering in a single local window will produce a slight blocking effect. In order to solve this problem, the present invention adopts a multi-scale iterative filtering method, specifically using the last filtering result and the superpixel area obtained in step S1 to perform guided filtering with a larger filter window size, and obtain guided filtering coefficients under different scale filter windows, that is, first retaining image details through a small scale, and then increasing the filter window radius when performing the next iteration, so as to achieve a smoothing effect. The guided filtering coefficients under specific different scale filter windows are calculated by the following formula:

[0065]

[0066]

[0067] in, and Represents the coefficient of pixel k when the scale is s, and is the guide image G in the final filtering area The mean and variance of is the filter area The number of pixels, λ represents a small positive number such as 10 -15 To prevent the denominator from being 0, Indicates that the input image I is in the filter area The mean of :

[0068]

[0069] S43. Finally, based on the guided filtering results obtained by filtering windows of different scales, that is, the guided filtering results of different local window sizes are fused, the final output image is calculated by the following formula:

[0070]

[0071]

[0072]

[0073] in, and represents the mean value of the coefficient of pixel k at different scales, n represents the number of scale filter windows, q k Represents the final filtered output value of pixel k.

[0074] S5. Based on the result of image filtering, an evaluation index is proposed for comparing the similarities and differences between the reference filtered image and the input image and the characteristics of the filtered image itself, which is used to reflect the evaluation of the image filtering effect, that is, to better reflect the quality of the image filtering output, which specifically includes the following steps:

[0075] S51, select and collect multiple images, such as 20 images, from an existing publicly available texture image dataset (such as the 200 texture image datasets published in the paper published by Xu Li et al. in 2012), use the canny operator with a very small threshold to perform edge detection on the input image, and obtain all edge information of the structure and details;

[0076] S52, separating the image structure edge from the edge information through manual labeling, thereby obtaining the image structure edge and the detail information edge respectively, such as Figure 3 As shown in the figure, (b) is obtained by edge extraction of the input image, (c) is the structural edge obtained by manual extraction, and (d) is the detail information edge obtained by subtraction.

[0077] S53, calculate the gradients in the x and y directions at each pixel of the output image and the filtered image, respectively, measure the gradients of the filtered image and the input image at the structure edge and detail edge, and finally integrate them into a unified evaluation index E Q :

[0078]

[0079]

[0080]

[0081] Among them, S and T represent the structure edge and detail information pixel sets respectively, I x (i) and I y (i) is the gradient of pixel i in the original image, Q x (j) and Q y (j) is the gradient of pixel j in the filtered image, Q x (i) and Q y (i) is the gradient of the filtered image pixel i, and the filtered image pixel j is the neighborhood of i, and N(i) represents the 3*3 neighborhood centered on the pixel i.

[0082] Please refer to Figure 4 As shown in FIG. 1 , the comparison diagram of the filtering results of the Man image using guided filtering, weighted guided filtering, gradient domain guided filtering and the method proposed in the present invention is shown. It can be clearly seen from the local enlarged image that the method proposed in the present invention has a better ability to retain the structure edge.

[0083] Please refer to Figure 5 As shown in the figure, the comparison of the filtering results of the Flower image using the joint bilateral filter, the rolling guide filter, the side window filter, the smoothing-sharpening filter and the method proposed in the present invention is shown. The image has obvious texture detail information. Similarly, through the local magnification image, it can be seen that the method proposed in the present invention can better retain the image structure edge while filtering the texture detail information.

[0084] Please refer to Figure 6 As shown in FIG. 1 , a comparison chart is shown in which different filtering methods are compared using the evaluation index proposed in the present invention on a collected and sorted data set of 20 images. It can be seen that the method proposed in the present invention has a better performance in this index.

[0085] In summary, all theoretical analysis and experimental results fully demonstrate that the method proposed in the present invention has a better effect than other filtering methods in smoothing image detail information while retaining structural edges.

[0086] Compared with the prior art, the edge-aware guided filtering method for fusing superpixels and window offsets provided by the present invention has the following advantages:

[0087] 1. In the selection of local filtering area, the present invention adopts the overlapping part of window offset and superpixel area as filtering area, considering that when processing structural edge pixels, the regular window centered on it will contain pixels with large differences. By using the idea of ​​window offset and fusing superpixel area, the pixels contained in the filtering area during local filtering are highly consistent with the currently processed pixels, thereby enhancing the ability of guided filtering to retain structural edges.

[0088] 2. When performing guided filtering, a subtle blocking effect may occur when using a local window of a single scale. The present invention adopts a multi-scale iterative method, by increasing the size of the local filtering window, using the last filtering result and the superpixel area to update the filtering output, and finally fusing the filtering outputs of different scales to obtain the output image. This not only well preserves the image structure edge information, but also smoothes the image detail information;

[0089] 3. When evaluating the image filtering effect, the present invention obtains an evaluation index that can better reflect the image filtering effect by comparing the similarities and differences between the filtered image and the original input image as well as the characteristics of the filtered image itself.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An edge-aware guided filtering method combining superpixels and window offsets, characterized in that: The following steps are involved: S1, use bilateral filtering to preprocess the input image, and then use the SLIC algorithm to perform superpixel segmentation on the preprocessed image to obtain a more accurate superpixel area; S2, calculating the RTV value in different local windows of each pixel of the input image, selecting the minimum RTV value for window shift, and obtaining the local optimal filtering window based on the RTV value; S3, selecting the overlapping part of the offset window and the superpixel area where the current pixel is located as the filtering area, and the pixels contained in the filtering area are highly similar to the current processing pixel; S4, retaining the structure edge while smoothing the detail information through local windows of different scales, combining the previous filtering result with the superpixel area to perform iterative guided filtering, and finally fusing the filtering results of different scales to obtain the final filtering output; S5. Based on the results of image filtering, an evaluation index is proposed for comparing the similarities and differences between the reference filtered image and the input image, as well as the characteristics of the filtered image itself, to reflect the evaluation of the image filtering effect; wherein, The step S5 specifically includes: S51, multiple images are selected and collected from the existing publicly available texture image data set, and the edge detection is performed on the input image using the canny operator with a very small threshold to obtain all edge information of the structure and details; S52, separating the image structure edge from the edge information through manual labeling, thereby obtaining the image structure edge and the detail information edge respectively; S53, calculate the gradients in the x and y directions at each pixel of the output image and the filtered image, respectively, measure the gradient differences between the filtered image and the input image at the structure edge and detail edge, and finally integrate them into a unified evaluation index E Q : Among them, S and T represent the structure edge and detail information pixel sets respectively, I x (i) and I y (i) is the gradient of pixel i in the original image, Q x (j) and Q y (j) is the gradient of the filtered image pixel j, Q x (i) and Q y (i) is the gradient of the filtered image pixel i, and the filtered image pixel j is the neighborhood of i, and N(i) represents the 3*3 neighborhood centered on the pixel i.

2. The edge-aware guided filtering method for fusing superpixels and window offsets according to claim 1, characterized in that: The step S2 specifically includes: S21. Define the information including noise and texture as the spatial oscillation of the image signal. Assuming that the information including noise and texture has a smaller amplitude than the adjacent structure edge, the chromaticity range △(ω k ) is used to measure whether the local window contains a structural edge; then, the local window with the smallest chromaticity range is selected by window offset. However, if the chromaticity range of the local window containing information such as noise and texture is the same as or close to the nearby edge structure, the RTV(ω) in the following formula (2) is finally used. k ) to measure the best local window: △(ω k )=I max (oh k )-I min (oh k ) expression(1) Among them, I max (ω k ) and I min (ω k ) represent the maximum and minimum values ​​of the pixel intensity in the local window, △(ω k ) represents the local window ω k Within the chromaticity range, represents the gradient of the input image I at pixel j, ε represents a very small positive number, Represents the square of the x-direction gradient of the input image I at pixel j, Represents the square of the y-direction gradient of the input image I at pixel j; S22. For each pixel in the input image, select a window with the smallest RTV value for offset to obtain a local optimal filtering window.

3. The edge-aware guided filtering method for fusing superpixels and window offsets according to claim 1, characterized in that: The step S4 specifically includes: S41, firstly, a filtering area obtained by a small-scale local window is subjected to self-guided filtering on the input image to obtain a filtering result; S42, then use the last filtering result and the superpixel area obtained in step S1 to perform guided filtering with a larger filtering window size, and obtain guided filtering coefficients under filtering windows of different scales: in, and Represents the coefficient of pixel k when the scale is s, and is the guide image G in the final filtering area The mean and variance of is the filter area The number of pixels, λ represents a very small positive number, Indicates that the input image I is in the filter area The mean of : S43, fusing the guided filtering results of different local window sizes, and calculating the final output image by the following formula: in, and represents the mean value of the coefficient of pixel k at different scales, n represents the number of scale filter windows, q k Represents the final filtered output value of pixel k.

Images