An edge-preserving image smoothing method with sparse gradient enhancement

Abstract

The present application provides a kind of sparse gradient enhanced edge preserving image smoothing method, comprising: introducing structure fidelity term on the basis of existing global optimization model containing data fidelity term, in structure fidelity term, the gradient of input image is preprocessed using the way of combining mapping function and bilateral filtering;And introduce sparse gradient enhancement term on the basis of structure fidelity term, in sparse gradient enhancement term, L2 norm and L1 norm are used to constrain the gradient of output image to enhance sparsity, L1 norm is iteratively solved by combining subgradient method, fast Fourier transform and alternating direction multiplier method, or L p norm is used to constrain the gradient of output image, and L p Norm is converted to L1 norm for solving by iterative reweighting method, and preconditioned conjugate gradient method is used to improve calculation efficiency in solving process.The present application can retain complete semantic information while showing sparse smoothing effect when processing simple and complex non-texture images and texture images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a sparse gradient enhancement method for edge-preserving image smoothing. Background Technology

[0002] Edge-preserving smoothing is a crucial image preprocessing technique. Its main goal is to smooth noise and unwanted details during image processing while preserving important contours and significant structural features. Image smoothing essentially decomposes an image into a base layer and a detail layer. The base layer reflects the image's structural information, while the detail layer contains its texture information. Further manipulation and recombination of the base and detail layers have wide applications in areas such as image edge extraction, image detail enhancement, image denoising, HDR (High Dynamic Range) tone mapping, guided depth map upsampling, and grayscale image colorization.

[0003] Existing edge-preserving smoothing algorithms can be divided into three main categories: weighted averaging-based smoothing algorithms, global optimization-based smoothing algorithms, and deep learning-based smoothing algorithms. Weighted averaging-based smoothing algorithms utilize well-defined filter kernels to preserve edges. These algorithms achieve smoothing by weighted averaging of pixel values ​​within a local window. Global optimization-based smoothing algorithms achieve edge preservation by designing fidelity and penalty terms. In these algorithms, the data term (fidelity term) is typically kept unchanged, and the relationship between the output and input images is constrained by the L2 norm to avoid image distortion. The penalty term penalizes the image gradient by introducing various norms to achieve a smoothing effect. Deep learning-based image smoothing algorithms learn traditional filters using different network architectures.

[0004] Smoothing algorithms based on global optimization typically begin by establishing a global optimization model, then solving this model to obtain the optimal solution, ultimately generating the output image. Existing global optimization models usually include two important components: a data term (fidelity term) and a penalty term (regularization term). Weighted least squares filtering (WLS) is a classic method based on global optimization; however, it is time-consuming because it involves solving a large linear system. To improve the computational efficiency of WLS, Min et al. proposed Fast Global Filter (FGS) based on weighted least squares. This method approximates the dense linear system by solving a series of one-dimensional subproblems, thus accelerating the original weighted least squares filtering algorithm. However, since it only considers one-dimensional information, this can lead to blocky filtering results. Therefore, Liu et al. proposed a semi-global weighted least squares filtering algorithm (SG-WLS) that incorporates some two-dimensional nearest neighbor information while preserving one-dimensional nearest neighbor information. To obtain sparse smoothing results, Bi et al. proposed an edge-preserving smoothing algorithm based on L1 norm image transform, which penalizes the image gradient using the L1 norm. Yang et al. proposed the L1 gradient reconstruction model (L1-R), which uses a mapping function to suppress texture information and then constrains the relationship between the output image gradient and the preprocessed image gradient through the L1 norm, thereby achieving image smoothing. Xu et al. proposed a filter based on minimizing the L0 norm gradient, which sharpens major edges while smoothing details by controlling the number of non-zero gradients. Xu et al. proposed the Relative Total Variational Filter (RTV) for textured images, a classic algorithm in global optimization, aiming to extract structural information from texture. Based on the idea of ​​RTV, in recent years, He et al. proposed the scale-aware bilateral total variational structure-preserving texture filter (BTV), which effectively quantifies texture and structural features by forming bilateral total variational regularization terms. Li et al. proposed the Enhanced Relative Total Variational Edge Preserving Smoothing Algorithm with Embedded Bilateral Filter (ERTVBF), which can simultaneously preserve significant edge information and weak structural information of interest. Edge preserving smoothing algorithms based on global optimization need to find the optimal solution globally, thus consuming significantly more time compared to weighted average smoothing algorithms. Furthermore, these algorithms are typically designed with only textured images or natural non-textured images in mind, which leads to a lack of robustness.

[0005] In daily life, images can be categorized into textured images and non-textured images. Further, based on the strength of their contained detail and structural information, images can be classified as simple images and complex images. Simple images contain significant structural information while only containing weak texture information, making them visually distinct. In contrast, complex images exhibit a coexistence of weak and strong structures, making them visually difficult to differentiate. Existing global optimization algorithms typically achieve image data fidelity only through data items (fidelity items), which may destroy the semantic information within the image. Summary of the Invention

[0006] Existing global optimization models typically consist of data fidelity terms and regularization terms. Data fidelity terms aim to ensure that the smoothed image u is similar to the input image g, and generally remain unchanged throughout the design process. Regularization terms achieve smoothing by imposing various norm constraints on the gradient of the output image. The general optimization framework is as follows:

[0007]

[0008] Where Φ(·) represents different penalties applied to the output image u, and λ is used to adjust the relationship between the data fidelity term and the regularization term. Experiments and analysis show that existing global optimization methods cannot effectively preserve the semantic information of images simply by designing data fidelity terms. This is because different input images contain texture and structural information of varying complexity, which may lead to the destruction of the image's structural information.

[0009] In existing methods, the least squares filtering algorithm with embedded bilateral filtering and the L1 gradient reconstruction model process the input image gradient through different preprocessing operations to achieve structure preservation. They respectively constrain the relationship between the output image gradient and the preprocessed image gradient using the L2 norm and L1 norm, thus achieving image smoothing. However, since they directly use the preprocessed image gradient as a regularization term to penalize the output image gradient, a trade-off must be made between smoothing quality and edge preservation performance. The least squares filtering algorithm with embedded bilateral filtering performs well in terms of structure preservation while improving the efficiency of the original weighted least squares filtering algorithm. However, this algorithm directly preprocesses the input image gradient through bilateral filtering and uses the L2 norm to constrain the similarity between the output image gradient and the preprocessed image gradient. This method is prone to undersmoothing when processing complex non-textured images and cannot effectively remove texture information when processing textured images.

[0010] In contrast, the L1 gradient reconstruction model also preprocesses the input image gradient. However, the difference lies in that the L1 gradient reconstruction model distinguishes between structural and texture information through a mapping function. Specifically, it determines whether the gradient of the input image belongs to texture or structure by manually setting a threshold. If the input image gradient is determined to be texture, the texture is suppressed, making the gradient smaller; if the input image gradient is determined to be structural, the structural information is preserved. Subsequently, the L1 norm is used to constrain the relationship between the output image and the gradient of the image after preprocessing by the mapping function, thereby achieving a smoothing effect. However, modern images exhibit rich diversity, and using a threshold to distinguish between texture and structural information has certain limitations. This is because this method is highly dependent on the choice of threshold. If the threshold is not chosen appropriately, it is easy to misclassify structural information as texture information, leading to the destruction of structural information; or misclassify texture information as structural information, retaining unnecessary details and resulting in unsmoothness. Furthermore, this model uses the L1 norm for constraint, and since the L1 norm is sparse, the appropriateness of the threshold selection further affects the image smoothing effect. Figure 2 The figure shows a comparison of the effects of different gradient preprocessing methods on simple and complex non-textured images. As can be seen from the figure, compared with L1-R (L1 gradient reconstruction edge-preserving filter model), the structure fidelity term proposed in this invention and BLF-LS (least square edge-preserving image smoothing with embedded bilateral filter) can preserve the complete structure of the image. However, this invention performs better in removing minute textures.

[0011] The inventors of this application discovered through research that the above-mentioned methods, when combining structure preservation with gradient penalty design, can lead to the destruction of image structure or the appearance of unsmoothness. In response to the above-mentioned technical problems, this invention proposes a global optimization model composed of three parts: data fidelity term, structure fidelity term, and sparse gradient enhancement term. It can not only preserve complete semantic information, but also exhibit sparse smoothing effect when processing both simple and complex non-texture images.

[0012] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0013] A sparse gradient-enhanced edge-preserving image smoothing method includes the following steps:

[0014] S1. Based on the existing global optimization model that includes a data fidelity term, a structure fidelity term is introduced. In the structure fidelity term, the gradient of the input image is preprocessed using a combination of a mapping function and bilateral filtering. First, the pixel values ​​of the input image are normalized from 0 to 255 to 0 to 1, and a threshold is set to compress subtle texture information in the image without distinguishing whether the gradient of the input image belongs to texture or structure. This completes the mapping function processing. The formula of the mapping function is as follows:

[0015]

[0016] Where sign(·) is the sign function, h is the gradient of the input image, σ is the threshold, and α is the degree of compression;

[0017] Next, the image gradient after being processed by the mapping function is processed by bilateral filtering so that the output image gradient retains only significant structural information. Finally, the relationship between the output image gradient and the image gradient after being preprocessed by the mapping function and bilateral filtering is constrained by the L2 norm, thereby achieving structural fidelity of the image and better preserving the complete semantic information of the image.

[0018] S2. Based on the structure fidelity term, a sparse gradient enhancement term is introduced. In this term, the L2 and L1 norms are used to constrain the output image gradient to enhance sparsity. The L1 norm is iteratively solved by combining the subgradient method, fast Fourier transform, and alternating direction multiplier method, thereby effectively achieving image smoothing while preserving edges; or...

[0019] Using L p The norm constrains the gradient of the output image, where 0 < p ≤ 1 or p = 2. When p = 1 or p = 2, L p The norm is a convex optimization problem, and when 0 < p < 1, L p The norm is a nonconvex optimization problem. In order to efficiently solve L when 0 < p < 1, p The problem of norms is solved by using an iterative reweighting method to address the L-norm problem when 0 < p < 1. p The problem is solved by converting the L1 norm to the L1 norm, which transforms the non-convex optimization problem into a convex optimization problem. Furthermore, the preconditional conjugate gradient method is used to improve computational efficiency during the solution process.

[0020] Furthermore, in step S1, the threshold σ for compressing the fine texture information in the image is set to 0.05 to 0.3, and the compression degree α is set to 2 to 3.

[0021] Furthermore, in step S2, when 0 < p < 1, L p The process of solving for the norm is as follows:

[0022] Given an input image g, the output image u is obtained through the following optimization formula:

[0023]

[0024] Wherein, λ and γ are balancing weights used to adjust the trade-off between the data fidelity term, the structure fidelity term, and the sparse gradient enhancement term; L pNorm; c(·) is the gradient compression function; f BLF (·) indicates bilateral filtering; This indicates the differentiation; x and y represent the x and y directions of the graph;

[0025] In the above optimized formula Rewrite it in the following form:

[0026]

[0027] The above equation can be rewritten in matrix form as follows:

[0028]

[0029] Among them, Diag(w * D is a diagonal matrix. x and D y Let the difference matrices be the x-direction and the y-direction, respectively. Then the above optimization formula can be rewritten in the following form:

[0030]

[0031] When solving the subproblem of minimizing the filtered output image u, the preconditional conjugate gradient method will be used to solve the following equation:

[0032]

[0033] Where ρ is the Lagrange penalty coefficient, and ρ > 0; k is the k-th iteration;

[0034] make η x and η y It is a dual variable;

[0035] For Z x and Z y The minimization subproblem is solved and expressed using the shrink operator as follows:

[0036]

[0037] For d x and d y The solution to the dual problem is as follows:

[0038]

[0039] Compared with existing technologies, the sparse gradient enhancement edge-preserving image smoothing method provided by this invention has the following advantages:

[0040] 1. This invention provides a novel global optimization model framework that can effectively smooth textured and non-textured images, as well as simple and complex images, while preserving edges, making the filter more robust.

[0041] 2. This invention not only considers the data fidelity of the image, but also the structural fidelity of the image, preserving both strong and weak structural information of the image.

[0042] 3. By introducing a structure fidelity term, fine texture information is suppressed to a certain extent, and the problem of relying on thresholds to classify texture and structure can be avoided.

[0043] 4. By introducing a sparse gradient enhancement term, it is easier to process low-frequency signals during image smoothing and achieve a sparse smoothing effect.

[0044] 5. This invention uses L p Norm constraints output gradient, L p Converting the norm to the L1 norm for solving yields a sparser and smoother result. When p = 1, the Fast Fourier Transform (FFT) is used to transform the spatial domain to the frequency domain for computation; when 0 < p < 1, the preconditional conjugate gradient (PCG) method is used to accelerate matrix operations, thereby improving computational efficiency. Specifically... Figure 3 The image shows a comparison of the smoothing effects of different filters after gradient preprocessing of simple and complex non-textured images. Figure 3 Medium smoothing results and Figure 2 The parameters for gradient preprocessing remain consistent. Observing the smoothed results, it can be seen that BLF-LS exhibits good smoothing effects when processing simple non-textured images; however, when dealing with complex non-textured images, undersmoothing occurs in areas like grass. In contrast, L1-R produces sparse smoothing effects; however, because L1-R destroys some structural information during gradient preprocessing of the input image, the smoothed image becomes somewhat blurry. Compared to the two filters mentioned above, the algorithm proposed in this invention not only preserves complete semantic information but also exhibits sparse smoothing effects when processing both simple and complex non-textured and textured images. Attached Figure Description

[0045] Figure 1 This is a qualitative analysis comparison chart of the present invention and other classic filters on noisy images.

[0046] Figure 2 This is a comparison chart showing the effects of the present invention and other gradient preprocessing methods with different filters on simple images and complex non-texture images.

[0047] Figure 3This is a comparison chart of the smoothing effects of the present invention and other filters after gradient preprocessing of simple and complex non-textured images.

[0048] Figure 4 This is a comparison chart showing the effects of different λ and γ values ​​on the smoothing effect of two-dimensional images when the sparse gradient enhancement term of this invention is constrained by the L1 norm.

[0049] Figure 5 This is a comparison chart showing the smoothing effect of the sparse gradient enhancement term of the present invention using different regularization constraints.

[0050] Figure 6 This is a comparison chart of the smoothing results of the present invention and various filters on non-textured images.

[0051] Figure 7 This is a comparison chart of the visual effects of the present invention and various filters on the NKS dataset.

[0052] Figure 8 This is a comparison chart of the smoothing results of the present invention and texture filters on simple textured images.

[0053] Figure 9 This is a comparison chart of the smoothing results of the present invention and texture filters on complex texture images.

[0054] Figure 10 This is a comparison chart of the visual effects of the present invention and various filters in enhancing image details.

[0055] Figure 11 This is a comparison chart of the visual effects of the present invention and various filters in edge extraction.

[0056] Figure 12 This is a comparison chart of the visual effects of the present invention and various filters in HDR tone mapping.

[0057] Figure 13 This is a comparison chart of the visual effects of the present invention and various filters in removing clipart compression artifacts.

[0058] Figure 14 This is a comparison chart of the image abstraction visual effects of the present invention and various filters. Detailed Implementation

[0059] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0060] Compared with existing global optimization models, the global optimization model constructed in this invention consists of three parts: a data fidelity term, a structure fidelity term, and a sparse gradient enhancement term. This invention not only preserves complete semantic information but also exhibits sparse smoothing effects when processing both simple and complex non-texture images. To achieve the above objectives, this invention provides the following technical solution:

[0061] A sparse gradient-enhanced edge-preserving image smoothing method includes the following steps:

[0062] S1. Based on the existing global optimization model that includes a data fidelity term, a structure fidelity term is introduced. In the structure fidelity term, the gradient of the input image is preprocessed using a combination of a mapping function and bilateral filtering. First, the pixel values ​​of the input image are normalized from 0 to 255 to 0 to 1, and a small threshold is set to compress the subtle texture information in the image without distinguishing whether the gradient of the input image belongs to texture or structure. This completes the mapping function processing. The formula of the mapping function is as follows:

[0063]

[0064] Where sign(·) is the sign function, h is the gradient of the input image, σ is the threshold, and α is the degree of compression;

[0065] Next, the image gradient after being processed by the mapping function is processed by bilateral filtering so that the output image gradient retains only significant structural information. Finally, the relationship between the output image gradient and the image gradient after being preprocessed by the mapping function and bilateral filtering is constrained by the L2 norm, thereby achieving structural fidelity of the image and better preserving the complete semantic information of the image.

[0066] S2. A sparse gradient enhancement term is introduced based on the structure-fidelity term. Sparse regularization constraints are used to preserve complete semantic information while achieving a more sparse smoothing effect. In the sparse gradient enhancement term, to obtain a sparse smoothing effect, the L2 norm and L1 norm are used to constrain the output image gradient to enhance sparsity. Since the L2 and L1 norms have different properties, the L2 norm can make some low-frequency signals close to 0 but not completely 0, while the L1 norm can make some low-frequency signals completely 0. Therefore, compared with the L2 norm, the L1 norm can achieve a more sparse smoothing effect. Although the L1 norm is not differentiable, this invention iteratively solves the L1 norm by combining the subgradient method, fast Fourier transform, and alternating direction multiplier method (ADMM), thereby effectively achieving image smoothing while preserving edges; or...

[0067] To achieve a sparser, smoother effect, L is used. p The norm constrains the gradient of the output image, where 0 < p ≤ 1 or p = 2. When p = 1 or p = 2, L p The norm is a convex optimization problem, and when 0 < p < 1, L p The norm is a nonconvex optimization problem. In order to efficiently solve L when 0 < p < 1, p Regarding the norm problem, this invention addresses the issue of L when 0 < p < 1 by using an iterative reweighting method.p The problem is solved by converting the L1 norm to the L1 norm, which transforms the non-convex optimization problem into a convex optimization problem. Furthermore, the preconditional conjugate gradient (PCG) method is used to improve computational efficiency during the solution process.

[0068] In a specific embodiment, in step S1, the threshold σ for compressing the subtle texture information in the image is set to 0.05–0.3, and the compression degree α is set to 2–3. This avoids the need to manually set thresholds to determine whether the gradient of the input image belongs to texture or structure. Modern images exhibit rich diversity, and directly distinguishing between structure and texture using thresholds has certain limitations. This is because this method heavily relies on the selection of the threshold. If an inappropriate threshold is chosen, it is easy to incorrectly classify structural information as texture information, leading to the destruction of structural information; or to incorrectly classify texture information as structural information, retaining unnecessary details and resulting in an unsmooth appearance.

[0069] To better understand this invention, the technical solutions provided by this invention will be described in detail below.

[0070] Unlike existing global optimization-based smoothing algorithms that consist of a data term and a penalty term, the model proposed in this invention consists of three parts: a data fidelity term, a structure fidelity term, and a sparse gradient enhancement term. Given an input image g, the output image u can be obtained using the following optimization equation (2):

[0071]

[0072] Wherein, λ and γ are balancing weights used to adjust the trade-off between the data fidelity term, the structure fidelity term, and the sparse gradient enhancement term; L p Norm; c(·) is the gradient compression function; f BLF (·) indicates bilateral filtering; This represents finding the differential; x and y represent the x and y directions of the graph. We will first introduce the solution process when p = 2, then the solution process when p = 1, and finally extend it to L when 0 < p < 1. p Norm. When p = 2, equation (2) becomes the following form:

[0073]

[0074] make Taking its derivative with respect to u and setting it to 0, it can be expressed as:

[0075]

[0076] Among them, D x and D yIt is a discrete differential operator, and equation (4) can be solved to obtain the following form:

[0077]

[0078] Where I is the identity matrix, Assuming the input image g has a size of H×W, solving equation (5) requires solving an HW×HW matrix, which is very time-consuming. Therefore, this invention advocates solving it in the Fourier domain, and the calculation method is as follows:

[0079]

[0080] When p = 1, equation (2) becomes the following form:

[0081]

[0082] To effectively solve the edge-preserving smoothing filtering algorithm with sparse gradient enhancement proposed in this invention, this invention employs an iterative approach combining alternating direction multipliers and Fast Fourier Transform. Let... By introducing Lagrange multipliers, an augmented Lagrange function is constructed, and equation (7) is expressed in the following form:

[0083]

[0084] Where, η x and η y Let be the dual variable, ρ be the Lagrange penalty coefficient, and ρ > 0. By introducing the augmented Lagrange method, the global optimization function is relaxed to a strictly convex function, thereby improving the robustness of the algorithm. Let The Lagrange penalty term and its dual term are combined and expressed in the following form:

[0085]

[0086] Then, equation (7) is reformulated in the following form:

[0087]

[0088] Equation (11) can be solved by iterative minimization. The (k+1)th iteration includes the following steps, where the minimization subproblem for solving the problem u (the filtered output image u) is as follows:

[0089]

[0090] Problem Solving Z x and Z y The minimization subproblem is as follows:

[0091]

[0092]

[0093] Regarding d x and d y The solution to the dual problem is as follows:

[0094]

[0095] Equation (12) requires a significant amount of time to solve as a linear system. Therefore, this invention employs the Fast Fourier Transform (FFT) to transform the problem from the spatial domain to the frequency domain, transforming the matrix inversion process into point-to-point division in the frequency domain, thereby improving computational efficiency. The closed-form solution of equation (12) is expressed in the following form:

[0096]

[0097] Among them, F(·) and F -1 (·) represent Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT), respectively. F represents the complex conjugate of the FFT, while F(1) is the FFT of the impulse function. It is the Hadamard product. All additions, multiplications, and divisions in the above formula are point-to-point operations.

[0098] For equations (13) and (14), the present invention uses a subgradient approach to obtain their closed-form solutions, and expresses them using the shrink operator in the following form:

[0099]

[0100] in, t x and t y They represent and sign(·) and max(·) are the sign function and the maximum value function, respectively. Based on the above formula derivation, the filter proposed in this invention can be described in detail as the following algorithm:

[0101] Input: Input image g, balance parameters λ and γ, ε to determine convergence

[0102] Output: Filtered image u

[0103] 1: Initialization: u {0} =g;

[0104] 2: Gradient preprocessing of the input image g;

[0105] 3: while|u {k} -u {k-1} |>εdo

[0106] 4: Calculate u using formula (17) {k+1}

[0107] 5: Calculate using formula (18)

[0108] 6: The result is obtained from formula (19)

[0109] 7: Calculate using formula (15)

[0110] 8: The calculation formula (16) yields the result.

[0111] 9: end while

[0112] 10: return u{ k}

[0113] To improve the robustness of the algorithm and obtain more sparse and smooth results, this invention employs L... p The norm (0 < p < 1) constrains the sparse gradient enhancement term. When 0 < p < 1, equation (2) is a non-convex optimization problem. To solve L p Norm, this invention uses an iterative reweighting method to transform L p The problem is transformed into solving the L1 norm. In equation (2)... Rewrite it in the following form:

[0114]

[0115] Equation (20) can be rewritten in matrix form as follows:

[0116]

[0117] Among them, Diag(w * D is a diagonal matrix. x and D y Let the difference matrices be the x-direction and the y-direction, respectively. Then, equation (2) can be rewritten in the following form:

[0118]

[0119] When solving the minimization subproblem of problem u (the filtered output image u), this invention will use the preconditional conjugate gradient (PCG) method to solve the following equation:

[0120]

[0121] For Z x and Z y The solution to the minimization subproblem is as follows:

[0122]

[0123] For d x and d y The solution to the dual problem is as follows:

[0124]

[0125] The structural information in an image contains important semantic information. However, existing smoothing algorithms based on global optimization typically only consider data fidelity in terms of image preservation, neglecting the preservation of structural information. Even when a structural fidelity term is designed, it is often also included as a smoothing term, which may lead to the destruction of the image's semantic information or the preservation of a large amount of unnecessary detail. In terms of image smoothing, most existing filtering algorithms can only perform a single smoothing task and cannot handle textured and non-textured images well simultaneously. This invention proposes a novel edge-preserving smoothing algorithm framework based on global optimization. This algorithm not only preserves the complete semantic information of the image but also has excellent edge-preserving ability compared to existing methods. It is applicable to both natural and textured images. The filter has excellent edge-preserving smoothing capabilities (see [link to relevant documentation]). Figure 1 .

[0126] To verify the necessity and rationality of using sparse regularization constraints, this invention will apply L1 regularization, L2 regularization, and L... to the sparse gradient enhancement term respectively. p Regularization is used for constraint. Simultaneously, the same structure-fidelity term is used to preprocess the gradient, but different parameters are adjusted for comparison in the experiments. L1 and L2 regularization have different properties. L2 regularization tends to make the gradient of the output image as small as possible and close to zero, while the most significant characteristic of L1 regularization is sparsity; it can set certain gradient values ​​to zero. Therefore, from the perspective of image smoothing, using the sparse gradient enhancement term of L1 regularization will make the image flatter. Through observation... Figure 4 It is evident that even parameter adjustments to the model employing L2 regularization fail to achieve the superior smoothness exhibited by L1 regularization. This invention utilizes iterative reweighting to adjust the L... p Regularization is converted to L1. By adjusting the value of ρ, even with a small γ, a sparser and smoother result can be obtained.

[0127] To briefly explain the role of parameters λ and γ in controlling the sparse gradient enhancement edge-preserving smoothing algorithm proposed in this invention... Figure 5This demonstrates the different visual effects produced by different values ​​of λ and γ on the image. When λ is fixed, increasing the size of γ yields a smoother effect. As we observed, γ plays a crucial role in controlling smoothness in the sparse gradient enhancement term.

[0128] Image smoothing with edge preservation aims to retain both significant edges and smooth low-amplitude details, seemingly a contradictory goal. A global optimization model is established to find a balance between these two. Existing images can be categorized into textured and non-textured images. Non-textured images typically contain complex scenes and structural information of varying intensities, while textured images contain periodic texture information with some similarity to the structure. Existing methods are usually designed only for either non-textured or textured images, resulting in good filtering performance for non-textured images but failing to remove regular texture information in textured images. The sparse gradient-enhanced edge-preserving smoothing algorithm proposed in this invention performs well in both non-textured and textured images.

[0129] To evaluate the performance of the proposed algorithm, comparative experiments were conducted on the BSDS500 dataset, the NKS dataset, and the texture dataset provided by Xu et al. The comparison methods included four local methods (BLF, GF, BTF, and RGF), two deep learning methods (DEAF and ResNet), and twelve global optimization methods (WLS, L0, RTV, FGS, RoG, BLF-LS, SWF, ILS, G-smooth, L1-R, SGSTF, and SSF). Since the BSDS500 dataset and the texture dataset provided by Xu et al. lack ground truth images, they were used for qualitative analysis, evaluating the smoothing quality subjectively by magnifying local regions in the images. The NKS dataset, which includes both the input image and the ground truth image, was used for quantitative analysis. The algorithm was implemented on an Intel i7-8750H CPU with 16GB of memory, and MATLAB 2018b as the software platform.

[0130] Figure 6This paper showcases the smoothing results of various algorithms on non-textured images and their magnified effects in local regions. Overall, each algorithm applies varying degrees of smoothing to the input image. By magnifying local areas of the image, it can be observed that many algorithms exhibit some degree of undersmoothing when dealing with more complex structures, such as L0, DEAF, ILS, BLF-LS, ResNet, and the Semi-Sparsity filter. The BLF-LS algorithm, in particular, alters the color at the junction of houses and the sky. WLS and G-smooth algorithms demonstrate excellent smoothing effects, but they destroy the original structural information of the image, making it difficult to distinguish information about houses on the mountain. While FGS and L1-R algorithms preserve good structural information and remove unnecessary details, they exhibit some slight artifacts. The sparse gradient-enhanced edge-preserving smoothing algorithm proposed in this invention can not only effectively smooth the details of complex scenes but also preserve the structural information of houses on the mountain and railings on the boat. Figure 7 This paper presents a comparison of the visual effects of various filters on the NKS dataset, where the cat's whiskers represent weak structural information in the image. Most algorithms destroy the whiskers after smoothing the image, while the proposed invention effectively preserves the weak structural information. Table 1 shows a quantitative comparison of the various algorithms on the NKS dataset. It can be seen that the proposed invention achieves the highest values ​​in PSNR, SSIM, and SSEQ metrics, outperforming other algorithms.

[0131] Table 1 compares the quantitative results of various methods on the NKS dataset in terms of PSNR, SSIM, and SSEQ.

[0132]

[0133]

[0134] Figure 8 Qualitative experiments comparing textured images filtered by different filters are presented. Overall, all filters effectively remove texture from the input image while preserving its overall structural information. However, RTV and RoG disrupt the edge structure in the magnified blue rectangle area. BTF and RGF fail to maintain the original shape of the structure in the magnified green area. SWF retains fine texture details at the edges. SGSTF's structure appears overly smooth. G-smooth and L1-R preserve relatively good structure, but still exhibit some jagged edges. In contrast, this invention not only smooths the texture but also preserves the complete structural information, including the original fine structures in the cartoon character. To better evaluate the robustness of this invention's texture removal algorithm, Figure 9This presentation showcases a visual comparison of the smoothing effects of various algorithms on complex textured images. As can be seen from the magnified images, RTV causes damage to eyebrow and facial contours; BTF, RGF, and SGSTF result in varying degrees of blurring in the smoothed image; while SWF and L1-R retain some texture at the edges of the smoothed image. In contrast, RoG, G-smooth, and this invention achieve edge preservation while effectively smoothing texture details.

[0135] To comprehensively evaluate this invention, a series of experiments were conducted, covering multiple tasks in the field of image photography, including detail enhancement, edge extraction, HDR tone mapping, clipart compression and artifact removal, and image abstraction. These experiments aimed to verify the performance and effectiveness of the invention in different application scenarios. To better evaluate the performance of each algorithm in various applications, the reference-free graph metrics ILNIQE and TMQI were introduced for analysis.

[0136] Image smoothing decomposes an image into a base layer and a detail layer. By manipulating and recombinating these layers, different application effects can be achieved. By subtracting the filtered image from the original image, the detail layer—containing the detailed information of the original image—is obtained. Then, the detail layer is magnified and added back to the original image, thereby enhancing the image's detail information and achieving image enhancement. In this experiment, the detail layer enhancement factor was uniformly set to 3.

[0137] Figure 10 The visual effects of various image detail enhancement algorithms are compared. Enlarged images show that L0, RTV, FGS, DEAF, ResNet, and SSF algorithms exhibit halo effects after image detail enhancement, while this invention avoids this phenomenon. To better evaluate the quality of detail enhancement, ILNIQE was used for evaluation. The quantitative analysis results of image enhancement for each algorithm are shown in Table 2, demonstrating that this invention is competitive in detail enhancement applications.

[0138] Table 2 shows the quantitative analysis results of each method in image detail enhancement.

[0139]

[0140] Accurately identifying the edge structure of an image is a challenging task because an image contains not only significant structural information but also complex texture information. Image smoothing can remove low-amplitude details while preserving significant edge structures. Then, using the Canny operator for edge extraction can improve the accuracy of edge acquisition.

[0141] Figure 11This paper presents a visual comparison of the edge extraction performance of various algorithms, with the input including both structurally clear houses and unnecessary details in grassy areas. Visually, compared to other methods, RTV and G-smooth are better at removing details from the grass while preserving the structure of the houses. However, this invention, compared to RTV and G-smooth, better preserves the structural information of the houses, avoiding damage to the roof structure, thus extracting more complete edge information.

[0142] High dynamic range (HDR) images have a wider range of brightness than standard images, but typical display devices cannot reproduce this range. To display HDR images on ordinary devices, they need to be converted to low dynamic range (LDR) images using HDR tone mapping. First, a logarithmic function is applied to the input HDR image to distribute brightness levels evenly over a larger range. Next, the image is decomposed into a base layer and a detail layer using an edge-preserving filter. The base layer is compressed, then recombined with the detail layer, and finally inverse-mapped to generate the tone-mapped image. In this experiment, the compression value is set to 0.25.

[0143] Table 3 shows the quantitative analysis results of each method on the dataset provided by Yeganeh.

[0144]

[0145] Figure 12 The visual effects of various algorithms in HDR tone mapping are compared. WLS, FGS, and DEAF exhibit varying degrees of haloing, while L0 and RTV show varying degrees of artifacts, and SSF lacks sufficient detail in its mapping. Compared to these algorithms, this invention does not produce artifacts and can realistically reflect the HDR tone mapping results. To better evaluate the tone mapping quality of the HDR images generated by each algorithm, TMQI was used as an evaluation metric, and 15 images provided by TMQI were used in the quantitative analysis. The specific evaluation results are shown in Table 3, demonstrating that this invention has certain advantages over many algorithms.

[0146] Clip art images typically exhibit piecewise constant characteristics, displaying distinct, sharp edges. However, when compressed using low-quality JPEG format, artifacts occur at the edges, resulting in blurred edges. Figure 13 The results of removing clipart compression artifacts using various filters are shown. BLF-LS and G-smooth still exhibit artifacts at the edges, while L1-R and the present invention result in insufficient edge preservation at p=1. In contrast, the L1-R filter used in this invention... p The norm method, at p=0.7, can both sharpen edges and effectively remove artifacts.

[0147] Image abstraction is a crucial application of image filtering. The goal of image abstraction is to simplify an image by removing unimportant details while preserving its salient structural information, thereby accurately representing the essential features of the image content. The method proposed in this invention is applicable to non-realistic image abstraction, enhancing edges based on detail flattening. The resulting image abstraction is shown below. Figure 14 As shown.

[0148] Ultimately, through experimental verification, this invention demonstrates superior edge preservation capabilities when processing both textured and non-textured images, and outperforms existing image smoothing methods in a variety of applications.

[0149] Compared with existing technologies, the sparse gradient enhancement edge-preserving image smoothing method provided by this invention has the following advantages:

[0150] 1. This invention provides a novel global optimization model framework that can effectively smooth textured and non-textured images, as well as simple and complex images, while preserving edges, making the filter more robust.

[0151] 2. This invention not only considers the data fidelity of the image, but also the structural fidelity of the image, preserving both strong and weak structural information of the image.

[0152] 3. By introducing a structure fidelity term, fine texture information is suppressed to a certain extent, and the problem of relying on thresholds to classify texture and structure can be avoided.

[0153] 4. By introducing a sparse gradient enhancement term, it is easier to process low-frequency signals during image smoothing and achieve a sparse smoothing effect.

[0154] 5. This invention uses L p Norm constraints output gradient, L p Converting the norm to the L1 norm for solving yields a sparser and smoother result. When p = 1, the Fast Fourier Transform (FFT) is used to transform the spatial domain to the frequency domain for computation; when 0 < p < 1, the preconditional conjugate gradient (PCG) method is used to accelerate matrix operations, thereby improving computational efficiency. Specifically... Figure 3 The image shows a comparison of the smoothing effects of different filters after gradient preprocessing of simple and complex non-textured images. Figure 3 Medium smoothing results and Figure 2The parameters for gradient preprocessing remain consistent. Observing the smoothed results, it can be seen that BLF-LS exhibits good smoothing effects when processing simple non-textured images; however, when dealing with complex non-textured images, undersmoothing occurs in areas like grass. In contrast, L1-R produces sparse smoothing effects; however, because L1-R destroys some structural information during gradient preprocessing of the input image, the smoothed image becomes somewhat blurry. Compared to the two filters mentioned above, the algorithm proposed in this invention not only preserves complete semantic information but also exhibits sparse smoothing effects when processing both simple and complex non-textured and textured images.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A sparse gradient-enhanced edge-preserving image smoothing method, characterized in that, Includes the following steps: S1. Based on the existing global optimization model that includes a data fidelity term, a structure fidelity term is introduced. In the structure fidelity term, the gradient of the input image is preprocessed using a combination of gradient compression function and bilateral filtering. First, the pixel values ​​of the input image are normalized from 0~255 to 0~1, and a threshold is set to compress the subtle texture information in the image without distinguishing whether the gradient of the input image belongs to texture or structure. This completes the gradient compression function processing. The formula of the gradient compression function is as follows: in, It is a symbolic function. The gradient of the input image. For the threshold, The degree of compression; Next, the image gradient after gradient compression is processed using bilateral filtering to ensure that the output image gradient retains only significant structural information. Finally, it is processed... The norm constraint determines the relationship between the output image gradient and the image gradient after preprocessing with gradient compression function and bilateral filtering, thereby achieving structural fidelity of the image and better preserving the complete semantic information of the image. S2. In addition to the structure fidelity term, a sparse gradient enhancement term is introduced. In the sparse gradient enhancement term, the following is adopted: Norm and Norms constrain the gradient of the output image to enhance sparsity. The norm is solved iteratively by combining the subgradient method, the fast Fourier transform, and the alternating direction multiplier method, thereby effectively achieving image smoothing while preserving edges; or, use The norm constrains the gradient of the output image, where or ,when or hour, Norm is a convex optimization problem, and when hour, Norm is a nonconvex optimization problem. In order to effectively solve when hour The problem of norms can be solved by using an iterative reweighting method. hour Norm conversion The norm is used to solve the problem, which is to transform the non-convex optimization problem into a convex optimization problem for solution. Furthermore, the preconditional conjugate gradient method is used to improve computational efficiency during the solution process.

2. The sparse gradient enhancement edge-preserving image smoothing method according to claim 1, characterized in that, In step S1, the threshold for compressing subtle texture information in the image is... Set the compression level to 0.05~0.

3. Set it to 2~3.

Images