Global sparse texture filtering method based on edge structure preservation

Abstract

The invention provides a global sparse texture filtering method based on edge structure preservation, which comprises the following steps of: introducing a texture suppression function into a penalty term, constraining the gradient of an output image, and suppressing texture, noise and unnecessary detail information in the image by setting two threshold values through the texture suppression function, then, the suppressed gradient is used as input of a denominator of a penalty term, so that the penalty term can fully distinguish the texture from the structure; a penalty term is constrained by adopting a sparse regular L1 norm, non-convex optimization is converted into convex optimization by introducing subgradient, and iterative solution is carried out by adopting an alternating direction multiplier method, so that better edge preservation is achieved; and an Lp norm with relatively high sparsity is adopted to constrain a penalty term, and a preprocessing conjugate gradient method is adopted to accelerate and improve the calculation efficiency, so that a more robust and sparsity image smoothing effect is achieved. According to the method, the robustness of an algorithm for distinguishing textures and structures can be improved, relatively good semantic information is reserved, and relatively good smooth performance of edge structure maintenance is achieved.

Description

Technical Field

[0001] The invention relates to the technical field of edge-preserving image smoothing, and in particular to a global sparse texture filtering method based on edge structure preservation. Background Art

[0002] Edge-preserving image smoothing has always been a hot topic in image processing research. It is mainly used to filter out texture and noise while retaining important structures and edges. Various algorithms have been proposed in the past few decades, and edge-preserving image smoothing has been widely used in many computer vision tasks, including detail enhancement, edge extraction, HDR mapping, image denoising, cutout artifact removal, and image animation stylization.

[0003] Existing edge-preserving image smoothing methods are mainly divided into three categories: smoothing algorithms based on local filters, smoothing algorithms based on global optimization, and smoothing algorithms based on deep learning. Among them, the smoothing algorithm based on global optimization usually establishes a global optimization framework, which consists of data terms and penalty terms. The data term is to ensure that the smoothed output image S is as close to the input image I as possible. The data term uses the L2 norm to constrain the deviation between the output and the input, and the penalty term uses various norms to constrain the gradient of the output image, thereby achieving an edge-preserving smoothing effect. The general framework based on the global optimization algorithm is as follows:

[0004]

[0005] Among them, λ is a smoothing parameter used to control the smoothing strength of the image; p is the two-dimensional pixel of the index image; δ(S p ) is the gradient penalty for the output image. That is, it produces a large response for texture and noise-significant areas, and imposes the smallest force constraint on the edge area of ​​the structure as much as possible.

[0006] Among the existing methods, the total variation (TV) model is the earliest type of texture structure decomposition algorithm. The weighted least squares (WLS) and L0 gradient minimization and relative total variation (RTV) models proposed later are effective in removing large-scale noise or texture, but they will smooth out small structure edges and weak structures, making it difficult to keep the edges intact while smoothing efficiently. The relative total variation (RTV) model uses the L2 norm to constrain the structural similarity between the input image and the output image, and imposes the L2 norm on the output image gradient to achieve the purpose of image smoothing. Although the relative total variation (RTV) model has high-quality smoothing performance, it is not friendly to the edge preservation ability of weak structural information, and may even cause the loss of small structures. On the one hand, the penalty term constrained by the L2 norm is used to find the approximation of the true value by minimizing the penalty term, which has weak sparsity and may not be the best at the critical point of distinguishing texture and structure. It is likely to cause the coexistence of texture and structure. To achieve high-quality smoothing, small structures and weak edges will be sacrificed. On the other hand, the penalty term of the relative total variation (RTV) model consists of two terms. Due to the different characteristics of texture and structure in the window, the texture and structure are distinguished by the ratio of the sum of the absolute values ​​of the gradient in the window to the absolute value of the sum of the gradient in the window. This has a good smoothing effect on ordinary texture images, but it will cause under-smoothing for textures of different scales. This is because there are some large-scale textures that are difficult to filter out and are equivalent to significant structures. If large-scale textures are to be smoothed, over-smoothing will occur, that is, weak structures and structural edges will be lost. In summary, the relative total variation (RTV) model based on the L2 norm has weak sparsity and its edge preservation ability needs to be improved, and it lacks robustness. Summary of the invention

[0007] In view of the technical problems that the existing relative total variation model based on L2 norm has the problem of over-smoothing when large-scale textures equivalent to significant structures are present, i.e., weak structures and structural edges are lost, sparsity is weak, edge preservation capability needs to be improved, and robustness is lacking, the present invention provides a global sparse texture filtering method based on edge structure preservation, which fully considers the situation of mixing structure and large-scale texture to improve the robustness of the algorithm in distinguishing texture from structure, retains better semantic information, and achieves better smoothing performance with edge structure preservation.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A global sparse texture filtering method based on edge structure preservation, the method comprising the following steps:

[0010] S1. Introduce the texture suppression function in the penalty term of the global optimization framework to constrain the gradient of the output image. The texture suppression function suppresses the texture, noise, and unnecessary detail information in the image by setting two thresholds σ1 and σ2, where σ1 is used to suppress smaller-scale textures, noise, and unnecessary details, while σ2 is used to suppress large-scale textures equivalent to significant structures, thereby achieving a better texture image smoothing effect. The formula of the texture suppression function is as follows:

[0011]

[0012] Where Δ is the output image gradient, σ1 and σ2 are thresholds, and exp(·) is an exponential function;

[0013] Then, the gradient suppressed by the texture suppression function is used as the input of the denominator of the penalty term, so that the penalty term can fully distinguish between texture and structure, impose different penalties on the two, and thus better preserve the semantic information of the image;

[0014] S2. Use the L1 norm with strong sparsity to constrain the penalty term. Using the L1 norm can achieve a more sparse effect. Although the sparse L1 norm is not differentiable, by introducing subgradients to convert the non-convex optimization problem into a convex optimization problem and using the alternating direction multiplier method for iterative solution, a better edge-preserving image smoothing algorithm can be achieved, avoiding the weak sparsity of the L2 norm and the inability to retain the semantic information of the image well.

[0015] S3, using L with strong sparsity p norm to constrain the penalty term, where 0<p<1, for L p In order to effectively solve the norm problem, the iterative reweighted L1 norm method is adopted to solve it, and the pre-processed conjugate gradient method is used to accelerate and improve the calculation efficiency, so as to achieve a more robust and sparse image smoothing effect.

[0016] Further, the step S2 comprises the following steps:

[0017] S21, the objective function of the edge-preserving image smoothing algorithm is defined as follows:

[0018]

[0019] Among them, S is the output image; I is the input image; λ is a balance parameter, which is used to adjust the relationship between the data term and the penalty term, and can directly affect the smoothing performance; ξ is a very small positive number to prevent division by zero; represents the L2 norm; p represents the index position; x and y represent the x and y directions of the image; D * is the sum of the absolute values ​​of the weighted gradients within the window; M *is the absolute value of the weighted gradient sum within the window; D * and M * The derivation formula is as follows:

[0020]

[0021] Among them, *∈{x,y} represents the x direction and the y direction; Ω(p) is the domain space centered at p with a window size of r×r, where r is represented by 2×k+1; g k (||pq||) is the Gaussian kernel function; k is the Gaussian kernel size; h is the texture suppression function at different scales; represents differential;

[0022] S22. Convert the objective function formula (2) into the L1 norm constraint penalty term:

[0023] First, solve the x direction, and the y direction is processed in the same way as the x direction; the penalty term of formula (2) is expanded as follows:

[0024]

[0025] Here the penalty term in the x direction is decomposed into a first-order term and a nonlinear weight part The processing in the y direction is the same. It is expressed as follows:

[0026]

[0027] Finally, rewrite the objective function and matrix it into the following formula:

[0028]

[0029] Among them, C x , C y is the backward difference matrix, and They are all diagonal matrices of size N×N, where N represents the total number of pixels in the image;

[0030] S23, using the alternating direction multiplier method to solve the objective function:

[0031] In order to effectively solve the above equation (7), let ψ x =U x C x S,ψ y =U y C y S, by constructing the augmented Lagrangian function, the objective function is as follows:

[0032]

[0033] For solving the final objective function (8), the objective function only contains the L1 norm and L2 norm and has no constraints, so the derivative can be directly minimized.

[0034] Further, the step S3 comprises the following steps:

[0035] S31, the penalty term of the objective function of the edge-preserving image smoothing algorithm is L p The norm constraint is defined as follows:

[0036]

[0037] S32, using iterative reweighting method to p The norm solution is transformed into solving the L1 norm problem. The weight of each round is obtained by dynamically weighting the value in the previous round of iteration, and the L1 norm is derived based on the L1 norm. p The solution method of the norm is to convert the non-convex function into a convex function through iterative minimization, and the weight W x pass To adjust, the following is the intermediate value in the x direction of the kth iteration:

[0038]

[0039] For the weight item After each iteration, the weights are updated in the next iteration. Here ξ is a small positive number to prevent division by zero, and the same is true for the y direction. Rewrite equation (10) and transform it into a matrix:

[0040]

[0041] in, and are the second weights. The solution of formula (11) is similar to the solution of formula (7) above, which will not be repeated here.

[0042] Compared with the prior art, the global sparse texture filtering method based on edge structure preservation provided by the present invention constrains the output image gradient by introducing a texture suppression function. First, two appropriate gradient thresholds are set to suppress large-scale textures equivalent to significant structures in the image, and to suppress unnecessary details, noise, and small-scale textures. Then, the weight part in the penalty term uses the gradient suppressed by the texture suppression function to ensure that overfitting or underfitting will not be caused while smoothing efficiently. Finally, the L1 norm with strong sparsity is used to constrain the penalty term, so that the proposed method model is as close to the true value as possible when minimizing, thereby improving the robustness of the algorithm. Therefore, the sparse edge-preserving image smoothing algorithm introduced with a texture suppression function proposed by the present invention achieves better smoothing performance while retaining the structural edge, and also has the following advantages:

[0043] 1. This paper proposes a new global optimization model framework, which can achieve high-quality smoothing results while effectively retaining edges, and adopts L1 norm constraint penalty terms. The model has a certain degree of sparsity.

[0044] 2. The present invention fully considers the existing problems of texture image smoothing, introduces a texture suppression function, suppresses multi-scale textures that are difficult to remove, and achieves a better smoothing effect.

[0045] 3. The present invention adopts L p The norm constraint penalty term achieves a more robust and sparse image smoothing effect, and adopts the idea of ​​iterative reweighting to convert L p The norm is transformed into the problem of solving the L1 norm. Compared with the penalty term of the L1 norm constraint, L p The penalty term model of the norm constraint has better sparsity. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a display of the results of extracting structures in different texture images provided by the present invention.

[0047] Figure 2 It is a texture suppression function provided by the present invention.

[0048] Figure 3 It is a comparison diagram of the smoothing effect of the present invention and other penalty items.

[0049] Figure 4 The global optimization algorithm (GSTF) provided by the present invention is based on L1 and L p (p = 0.4) and L p (p=0.8) Comparison of regularized smoothing results.

[0050] Figure 5 This is a diagram showing the influence of adjusting different parameters k on the smoothing result of the texture image in the present invention.

[0051] Figure 6 This is a diagram showing the influence of different parameters of the present invention on the smoothing result of a natural image.

[0052] Figure 7 3 is a diagram comparing the present invention with RTV by setting different λ and ρ (adjusting the smoothing strength).

[0053] Figure 8 This is a comparison diagram of the smoothing results of the present invention and the texture filter on large-scale texture images.

[0054] Fig. 9 This is a comparison diagram of the smoothing results of the present invention and the texture filter on a complex texture image.

[0055] Fig.10 This is a comparison diagram of the smoothing results of texture images of different scales between the present invention and the texture filter.

[0056] Fig.11 This is a comparison diagram of the visual effects of the present invention and various filters in edge extraction.

[0057] Fig.12 This is a comparison chart of the visual effects of the present invention and various filters in HDR tone mapping.

[0058] Fig.13 This is a comparison chart of the visual effects of the present invention and various filters in removing clip art compression artifacts.

[0059] Fig.14 This is a comparison diagram of the abstract visual effects of the present invention and various filters on images. DETAILED DESCRIPTION

[0060] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0061] Please refer to Figure 1 As shown, the edge-preserving image smoothing effect of a texture image is performed using a global sparse texture filtering method based on edge structure preservation provided by the present invention. The method of the present invention includes the following steps:

[0062] S1. Introduce a texture suppression function in the penalty term of the global optimization framework to constrain the gradient of the output image. The texture suppression function suppresses the texture, noise, unnecessary details and other information in the image by setting two thresholds σ1 and σ2, where σ1 is used to suppress smaller-scale textures, noise, and unnecessary details, while σ2 is used to suppress large-scale textures equivalent to significant structures, thereby achieving a better texture image smoothing effect. The formula of the texture suppression function is as follows:

[0063]

[0064] Where Δ is the output image gradient, σ1 and σ2 are thresholds, and exp(·) is an exponential function;

[0065] Here Δ is the gradient of the output image, and the effect of texture gradient suppression is as follows: Figure 2As shown, when processing textures of different scales, the edges are retained as much as possible and the textures between small structures and large structures are compressed, while unnecessary details, noise and small-scale textures are filtered out, so as to achieve a better effect of maintaining weak structures and small structure edges. The texture suppression function used in the present invention here is to suppress the large-scale textures that cannot be filtered out by the RTV model. This is based on the characteristics of the RTV model. The RTV model has a large response in the texture area and a small response in the structure area (that is, different weights are given to texture and structure). By processing the image gradient of the denominator, the large-scale texture gradient is suppressed or the small-scale texture, noise, and unnecessary detail gradient are set to 0, so that the response to these areas is greater, and then the smoothing performance of textures of different scales is better.

[0066] Then, the gradient suppressed by the texture suppression function is used as the input of the denominator of the penalty term, so that the penalty term can fully distinguish between texture and structure, impose different penalties on the two, and thus better retain the semantic information of the image.

[0067] S2. Since the L2 norm has weak sparsity and cannot retain the semantic information of the image well, the present invention adopts the L1 norm with strong sparsity to constrain the penalty term. The use of the L1 norm can achieve a more sparse effect. Although the sparse L1 norm is not differentiable, the non-convex optimization problem is converted into a convex optimization problem by introducing subgradients, and the alternating direction multiplier method (ADMM) is used for iterative solution, which can achieve a better edge-preserving image smoothing algorithm and avoid the weak sparsity of the L2 norm and the inability to retain the semantic information of the image well.

[0068] As a specific embodiment, step S2 includes the following steps:

[0069] S21, the objective function of the edge-preserving image smoothing algorithm is defined as follows:

[0070]

[0071] Among them, S is the output image; I is the input image; λ is a balance parameter, which is used to adjust the relationship between the data term and the penalty term, and can directly affect the smoothing performance; ξ is a very small positive number to prevent division by zero; represents the L2 norm; p represents the index position; x and y represent the x and y directions of the image; D * is the sum of the absolute values ​​of the weighted gradients within the window; M * is the absolute value of the weighted gradient sum within the window; D * and M * The derivation formula is as follows:

[0072]

[0073] Among them, *∈{x,y} represents the x direction and the y direction; Ω(p) is the domain space centered at p with a window size of r×r, where r is represented by 2×k+1; g k (||pq||) is the Gaussian kernel function; k is the Gaussian kernel size; h is the texture suppression function at different scales; It means to find the differential.

[0074] Here, for the characteristics of texture and structure, the D * The size is comparable. Since the gradient of the structure in the window is almost in one direction and the gradient of the texture in the window is divergent in all directions, the M of the structure area * Much larger than the texture area M * Based on this feature, the present invention combines the two to give different penalties to texture and structure to the greatest extent, that is, D * As a molecule, M * As the denominator, we have formula (2).

[0075] like Figure 3 As shown in Figure 2, the proposed GSTF1 is compared with the L1-RTV and RTV methods in two types of figures. Figure 3 In the first row, we can intuitively see that L1-RTV retains good details, but it also exposes obvious deficiencies in processing small-scale textures, such as Figure 3 (c) As shown in Figure 2, GSTF1 has made a good trade-off between retaining weak and small structure edges and removing small textures. It can not only filter out textures and noise, but also identify weak and small structure edges with less destructiveness. Figure 3 (d) As shown; RTV effectively smoothes while destroying weak structures, such as Figure 3 (b) is shown. Figure 3 The second line of Figure 3 As shown in (e), there is a lot of noise mixed with the structure. When the RTV method is effectively smoothed, the edge structure is not very clear and even has artifacts, such as Figure 3 (f) As shown in the figure, although the RTV model with L1 norm constraint solves the problem of edge existence, the artifacts are not filtered out, as shown in Figure 3 (g) As shown; On the other hand, the proposed GSTF1 solves the existing edge and artifact problems, such as Figure 3 (h) In general, the GSTF1 method is useful in preserving small structures and has a good suppression effect on unnecessary details, textures, noise, and artifacts.

[0076] In order to make the solution sparse, the L1 norm is used to constrain the regularization term, and the alternating direction multiplier method (ADMM) is used to iteratively solve the proposed algorithm. The global optimization model based on GSTF1 proposed in the present invention consists of a quadratic term and a linear term, which is finally converted into solving a series of linear equations by minimizing the objective function.

[0077] S22. Convert the objective function formula (2) into the L1 norm constraint penalty term:

[0078] The present invention first solves the x direction, and the y direction is processed in the same way as the x direction; the penalty term of formula (2) is expanded as follows:

[0079]

[0080] Here the penalty term in the x direction can be decomposed into a first-order term and a nonlinear weight part The processing in the y direction is the same. It is expressed as follows:

[0081]

[0082] Finally, the objective function is rewritten as follows:

[0083]

[0084] Here, S and I are both N×1 vectors, where N represents the total number of pixels in the image. The above formula is then written in matrix form as follows:

[0085]

[0086] Among them, C x , C y is the backward difference matrix, and They are all diagonal matrices of size N×N.

[0087] S23, using the alternating direction multiplier method to solve the objective function:

[0088] In order to effectively solve the above equation (7), let ψ x =U x C x S,ψ y =U y C y S, by constructing the augmented Lagrangian function, the objective function is as follows:

[0089]

[0090] For solving the final objective function (8), the objective function only contains the L1 norm and L2 norm and has no constraints, so we can directly minimize the derivative. Specifically, here we respectively calculate S and ψ x , y , η x , η y Solve the problem. The solution process is as follows:

[0091] 1) First, fix ψ x , y , η x , η y , we can solve for S:

[0092]

[0093] The above formula is solved by iterative minimization, and the iterative formula is as follows:

[0094]

[0095] Where E is the identity matrix, where Corresponding Here L t is a five-point non-homogeneous sparse Laplace matrix. The fast preconditioned conjugate gradient method (PCG) can be used to solve the linear equations and finally obtain the output image S t+1 .

[0096] 2) Secondly, fix S and η x , η y , respectively solve for ψ x , y :

[0097]

[0098] The shrink operation here can be written as shrink(θ,τ)=sign(θ)max(|θ|-τ,0), where θ is expressed as τ is expressed as sign(·) and max(·) are the sign function and the maximum value function respectively.

[0099] 3) Finally, fix S and ψ x , y , respectively solve for η x , η y :

[0100]

[0101] Finally, according to the above formula derivation, the image smoothing algorithm proposed in the present invention can be described as the following algorithm:

[0102] Input: input image I, balance parameters λ and ρ, convergence error ε

[0103] Output: Filtered image S

[0104] 1: Initialization: S {0} =I;

[0105] 2: while do

[0106] 3: Compress the gradient of the input image I;

[0107] 4: Calculate formula (6) to obtain

[0108] 5: Calculate formula (12) to obtain S {t+1} ;

[0109] 6: Calculate formula (13) to obtain

[0110] 7: Calculate formula (14) to obtain

[0111] 8: Calculate formula (15) to obtain

[0112] 9: Calculate formula (16) to obtain

[0113] 10: end while

[0114] 11: return S {t}

[0115] S3, using L with strong sparsity p norm to constrain the penalty term, where 0<p<1, for L p In order to effectively solve the norm problem, the iterative reweighted L1 norm method is adopted to solve it, and the pre-processed conjugate gradient method is used to accelerate and improve the calculation efficiency, so as to achieve a more robust and sparse image smoothing effect.

[0116] As a specific embodiment, step S3 includes the following steps:

[0117] S31, the penalty term of the objective function of the edge-preserving image smoothing algorithm is L p The norm constraint is defined as follows:

[0118]

[0119] S32, using iterative reweighting method to p The norm solution is transformed into solving the L1 norm problem.p The solution of the norm is a non-convex optimization problem. Therefore, solving the objective function should be transformed into a simpler problem that can be solved currently. The present invention will adopt an iterative reweighting method, and the weight of each round is dynamically weighted by the value in the previous round of iteration. And based on the L1 norm, the solution method of the L p norm is derived. By iterative minimization, the non-convex function is transformed into a convex function, and the weight W x is adjusted through . The following is the intermediate value in the x direction at the k-th iteration:

[0120]

[0121] For the weight term After each iteration ends, the weight is updated in the next round of iteration Here, ξ is a very small positive number to prevent division by zero, and the same applies to the processing in the y direction; rewrite Equation (10) and matrixify it as:

[0122]

[0123] Among them, and are both the second-term weights, and they are also diagonal matrices of size N×N. The solution of Equation (11) is similar to the solution process of Equation (7) above, and will not be elaborated here. Minimizing the objective function is equivalent to iteratively solving a series of linear equations, and pay attention to calculating the weight W during iteration. To accelerate the solution of the linear equations, the preconditioned conjugate gradient method (PCG) is used to optimize the objective function here. The algorithm of the present invention is easy to solve, and finally a good running efficiency is obtained.

[0124] Figure 4 Shows the smoothing results of the algorithm of the present invention based on L1 and L p (p = 0.4) and L p (p = 0.8) regularization. It can be clearly seen from the figure that the image smoothed by the GSTF model using L1 regularization is more flattened, while the GSTF model using L p (p = 0.8) constraint does not cause the small structure edges and weak structures to be damaged while effectively smoothing, which is better than the GSTF models constrained by L1 and L p (p = 0.4) norms. Therefore, in order to achieve a more accurate edge-preserving effect, for the GSTF model constrained by L p (0 < p < 1), the present invention will adopt p = 0.8 for subsequent experimental comparison.

[0125] The method proposed in the present invention mainly adjusts two parameters, the Gaussian kernel size k and the Lagrangian penalty parameter ρ. The larger the Gaussian kernel size k is set, the smoother the image is, and the smoothed image almost only retains the main structure, which will cause the small structures that need to be retained to be smoothed out. In actual situations, the Gaussian kernel size k is generally controlled between [1, 3]. The Gaussian kernel size for texture images is relatively large, while the opposite is true for non-texture images. The Lagrangian penalty parameter ρ is an important parameter for controlling the smoothing strength of the method proposed in the present invention. The larger ρ is, the greater the smoothing strength is. On the contrary, the retention of details is more obvious. Generally, ρ is set to [3e 2 , 15e 2 ]among.

[0126] In addition, the method of the present invention also needs to adjust the compression thresholds σ1 and σ2 of the output image gradient S. Here, σ1 and σ2 play a key role. It suppresses large-scale textures and effectively suppresses some noise, textures, unnecessary details, etc., which can directly affect the final smoothing effect. Usually, σ1 and σ2 are set to multiples of k, where σ1 is set to 0.5 times of k and σ2 is set to between 2-5 times of k. Figure 5 By adjusting the Gaussian kernel size k, the smoothing effect of different Gaussian kernel sizes k on texture images is demonstrated. Here, the convolution kernel sizes are set to k = 1.5, k = 2.5, k = 3, and k = 3.5. It can be seen that when other parameters are fixed, the larger the k, the better the smoothing effect. On the contrary, the smaller the k, the more details and textures are retained.

[0127] Figure 6 The effect of adjusting the Gaussian kernel size k and the Lagrangian penalty parameter ρ on the smoothing results of natural images is shown. Obviously, when k is fixed to 1, a larger ρ will make the smoothing stronger. When ρ is fixed to 300, it can be found that the effect of adjusting k is not particularly obvious, indicating that controlling ρ is more effective. Of course, the best smoothing effect will be obtained by coordinating the two parameters.

[0128] Since the present invention adopts the ADMM solution method, the proposed method is mainly divided into three stages in each iteration. The first is to solve the linear system S, which includes gradient compression, weight calculation and linear system solution. The gradient compression includes two one-dimensional linear operations with a complexity of O(N), where N is the total number of pixels in the image, and the weight calculation involves two convolution operations with a complexity of O(k 2N), where the Gaussian kernel separable operation is used to decompose it into two one-dimensional filters, thereby obtaining a method with a complexity of O(kN). Finally, the linear system of the five-point non-homogeneous Laplace matrix is ​​solved. Direct inversion is very time-consuming, so several acceleration methods are considered, such as preconditioned conjugate gradient method (PCG), LU decomposition, etc., which can achieve linear speed. The present invention will use PCG acceleration, which can achieve better smoothing effect in a very short time; the second step is to solve the constrained L1 norm. For the non-convex L1 norm, the subgradient is introduced to obtain its closed-form solution. Since it is solved element by element, the complexity is also linear O(N); the third step is to solve the Lagrange dual multiplier, and the gradient ascent method is used to approximate the solution to the problem, with a complexity of O(N).

[0129] All algorithms were tested on a system with i5-9300H CPU, 16GB memory and MATLAB 2018b. The present invention uses three color images with different resolutions for evaluation and compares them with seven algorithms, as shown in Table 1. The method proposed in the present invention is slightly slower than L0, RTV and ROG, but performs better in processing the details of the smoothing result. The algorithm of the present invention is faster than G-smooth, SDF and SSF.

[0130] Table 1 Comparison of running time of different algorithms on three-channel color images (measured in seconds)

[0131]

[0132] During the algorithm execution, both RTV and GSTF1 methods solve the linear system to obtain the final smoothing result, and both are globally optimized filters. Figure 7 As shown in Figure 2, the results of RTV and GSTF1 under different smoothing strengths are compared. It is obvious that with the increase of smoothing strength, the GSTF1 method causes less damage to small edges and weak structures, while RTV blurs the edges of small structures and weak structures, indicating that the algorithm proposed in this invention performs better in protecting edges and smoothing details, so the proposed GSTF1 has a better edge structure preservation effect.

[0133] The proposed edge-preserving smoothing algorithm is evaluated on different image processing tasks (image smoothing, HDR mapping, edge extraction, image abstraction, artifact removal). A large number of experiments are conducted on the NKS dataset, the SPS dataset, and the dataset provided by Xu et al. 0.8The proposed algorithm is compared with the best algorithms currently available: local filters (GF, BTF, RGF, TF), global optimization algorithms (WLS, L0, RTV, FGS, SDF, SGWLS, BLF-LS, G-smooth, L1-R, SSF, BTV), and deep learning methods (DEAF, ResNet, DeepFsps). The comparison is mainly aimed at the image smoothing quality. The present invention also uses quantitative indicators PSNR and SSIM for comparison, and verifies the effectiveness of the proposed algorithm through a series of experiments.

[0134] like Figure 8 As shown in (a), texture is everywhere in the image. The BTF method blurs the edges of the structure while smoothing the texture, resulting in artifacts in the smoothed image, such as Figure 8 (b) As shown. However, the RGF method is difficult to filter out large-scale textures, such as Figure 8 (c) TF and G-smooth filter out obvious textures, but there are some obvious texture lines, such as Figure 8 (d) and (e) are shown. RTV, BTV and DeepFsps methods can smooth the texture well, but the small structures and the edges of these structures are destroyed and become very blurred, such as Figure 8 (f), (g), and (h). In contrast, the method GSTF1 of the present invention and the proposed GSTF 0.8 This method can not only filter out large-scale textures and texture lines, but also retain these structures and structural edges, achieving effective smoothing. Figure 8 As shown in (i) and (j).

[0135] like Fig. 9 As shown in the woven texture in (a), the small pattern structures in the woven fabric are of similar size to the block texture. How to remove the texture while retaining the structure edges and small structure wicks is a tricky problem. The BTF and G-smooth methods have a good texture smoothing effect on the woven background, but they destroy the small structure wicks and there are artifacts, such as Fig. 9 (b) and (f) are shown. However, L0, SGWLS, FGS and DEAF do not fully consider the relationship between texture and structure. Although they preserve small structures very well, their smoothing effect is very poor, as shown in Fig. 9 (c), (d), (e), and (i). RTV and BTV have good texture removal effects, but they seriously damage the small structure wick and its edges, such as Fig. 9 (g) and (h). The deep learning method DeeFsps fully smoothes the texture, but also penalizes small structural gradients, resulting in over-smoothing, such as Fig. 9It is obvious that the proposed GSTF1 method fully considers the trade-off between texture and structure, and retains these small structures such as Fig. 9 (k) but also weakens Fig. 9 (k) The wick in the lower right corner. On the contrary, the proposed GSTF1 takes this problem into full consideration. Because it has better sparsity than the method using the L1 norm, it achieves a better smoothing effect on the texture image without destroying the edges of small structures, such as Fig. 9 (l) as shown.

[0136] like Fig.10 The texture image with various complex scales shown in (a) has small-scale textures and structures in the marked box, which is a huge challenge for the algorithm. The RGF method has artifacts after filtering, such as Fig.10 (b) Although the SDF, RTV and BTV methods remove large-scale textures, they all destroy the edges of small structures to a certain extent, as shown in Fig.10 (c), (d), and (e). The deep learning method DeepFsps does not have scale perception. Although the smoothing effect is very good, the small structures are severely damaged and over-smoothing occurs. In contrast, the proposed method of the present invention, GSTF1, retains the texture while filtering it out. Fig.10 (g) The small structure in the method box, but the edge of the small structure is slightly damaged. 0.8 Methods are more sparse and can solve this problem, such as Fig.10 As shown in (h), the superiority of its edge-preserving smoothing algorithm is demonstrated.

[0137] The method of the present invention has good processing performance for multi-scale texture images and avoids the phenomenon of under-smoothing. Therefore, the method can be said to be very effective in processing more complex tasks. Table 2 below shows the quantitative results of each method on the NKS and SPS datasets. It is obvious that the GSTF1 algorithm and GSTF proposed in the present invention are better than the GSTF 0.8 The algorithm has certain advantages in comparison of PSNR and SSIM indicators.

[0138] Table 2 Quantitative analysis results on NKS and SPS datasets

[0139]

[0140] Edge extraction is an application of image smoothing. For natural images with a lot of noise or details, it is difficult to obtain satisfactory results by directly performing edge extraction. After smoothing the image, unnecessary textures, details or noise are filtered out, and the main structural edges in the image are retained before edge extraction. The present invention uses the canny operator for edge detection to obtain a more ideal result.

[0141] Fig.11 The visual effect of edge extraction of the algorithm is shown. Fig.11 (a) contains both texture information and structural information. Visually, RGF, L0, and SSF have poor edge extraction effects, and these methods retain a large amount of texture or details. BTF, G-smooth, RTV, and BTV have better overall edge extraction effects, but Fig.11 The edges of the small structures in the green boxes in the lower half of (b), (e), (f), and (g) are all damaged to a certain extent. In contrast, the edge of GSTF1 in the method proposed in the present invention has some artifacts, resulting in inaccurate edges, such as Fig.11 (i) GSTF 0.8 This problem is solved well, and no damage is caused when extracting the edge, such as Fig.11 Therefore, the method proposed in the present invention is effective.

[0142] HDR tone mapping is another application of retaining edges and increasing brightness. HDR aims to compress the intensity of high dynamic range images and keep the original details and colors unchanged to form a dynamic image. It is divided into a base layer and details. The base layer is a smooth output of the input HDR image. The base layer is then nonlinearly mapped to a low dynamic range and recombined with the detail layer to finally get a tone mapped image.

[0143] Fig.12 The comparison results of HDR tone mapping on various algorithms are shown. It is obvious that the LDR image generated by WLS is not the expected result and is darker overall. However, artifacts appear in L0, resulting in a very blurry LDR image. The LDR image generated by G-smooth has a halo, and the details are not obvious. The algorithm GSTF1 proposed in the present invention overcomes the above shortcomings, but the processing of detail edges in the GSTF of the present invention is 0.8 The algorithm is better. The present invention uses TMQI to quantitatively evaluate the image quality of HDR tone mapping, using 15 HDR images provided by Yeganeh et al. The specific quantitative results are shown in Table 3 below. The image smoothing algorithm of the present invention has certain advantages over other methods.

[0144] Table 3 Quantitative analysis results of each method on the dataset provided by Yeganeh

[0145]

[0146] Image compression artifact removal aims to remove artifacts that occur during the compression process. This type of image is piecewise constant and has sharp edges. When it is compressed in JPEG format, it will cause artifacts near the edges, such as Fig.13 As shown in (a), the texture filtering algorithm is more suitable for this application. It can smooth out unnecessary details while retaining the edge structure. Therefore, the proposed algorithm can be used to reduce the artifacts caused by image compression. The L0 algorithm can eliminate most compression artifacts, but it directly smoothes out weak structures and their edges, such as Fig.13 (b) When dealing with clipart compression artifacts, the SDF and BTV algorithms blur the edges instead of sharpening them. Fig.13 (c) and (e) are shown. The RTV algorithm is used to remove compression artifacts and achieves some results, but it causes a certain degree of damage to the edges of the structure, such as Fig.13 (d) As shown in the figure. The G-smooth method proposed by Liu et al. can sharpen the edges, but introduces artifacts to small structure areas, such as Fig.13 (f) As shown in the blue box. The method GSTF1 and GSTF proposed in this invention 0.8 The best performance is obtained for both edge sharpening and preservation of small structures, such as Fig.13 As shown in (g) and (h).

[0147] Image abstraction is one of the applications of image smoothing. Image abstraction aims to remove unnecessary details in the image after filtering, and abstract the edges and structures to form a cartoon-like picture. In order to better reflect the information that the image wants to convey, the result of image abstraction is as follows: Fig.14 shown.

[0148] The proposed method is extensively evaluated and performs well on both textured and non-textured images, with better edge preservation than existing filtering algorithms.

[0149] Compared with the prior art, the global sparse texture filtering method based on edge structure preservation provided by the present invention constrains the output image gradient by introducing a texture suppression function. First, two appropriate gradient thresholds are set to suppress large-scale textures equivalent to significant structures in the image, and to suppress unnecessary details, noise, and small-scale textures. Then, the weight part in the penalty term uses the gradient suppressed by the texture suppression function to ensure that overfitting or underfitting will not be caused while smoothing efficiently. Finally, the L1 norm with strong sparsity is used to constrain the penalty term, so that the proposed method model is as close to the true value as possible when minimizing, thereby improving the robustness of the algorithm. Therefore, the sparse edge-preserving image smoothing algorithm introduced with a texture suppression function proposed by the present invention achieves better smoothing performance while retaining the structural edge, and also has the following advantages:

[0150] 1. This paper proposes a new global optimization model framework, which can achieve high-quality smoothing results while effectively retaining edges, and adopts L1 norm constraint penalty terms. The model has a certain degree of sparsity.

[0151] 2. The present invention fully considers the existing problems of texture image smoothing, introduces a texture suppression function, suppresses multi-scale textures that are difficult to remove, and achieves a better smoothing effect.

[0152] 3. The present invention adopts L p The norm constraint penalty term achieves a more robust and sparse image smoothing effect, and adopts the idea of ​​iterative reweighting to convert L p The norm is transformed into the problem of solving the L1 norm. Compared with the penalty term of the L1 norm constraint, L p The penalty term model of the norm constraint has better sparsity.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A global sparse texture filtering method based on edge structure preservation, characterized in that: The method comprises the following steps: S1. Introduce the texture suppression function in the penalty term of the global optimization framework to constrain the gradient of the output image. The texture suppression function suppresses the texture, noise, and unnecessary detail information in the image by setting two thresholds σ1 and σ2, where σ1 is used to suppress smaller-scale textures, noise, and unnecessary details, while σ2 is used to suppress large-scale textures equivalent to significant structures, thereby achieving a better texture image smoothing effect. The formula of the texture suppression function is as follows: Where Δ is the output image gradient, σ1 and σ2 are thresholds, and exp(·) is an exponential function; Then, the gradient suppressed by the texture suppression function is used as the input of the denominator of the penalty term, so that the penalty term can fully distinguish between texture and structure, impose different penalties on the two, and thus better preserve the semantic information of the image; S2. Use the L1 norm with strong sparsity to constrain the penalty term. Using the L1 norm can achieve a more sparse effect. Although the sparse L1 norm is not differentiable, by introducing subgradients to convert the non-convex optimization problem into a convex optimization problem and using the alternating direction multiplier method for iterative solution, a better edge-preserving image smoothing algorithm can be achieved, avoiding the weak sparsity of the L2 norm and the inability to retain the semantic information of the image well. S3, using L with strong sparsity p norm to constrain the penalty term, where 0<p<1, for L p In order to effectively solve the norm problem, the iterative reweighted L1 norm method is adopted to solve it, and the pre-processed conjugate gradient method is used to accelerate and improve the calculation efficiency, so as to achieve a more robust and sparse image smoothing effect.

2. The global sparse texture filtering method based on edge structure preservation according to claim 1, characterized in that: The step S2 comprises the following steps: S21, the objective function of the edge-preserving image smoothing algorithm is defined as follows: Among them, S is the output image; I is the input image; λ is a balance parameter, which is used to adjust the relationship between the data term and the penalty term, and can directly affect the smoothing performance; ξ is a very small positive number to prevent division by zero; represents the L2 norm; p represents the index position; x and y represent the x and y directions of the image; D * is the sum of the absolute values ​​of the weighted gradients within the window; M * is the absolute value of the weighted gradient sum within the window; D * and M * The derivation formula is as follows: Among them, *∈{x,y} represents the x direction and the y direction; Ω(p) is the domain space centered at p with a window size of r×r, where r is represented by 2×k+1; g k (||pq||) is the Gaussian kernel function; k is the Gaussian kernel size; h is the texture suppression function at different scales; represents differential; S22. Convert the objective function formula (2) into the L1 norm constraint penalty term: First, solve the x direction, and the y direction is processed in the same way as the x direction; the penalty term of formula (2) is expanded as follows: Here the penalty term in the x direction is decomposed into a first-order term and a nonlinear weight part The processing in the y direction is the same. It is expressed as follows: Finally, rewrite the objective function and matrix it into the following formula: Among them, C x , C y is the backward difference matrix, and They are all diagonal matrices of size N×N, where N represents the total number of pixels in the image; S23, using the alternating direction multiplier method to solve the objective function: In order to effectively solve the above equation (7), let ψ x =U x C x S,ψ y =U y C y S, by constructing the augmented Lagrangian function, the objective function is as follows: For solving the final objective function (8), the objective function only contains the L1 norm and L2 norm and has no constraints, so the derivative can be directly minimized.

3. The global sparse texture filtering method based on edge structure preservation according to claim 2, characterized in that: The step S3 comprises the following steps: S31, the penalty term of the objective function of the edge-preserving image smoothing algorithm is L p The norm constraint is defined as follows: S32, using iterative reweighting method to p The norm solution is transformed into solving the L1 norm problem. The weight of each round is obtained by dynamically weighting the value in the previous round of iteration, and the L1 norm is derived based on the L1 norm. p The solution method of the norm is to convert the non-convex function into a convex function through iterative minimization, and the weight W x pass To adjust, the following is the intermediate value in the x direction of the kth iteration: For the weight item After each iteration, the weights are updated in the next iteration. Here ξ is a small positive number to prevent division by zero, and the same is true for the y direction. Rewrite equation (10) and transform it into a matrix: in, and are the second weights. The solution of formula (11) is similar to the solution of formula (7) above, which will not be repeated here.

Images