Image deblurring method based on enhanced total variation

Abstract

The present application relates to a kind of image deblurring methods based on enhanced total variation, comprising: based on the difference of weight anisotropy and isotropy total variation, then coupling l0 norm, obtain enhanced total variation method;Using the l0 norm constraint of image gradient, the blurred image B of input is preprocessed;Further sparse processing is carried out to the blurred image after pre-processing;Image deblurring model is constructed;Estimate latent image, obtain intermediate latent image I;Blurred kernel estimation is carried out, and the final clear image is obtained using non-blind deconvolution method, to obtain image deblurring result.The present application does not need a large amount of computing cost, it is time-saving in computing cost;More robust, applicable to different types of blurred image;More suitable for sparse representation in image deblurring, can remove the adverse structure in latent image in depth, deblurring effect is more significant, the present application model is simple, the thought is clear, in execution calculation efficiency is high, model generalization ability is strong, and the recovery effect is more visualized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image deblurring method based on enhanced total variation. Background Technology

[0002] With the significant development of economy and technology, artificial intelligence has been integrated into people's lives, and clear images are the carriers of important information. With continuous technological improvements, deblurring techniques have reached a certain level of maturity, mainly divided into prior-based and edge-based methods. Prior-based deblurring techniques primarily rely on the feature differences between clear and blurred images. For example, the dark channel prior utilizes the feature information that dark pixels in a clear image are sparser than those in a blurred image. However, since it extracts dark pixels from locally overlapping blocks, the computation is very time-consuming. Similar methods include extreme channel priors and local maximum gradient priors. This prior method has a certain drawback: it will fail when the image's feature information is not obvious. Edge-based deblurring methods mainly use salient edges to estimate the blur kernel. For example, the classic salient edge method uses different filtering techniques to obtain salient edges, which has high computational efficiency. However, once the image is severely blurred, it becomes difficult to extract salient edges, and the method will fail. However, based on the inherent characteristic that natural images are sparse, simply using existing total variation techniques to process blurred images only considers the shallow sparsity of the image, without going into depth to remove undesirable structures or to provide a sparse representation suitable for deblurring techniques.

[0003] Based on the above analysis, existing deblurring techniques either suffer from high computational costs and algorithmic complexity due to their reliance on prior feature extraction, or they lack robustness to significant image edges, failing to capture these edges in severely blurred images and thus rendering the method ineffective. Others only superficially consider the sparsity of the image without providing a suitable sparse expression to remove unfavorable structures, resulting in unsatisfactory deblurring results. Addressing these limitations, and given current hardware conditions, designing an image deblurring method that is adaptable to various scenarios, computationally efficient, and produces good restoration results has become an urgent technical problem to be solved. Summary of the Invention

[0004] To address the problem of high algorithm complexity and inability to capture significant edges in severely blurred images, the present invention aims to provide an image deblurring method based on enhanced total variation that is adaptable to various scenarios, computationally efficient, provides good restoration results, and achieves more significant deblurring effects.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an image deblurring method based on enhanced total variation, the method comprising the following sequential steps:

[0006] (1) Based on the difference between the total variation of weights with different anisotropy and different isotropy, the l0 norm is coupled to obtain the enhanced total variation method;

[0007] (2) Using the l0 norm constraint of the image gradient, the input blurred image B is preprocessed to coarsely remove undesirable details and obtain the preprocessed blurred image.

[0008] (3) The enhanced total variation method is used to further sparse the preprocessed blurred image, and the harmful details of the significant interference edges are removed in a fine manner to obtain the blurred image after sparse processing.

[0009] (4) Construct an image deblurring model;

[0010] (5) Based on the image deblurring model, latent image estimation is performed on the sparsely processed blurred image to obtain the intermediate latent image I;

[0011] (6) Estimate the blur kernel based on the image deblurring model and the intermediate latent image I, and return to step (5) to perform n alternating iterations to obtain the blur kernel. ;

[0012] (7) Based on fuzzy kernel The final clear image is obtained by using a non-blind deconvolution method, resulting in the image deblurring result.

[0013] Step (1) specifically refers to the following formula based on the total variation of anisotropy:

[0014]

[0015] Where z represents the input; D represents the differential operator, D x D y Let (i, j) represent the differences in the x and y directions, respectively; (i, j) represents the position. ; The 1-norm represents the sum of the absolute values ​​of the elements;

[0016] The formula for total variation based on isotropy is as follows:

[0017]

[0018] The expression based on the l0 norm is as follows:

[0019]

[0020] In the formula, The zero norm represents the number of non-zero elements;

[0021] The expression for the enhanced total variational method is as follows:

[0022] .

[0023] Step (2) specifically refers to: using the gradient l0 norm to remove undesirable details in the intermediate latent image I, the formula of which is as follows:

[0024]

[0025] in, The gradient l0 norm is sparse.

[0026] Step (3) specifically refers to: using the enhanced total variation method to further sparse the preprocessed blurred image, resulting in:

[0027]

[0028] make:

[0029]

[0030] but:

[0031]

[0032] In the formula, D x D y These represent the differences in the x and y directions, respectively; The 1-norm represents the sum of the absolute values ​​of the elements; The zero norm represents the number of non-zero elements.

[0033] In step (4), the expression for the image deblurring model is as follows:

[0034]

[0035]

[0036] Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K. Both represent penalty parameters. For data fidelity items, To preserve the details, To enhance the total variation term, B is the fuzzy kernel term; D is the input fuzzy image. x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm.

[0037] Step (5) specifically refers to: introducing auxiliary variables. In the 0 norm , replace In The image deblurring model is then rewritten as:

[0038]

[0039]

[0040] Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K, where B is the input blurred image, and D is... x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm. All represent penalty parameters; All parameters are positive. The above equation is solved through the following alternating iterations:

[0041] Solving for v: Keeping I and u constant, solve for v using the convexity algorithm.

[0042]

[0043]

[0044] make Then the objective function of the above equation can be rewritten as:

[0045]

[0046] The nth iteration obtained by the convexity algorithm is:

[0047]

[0048]

[0049] in, For a positive weight parameter, a new replacement variable is introduced using the split Bregman method. , and Lagrange multiplier b x b y ,Will replace , replace Then subproblem v n+1 Reconstructed as:

[0050]

[0051] in, It is a positive constant. The above expression can be divided into the following sub-problems:

[0052]

[0053]

[0054]

[0055] Regarding the solution for the e-norm in the above formula, the solution is:

[0056]

[0057] in, , ;

[0058] Solving for u: Introducing auxiliary variables In the 0 norm ,but:

[0059]

[0060]

[0061] Solution I: Using the u and v obtained above, the solution for image I is as follows:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] in, Represents the Fast Fourier Transform. Indicates inverse transformation, It is a conjugate gradient operation. This represents a two-dimensional gradient operation, and Indicates two directions: horizontal and vertical. It also represents both horizontal and vertical directions.

[0068] Step (6) specifically includes the following sequential steps:

[0069] (6a) Given the fuzzy kernel size Furthermore, the size of the given blur kernel should be smaller than the size of the image;

[0070] (6b) Next, initialize the blur kernel and determine the number of updated layers based on the blur kernel size, continuously performing upsampling, wherein the number of layers is determined by... Repeat the process until the size of the coarsest layer reaches 7×7;

[0071] (6c) The input fuzzy image B and the intermediate latent image I are calculated using the following fuzzy kernel estimation formula:

[0072]

[0073] In the formula, K represents the fuzzy kernel. Indicates the penalty parameter;

[0074] Using Fast Fourier Transform:

[0075]

[0076] in, and These represent the Fast Fourier Transform and the Inverse Fourier Transform, respectively.

[0077] Step (7) specifically includes the following sequential steps:

[0078] (7a) First, the latent image I1 is estimated using a method with a Laplace prior;

[0079] (7b) Next, estimate the latent image I0:

[0080]

[0081] In the formula, K represents the blur kernel, and B is the input blurred image. The zero norm represents the number of non-zero elements; It is a 2-norm. Indicates the penalty parameter;

[0082] (7c) Then calculate the difference plot between the two estimated plots I1 and I0, and apply a bilateral filter to it.

[0083] Finally, the filtered difference map is subtracted from I1 to remove artifacts.

[0084] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, compared with the prior art, the present invention is simple to operate, has low computational cost, strong model generalization ability, and is applicable to various scenarios. It focuses on the essential characteristic of the inherent sparsity of natural images and provides a new enhanced total variation method to further remove unfavorable structures in the latent image, making it more adaptable to the sparse representation of natural images and restoring a clear image from a blurred image. Second, compared with existing prior-based deblurring methods, the present invention does not require a large amount of computational cost, thus saving time in terms of computational cost. Compared with existing edge-based methods, the present invention is more robust and applicable to different types of blurred images. Compared with existing methods based on the sparsity of natural images, the present invention is more suitable for sparse representation in image deblurring, can deeply remove unfavorable structures in the latent image, and has a more significant deblurring effect. The model of the present invention is simple, the idea is clear, and the execution is computationally efficient, the model has strong generalization ability, and the restoration effect is more visualized. Attached Figure Description

[0085] Figure 1 This is a flowchart of the method of the present invention;

[0086] Figure 2 A comparison of the restoration effects of this invention with existing prior-based deblurring techniques;

[0087] Figure 3 A comparison of the restoration effects of this invention with existing edge-based deblurring techniques;

[0088] Figure 4 The present invention exhibits more robust deblurring performance in different scenarios. Detailed Implementation

[0089] like Figure 1 As shown, an image deblurring method based on enhanced total variation includes the following sequential steps:

[0090] (1) Based on the difference between the total variation of weights with different anisotropy and different isotropy, the l0 norm is coupled to obtain the enhanced total variation method;

[0091] (2) Using the l0 norm constraint of the image gradient, the input blurred image B is preprocessed to coarsely remove undesirable details and obtain the preprocessed blurred image.

[0092] (3) The enhanced total variation method is used to further sparse the preprocessed blurred image, and the harmful details of the significant interference edges are removed in a fine manner to obtain the blurred image after sparse processing.

[0093] (4) Construct an image deblurring model;

[0094] (5) Based on the image deblurring model, latent image estimation is performed on the sparsely processed blurred image to obtain the intermediate latent image I;

[0095] (6) Estimate the blur kernel based on the image deblurring model and the intermediate latent image I, and return to step (5) to perform n alternating iterations to obtain the blur kernel. ;

[0096] (7) Based on fuzzy kernel The final clear image is obtained by using a non-blind deconvolution method, resulting in the image deblurring result.

[0097] Step (1) specifically refers to the following formula based on the total variation of anisotropy:

[0098]

[0099] Where z represents the input; D represents the differential operator, D x D y Let (i, j) represent the differences in the x and y directions, respectively; (i, j) represents the position. ; The 1-norm represents the sum of the absolute values ​​of the elements;

[0100] The formula for total variation based on isotropy is as follows:

[0101]

[0102] The expression based on the l0 norm is as follows:

[0103]

[0104] In the formula, The zero norm represents the number of non-zero elements;

[0105] The expression for the enhanced total variational method is as follows:

[0106] .

[0107] Step (2) specifically refers to: using the gradient l0 norm to remove undesirable details in the intermediate latent image I, the formula of which is as follows:

[0108]

[0109] in, The gradient l0 norm is sparse.

[0110] Step (3) specifically refers to: using the enhanced total variation method to further sparse the preprocessed blurred image, resulting in:

[0111]

[0112] make:

[0113]

[0114] but:

[0115]

[0116] In the formula, D x D y These represent the differences in the x and y directions, respectively; The 1-norm represents the sum of the absolute values ​​of the elements; The zero norm represents the number of non-zero elements.

[0117] In step (4), the expression for the image deblurring model is as follows:

[0118]

[0119]

[0120] Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K. Both represent penalty parameters. For data fidelity items, To preserve the details, To enhance the total variation term, B is the fuzzy kernel term; D is the input fuzzy image. x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm.

[0121] Step (5) specifically refers to: introducing auxiliary variables. In the 0 norm , replace In The image deblurring model is then rewritten as:

[0122]

[0123]

[0124] Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K, where B is the input blurred image, and D is... x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm. All represent penalty parameters; All parameters are positive. The above equation is solved through the following alternating iterations:

[0125] Solving for v: Keeping I and u constant, solve for v using the convexity algorithm.

[0126]

[0127]

[0128] make Then the objective function of the above equation can be rewritten as:

[0129]

[0130] The nth iteration obtained by the convexity algorithm is:

[0131]

[0132]

[0133] in, For a positive weight parameter, a new replacement variable is introduced using the split Bregman method. , and Lagrange multiplier b x b y ,Will replace , replace Then subproblem v n+1 Reconstructed as:

[0134]

[0135] in, It is a positive constant. The above expression can be divided into the following sub-problems:

[0136]

[0137]

[0138]

[0139] Regarding the solution for the e-norm in the above formula, the solution is:

[0140]

[0141] in, , ;

[0142] Solving for u: Introducing auxiliary variables In the 0 norm ,but:

[0143]

[0144]

[0145] Solution I: Using the u and v obtained above, the solution for image I is as follows:

[0146]

[0147]

[0148]

[0149]

[0150]

[0151] in, Represents the Fast Fourier Transform. Indicates inverse transformation, It is a conjugate gradient operation. This represents a two-dimensional gradient operation, and Indicates two directions: horizontal and vertical. It also represents both horizontal and vertical directions.

[0152] Step (6) specifically includes the following sequential steps:

[0153] (6a) Given the fuzzy kernel size Furthermore, the size of the given blur kernel should be smaller than the size of the image;

[0154] (6b) Next, initialize the blur kernel and determine the number of updated layers based on the blur kernel size, continuously performing upsampling, wherein the number of layers is determined by... Repeat the process until the size of the coarsest layer reaches 7×7;

[0155] (6c) The input fuzzy image B and the intermediate latent image I are calculated using the following fuzzy kernel estimation formula:

[0156]

[0157] In the formula, K represents the fuzzy kernel. Indicates the penalty parameter;

[0158] Using Fast Fourier Transform:

[0159]

[0160] in, and These represent the Fast Fourier Transform and the Inverse Fourier Transform, respectively.

[0161] Step (7) specifically includes the following sequential steps:

[0162] (7a) First, the latent image I1 is estimated using a method with a Laplace prior;

[0163] (7b) Next, estimate the latent image I0:

[0164]

[0165] In the formula, K represents the blur kernel, and B is the input blurred image. The zero norm represents the number of non-zero elements; It is a 2-norm. Indicates the penalty parameter;

[0166] (7c) Then calculate the difference plot between the two estimated plots I1 and I0, and apply a bilateral filter to it.

[0167] Finally, the filtered difference map is subtracted from I1 to remove artifacts.

[0168] Table 1 shows the running time of experiments conducted on blurred images of the same size (255×255, 600×600, and 800×800). The results indicate that the present invention saves more time than prior-based deblurring methods. Figure 2A set of corresponding visual comparison images at a size of 255×255 are presented, where (a) is the blurred image, (b) is the effect image using reference 1, (c) is the effect image using reference 2, (d) is the effect image using reference 3, and (e) is the effect image of the present invention. It can be seen that the restoration result of the present invention is comparable to the methods of references 1 to 3, and takes less time. The above two sets of experiments demonstrate that the present invention can balance time efficiency and restoration results, and has a good deblurring effect on blurred images in different scenarios. Document 1 is "Blind image deblurring using darkchannel prior", Proceedings of the Conference on Computer Vision and PatternRecognition (CVPR), (2016); Document 2 is "Image deblurring via extreme channels prior", Proceedings of the Conference on Computer Vision and PatternRecognition (CVPR), (2017); Document 3 is "Blind Image deblurring with local maximum gradient prior", Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), (2019).

[0169] Table 1 Comparison of running time between the present invention and the methods in references [1-3]

[0170] Reference [1] 158.37 720.62 1270.74 Reference [2] 20.54 94.63 169.64 Reference [3] 82.85 362.92 626.30 This method 8.85 50.45 94.33

[0171] Figure 3This paper demonstrates that, compared to edge-based deblurring methods (references 4 and 5), the present invention is more robust in restoring blurred images where severe blurring results in indistinct edges, while edge-based deblurring methods are unsatisfactory. In the figures, (a) is the blurred image, (b) is the result using reference 4, (c) is the result using reference 5, and (d) is the result of the present invention. Reference 4 is "Fast motion deblurring", ACM Trans. Graph, (2009); Reference 5 is "Kernel estimation from salient structure for robust motion deblurring", Signal Process, Image Commun, (2013).

[0172] Figure 4 This demonstrates the robustness of the invention in different scenarios. The invention exhibits good restoration effects on blurred text, low-light images, faces, and natural images. (a) shows the deblurring result for the text image, (b) for the low-light image, (c) for the face image, and (d) for the natural image; from top to bottom, these represent blurred and clear images, respectively. The clear restoration results demonstrate the strong generalization ability of the invention.

[0173] The above experimental results verify that Table 1 shows that the present invention is highly efficient in terms of runtime for defuzzification. Figure 2 and 3 This invention demonstrates that it has a better restoration effect compared to other documents, wherein, Figure 2 Compared to prior-based deblurring methods, Figure 3 Compared to edge-based deblurring methods. Figure 4 This demonstrates the strong applicability of the invention across various scenarios and its robustness. In short, the invention is effective both theoretically and experimentally.

[0174] In summary, this invention effectively achieves high-quality restoration results, balancing time efficiency and deblurring effectiveness. It reduces computational costs while improving image resolution. Furthermore, this method has broad applicability, demonstrating good processing results for various types of blurred images, including text, low-light images, faces, and natural images. Compared to traditional deblurring methods based on image feature priors, it significantly improves computational efficiency while obtaining clear images. Compared to traditional methods for deblurring significant image edges, this method is more universal and robust, capable of restoring relatively clear images even in severely blurred conditions. Compared to general total variational methods based on sparse representations, this invention can deeply remove harmful structures from images, further obtaining significant edges beneficial for kernel estimation. The sparsity of the representation is also more suitable for image deblurring, resulting in a more pronounced deblurring effect.

[0175] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. An image deblurring method based on enhanced total variation, characterized in that: The method includes the following steps in sequence: (1) Based on the difference between the total variation of weights with different anisotropy and different isotropy, the l0 norm is coupled to obtain the enhanced total variation method; (2) Using the l0 norm constraint of the image gradient, the input blurred image B is preprocessed to coarsely remove undesirable details and obtain the preprocessed blurred image. (3) The enhanced total variation method is used to further sparse the preprocessed blurred image, and the harmful details of the significant interference edges are removed in a fine manner to obtain the blurred image after sparse processing. (4) Construct an image deblurring model; (5) Based on the image deblurring model, latent image estimation is performed on the sparsely processed blurred image to obtain the intermediate latent image I; (6) Estimate the blur kernel based on the image deblurring model and the intermediate latent image I, and return to step (5) to perform n alternating iterations to obtain the blur kernel. ; (7) Based on fuzzy kernel The final clear image is obtained by using a non-blind deconvolution method, resulting in the image deblurring result. Step (1) specifically refers to the following formula based on the total variation of anisotropy: ； Where z represents the input; D represents the differential operator, D x D y Let (i, j) represent the differences in the x and y directions, respectively; (i, j) represents the position. ; The 1-norm represents the sum of the absolute values ​​of the elements; The formula for total variation based on isotropy is as follows: ； The expression based on the l0 norm is as follows: ； In the formula, The zero norm represents the number of non-zero elements; The expression for the enhanced total variational method is as follows: ； Step (3) specifically refers to: using the enhanced total variation method to further sparse the preprocessed blurred image, resulting in: ； make: ； but: ； In the formula, D x D y These represent the differences in the x and y directions, respectively; The 1-norm represents the sum of the absolute values ​​of the elements; The zero norm represents the number of non-zero elements.

2. The image deblurring method based on enhanced total variation as described in claim 1, characterized in that: Step (2) specifically refers to: using the gradient l0 norm to remove undesirable details in the intermediate latent image I, the formula of which is as follows: ； in, The gradient l0 norm is sparse.

3. The image deblurring method based on enhanced total variation as described in claim 1, characterized in that: In step (4), the expression for the image deblurring model is as follows: ； ； Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K. Both represent penalty parameters. For data fidelity items, To preserve the details, To enhance the total variation term, B is the fuzzy kernel term; D is the input fuzzy image. x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm.

4. The image deblurring method based on enhanced total variation according to claim 1, characterized in that: Step (5) specifically refers to: introducing auxiliary variables. In the 0 norm , replace In The image deblurring model is then rewritten as: ； ； Where K represents the fuzzy kernel, This represents the convolution operation between the intermediate latent image I and the blur kernel K, where B is the input blurred image, and D is... x D y These represent the differences in the x and y directions, respectively; The zero norm represents the number of non-zero elements; It is a 2-norm. All represent penalty parameters; All parameters are positive. The above equation is solved through the following alternating iterations: Solving for v: Keeping I and u constant, solve for v using the convexity algorithm. ； ； make Then the objective function of the above equation can be rewritten as: ； The nth iteration obtained by the convexity algorithm is: ； ； in, For a positive weight parameter, a new replacement variable is introduced using the split Bregman method. , and Lagrange multiplier b x b y ,Will replace , replace Then subproblem v n+1 Reconstructed as: ； in, It is a positive constant. The above expression can be divided into the following sub-problems: ； ； ； Regarding the solution for the e-norm in the above formula, the solution is: ； in, , ; Solving for u: Introducing auxiliary variables In the 0 norm ,but: ； ； Solution I: Using the u and v obtained above, the solution for image I is as follows: ； ； ； ； ； in, Represents the Fast Fourier Transform. Indicates inverse transformation, It is a conjugate gradient operation. This represents a two-dimensional gradient operation, and Indicates two directions: horizontal and vertical. It also represents both horizontal and vertical directions.

5. The image deblurring method based on enhanced total variation according to claim 1, characterized in that: Step (6) specifically includes the following sequential steps: (6a) Given the fuzzy kernel size Furthermore, the size of the given blur kernel should be smaller than the size of the image; (6b) Next, initialize the blur kernel and determine the number of updated layers based on the blur kernel size, continuously performing upsampling, wherein the number of layers is determined by... Repeat the process until the size of the coarsest layer reaches 7×7; (6c) The input fuzzy image B and the intermediate latent image I are calculated using the following fuzzy kernel estimation formula: ； In the formula, K represents the fuzzy kernel. Indicates the penalty parameter; Using Fast Fourier Transform: ； in, and These represent the Fast Fourier Transform and the Inverse Fourier Transform, respectively.

6. The image deblurring method based on enhanced total variation according to claim 1, characterized in that: Step (7) specifically includes the following sequential steps: (7a) First, the latent image I1 is estimated using a method with a Laplace prior; (7b) Next, estimate the latent image I0: ； In the formula, K represents the blur kernel, and B is the input blurred image. The zero norm represents the number of non-zero elements; It is a 2-norm. Indicates the penalty parameter; (7c) Then calculate the difference plot between the two estimated plots I1 and I0, and apply a bilateral filter to it. Finally, the filtered difference map is subtracted from I1 to remove artifacts.

Images