A natural image deblurring method based on structure extraction

Abstract

The present invention provides a natural image deblurring method based on structure extraction, which estimates the blur kernel after denoising preprocessing the image, and then performs a non-blind deconvolution step on the image to obtain the deblurred image; the method comprises the following steps: step S1, denoising preprocessing the original blurred image P1 to obtain P2; step S2, estimating the intermediate latent image of the preprocessed image P2 to obtain the image P3; step S3, extracting the main structure of the intermediate latent image P3, and obtaining the image P4 after some adjustments; step S4, using the image P4 for kernel estimation; step S5, performing non-blind deconvolution to obtain the deblurred image P5. The application of this technical solution can avoid the adverse effects of noise on the subsequent kernel estimation step. The continuity of the blur kernel can be guaranteed while ensuring the sparsity of the blur kernel.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a natural image deblurring method based on structure extraction. Background Art

[0002] Deblurring has always been a hot topic in the field of image processing and has developed rapidly in recent years. With the popularity of various electronic camera products, especially small high-resolution cameras, taking photos anytime and anywhere has become possible. These electronic devices are small in size, light in weight, and in most cases handheld, making stability a major problem. At the moment of capturing the photo, if the camera shakes, the photo quality will deteriorate or even seriously degrade. The artifacts caused by blur can be alleviated by reducing the exposure time, but this will introduce noise into the image. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a natural image deblurring method based on structure extraction, so as to avoid the adverse effect of noise on the subsequent kernel estimation step, and ensure the continuity of the blur kernel while ensuring the sparsity of the blur kernel.

[0004] To achieve the above object, the present invention adopts the following technical scheme: a natural image deblurring method based on structure extraction, which estimates the blur kernel after denoising preprocessing the image, and then performs the image non-blind deconvolution step to obtain the deblurred image; comprising the following steps:

[0005] Step S1, performing denoising preprocessing on the original blurred image P1 to obtain P2;

[0006] Step S2, performing intermediate latent image estimation on the preprocessed image P2 to obtain image P3;

[0007] Step S3, extracting the main structure of the intermediate latent image P3, and obtaining the image P4 after some adjustments;

[0008] Step S4, using image P4 to perform kernel estimation;

[0009] Step S5: perform non-blind deconvolution to obtain a deblurred image P5.

[0010] In a preferred embodiment, the denoising method used in step S1 is bilateral filtering.

[0011] In a preferred embodiment, step S2 estimates the intermediate latent image of the preprocessed image P2 to obtain the image P3, which specifically includes the following steps:

[0012] The intermediate image estimation model is defined as:

[0013]

[0014] in is the gradient of image L, k is the blur kernel, B is the blurred image, ψ1 and ψ2 are weights, represents the element-wise multiplication operator, J is an all-1 matrix, and M is a 2D binary mask. To solve (1), k is fixed to optimize L, and a semi-quadratic penalty method is used to solve highly non-convex functions. An auxiliary variable v is introduced to replace Transform (1) into the following optimization problem:

[0015]

[0016] Where ψ3 is the weight. When it approaches ∞, the solution of (2) converges to the solution of (1). We solve (2) by iteratively updating v and L.

[0017] By fixing all variables except v, (2) simplifies to:

[0018]

[0019] v is obtained by the Newton-Paphson method; then v is fixed and L is optimized by minimizing the following formula:

[0020]

[0021] Use fast Fourier transform to calculate L; based on Paseva l's theorem, the solution of L is expressed as:

[0022]

[0023] Among them, F and F -1 are the fast Fourier transform and its inverse operation, is a conjugate operator. F(B), F(k), F(v), They are B, k, v, The Fourier transform of .

[0024] In a preferred embodiment, in step S3, the main structure of the intermediate latent image P3 is first extracted, and the image P4 is obtained after adjustment, which specifically includes the following steps:

[0025] Step S31, extracting the main structure of the image; the main structure is extracted by minimizing the following objective function:

[0026]

[0027] Where S is the structure map to be obtained, B is the channel of the brightness or log brightness of the input blurred image, i is the pixel index in the two-dimensional image, and S i , B iare the pixel values ​​with index i in the structure graph S and the input blurred image B respectively; the second item is the RTV regularization term; ξ is a small positive number; where T is the pixel-level windowed total variation, expressed as follows (7), and the L term is used to distinguish the salient structure from the texture elements, which captures the overall spatial variation, expressed as follows (9).

[0028]

[0029] where j belongs to Ω(i), i.e., the rectangular area centered at pixel i; T x (i) and T y (i) are the total windowed changes of pixel i in the x and y directions, respectively, and the absolute spatial difference within the window Ω(i) is calculated; w i,j is a weight function defined according to spatial attraction, expressed as:

[0030]

[0031] Among them, x i ,y i is the horizontal and vertical coordinates of i, x j ,y j is the horizontal and vertical coordinates of j, and σ controls the spatial size of the window;

[0032]

[0033] where j belongs to Ω(i), i.e., the rectangular area centered on pixel i; L x (i) L y (i) are the windowed intrinsic variations of pixel i in the x and y directions, respectively.

[0034] Step S32, enhancing the main structure image S; using an impact filter to enhance the extracted structure image, which reduces the ringing artifacts caused by blur;

[0035]

[0036]

[0037] in S x , S xx is the first and second derivative of S in the x direction, S xy S is derived once in the x direction and then twice in the y direction. y , S yy It is the first and second derivative of S in the y direction.

[0038] Step S33: extracting image High frequency components; significant structural diagram after the above enhancement Then, use the discrete filter and Generate high frequency images

[0039] In a preferred embodiment, step S4 is specifically implemented as follows:

[0040]

[0041] Subject to the constraints k≥0,∑k=1; here is the structural diagram of the unknown clear image in the high-frequency space, k is the unknown blur kernel, η1 and η2 are the adjustable weights for controlling the sparsity and smoothness of the blur kernel, respectively. is a 2D convolution operator; (11) consists of three items; the first item is the data fidelity item; the second and third items are Gaussian regularization items composed of image intensity and gradient, respectively.

[0042] In a preferred embodiment, step S5 is specifically implemented as follows:

[0043] In the non-blind deconvolution stage of the image, among many priors, the super Laplacian prior strikes a balance between accuracy and speed, so the continuous method is used to solve the following cost function:

[0044]

[0045] in and are the high-frequency components of the blurred image in the x and y directions, respectively, and τ and β are adjustable weights.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Before performing the image deblurring step, the present invention performs a denoising preprocessing on the original natural image, which greatly reduces the influence of high-frequency noise in the natural image on the blur kernel estimation;

[0048] 2. When estimating the intermediate latent image, the present invention follows the characteristics of natural images and adopts super-Laplacian image prior. In addition, a mask is introduced to encode the edge area, and a smooth prior is used to eliminate noise and ringing artifacts in the local smooth area;

[0049] 3. When performing kernel estimation, the present invention selects good image features, that is, the main structure of the image. The present invention adopts an effective structure selection method to obtain effective, good and significant edges;

[0050] 4. The present invention adopts a simple hyper-Laplacian prior in the final image non-blind deconvolution step. This method is fast and robust to small kernel errors and is beneficial to the real-time performance of the algorithm.

[0051] The present invention mainly has three major steps: the first is to perform denoising preprocessing on the original natural image to reduce the influence of high-frequency noise in the natural image on the blur kernel estimation; the second is the blur kernel estimation, which is used for deconvolution with the blurred image; the third is the final image non-blind deconvolution to obtain the final reconstructed clear deblurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a structural block diagram of a preferred embodiment of the present invention;

[0053] Figure 2 is the original natural input image in the preferred embodiment of the present invention;

[0054] Figure 3 The image P2 is the image after the denoising preprocessing in step S1 in the preferred embodiment of the present invention;

[0055] Figure 4 The intermediate latent image P3 obtained in step S2 of the preferred embodiment of the present invention;

[0056] Figure 5 This is a main structural diagram of step S3 in a preferred embodiment of the present invention;

[0057] Figure 6 It is an enhanced diagram of the structure diagram in step S3 in the preferred embodiment of the present invention;

[0058] Figure 7 The high-frequency image P4 of the structure enhancement image in step S3 in the preferred embodiment of the present invention;

[0059] Figure 8 is the blur kernel estimated in step S4 in the preferred embodiment of the present invention;

[0060] Fig. 9 It is a clear deblurred image finally obtained by image non-blind deconvolution reconstruction in step S5 in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0063] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0064] like Figure 1-9 As shown in the figure, a natural image deblurring method based on structure extraction has three major steps: first, denoising preprocessing of the original natural image is performed to reduce the influence of high-frequency noise in the natural image on blur kernel estimation; second, blur kernel estimation is used for deconvolution with the blurred image; third, the final image is non-blindly deconvolved to obtain the final reconstructed clear deblurred image. It includes the following steps:

[0065] Step S1, performing denoising preprocessing on the original blurred image P1 to obtain P2;

[0066] Step S2, performing intermediate latent image estimation on the preprocessed image P2 to obtain image P3;

[0067] Step S3, extracting the main structure of the intermediate latent image P3, and obtaining the image P4 after some adjustments;

[0068] Step S4, using image P4 to perform kernel estimation;

[0069] Step S5: perform non-blind deconvolution to obtain a deblurred image P5.

[0070] In this embodiment, the denoising method used in step S1 is bilateral filtering. Figure 3 This is a result diagram obtained by performing denoising preprocessing on the original image in step S1 in the embodiment of the present invention.

[0071] In this implementation example, step S2 estimates the intermediate latent image of the preprocessed image P2 to obtain the image P3, which specifically includes the following steps:

[0072] The intermediate image estimation model is defined as:

[0073]

[0074] in is the gradient of image L, k is the blur kernel, B is the blurred image, ψ1 and ψ2 are weights, represents the element-wise multiplication operator, J is a matrix of all 1s, and M is a 2D binary mask. To solve (1), we can optimize L by fixing k and use a semi-quadratic penalty method to solve highly non-convex functions. We introduce an auxiliary variable v to replace Transform (1) into the following optimization problem:

[0075]

[0076] Where ψ3 is the weight. When it approaches ∞, the solution of (2) converges to the solution of (1). (2) is solved by iteratively updating v and L.

[0077] By fixing all variables except v, (2) simplifies to:

[0078]

[0079] v can be obtained by the Newton-Paphson method.

[0080] Then, fixing v, L can be optimized by minimizing:

[0081]

[0082] Use fast Fourier transform to calculate L. Based on Paseva l's theorem, the solution of L is expressed as:

[0083]

[0084] Among them, F and F -1 are the fast Fourier transform and its inverse operation, is a conjugate operator. F(B), F(k), F(v), They are B, k, v, The Fourier transform of .

[0085] Figure 4 is the intermediate latent image obtained.

[0086] In this implementation example, in step S3, the main structure of the intermediate latent image P3 is first extracted, and the image P4 is obtained after some adjustments, which specifically includes the following steps:

[0087] Step S31, extract the main structure of the image. The main structure is extracted by minimizing the following objective function:

[0088]

[0089] Where S is the structure map to be obtained, B is the brightness (or log brightness) channel of the input blurred image, i is the pixel index in the two-dimensional image, and S i , B i are the pixel values ​​with index i in the structure graph S and the input blurred image B respectively. The second term is the RTV regularization term. ξ is a small positive number to avoid division by zero. Where T is the pixel-level windowed total variation, expressed as:

[0090]

[0091] where j belongs to Ω(i), which is the rectangular area centered on pixel i. x (i) and T y (i) is the total windowed change in the x and y directions for pixel i, calculating the absolute spatial difference within the window Ω(i). i,j is a weight function defined according to spatial attraction, expressed as:

[0092]

[0093] Among them, x i ,y i is the horizontal and vertical coordinates of i, x j ,y j is the horizontal and vertical coordinates of j, and σ controls the spatial size of the window.

[0094] The L term is used to distinguish the salient structure from the texture elements, which captures the overall spatial variation. It is expressed as:

[0095]

[0096] where j belongs to Ω(i), i.e., the rectangular area centered on pixel i; L x (i) L y (i) are the windowed intrinsic variations of pixel i in the x and y directions, respectively.

[0097] Figure 5 Figure 2 is the main structure extracted from the intermediate latent image.

[0098] Step S32: Enhance the main structure image S. Use an impact filter to enhance the extracted structure image, which can reduce the ringing artifacts caused by blur.

[0099]

[0100]

[0101] in S x , S xx is the first and second derivative of S in the x direction, S xy S is derived once in the x direction and then twice in the y direction. y , S yy It is the first and second derivative of S in the y direction.

[0102] Figure 6 This is an enhanced diagram of the structure diagram.

[0103] Step S33: extracting image High frequency components. After obtaining the above enhanced significant structure diagram Then I used a discrete filter and Generate high frequency images

[0104] Figure 7 It is the high frequency component map of the structure enhancement map.

[0105] In this implementation example, step S4 is specifically implemented as follows:

[0106]

[0107] Subject to the constraints k≥0,∑k=1. Here is the structural diagram of the unknown clear image in the high-frequency space, k is the unknown blur kernel, η1 and η2 are the adjustable weights for controlling the sparsity and smoothness of the blur kernel, respectively. is a 2D convolution operator. (11) consists of three items. The first item is the data fidelity item, which makes the input and output consistent in content. The second and third items are Gaussian regularization items composed of image intensity and gradient, respectively, which can ensure the continuity of the blur kernel while ensuring the sparsity of the blur kernel.

[0108] Figure 8 is the estimated blur kernel.

[0109] In this implementation example, step S5 is specifically implemented as follows:

[0110] In the non-blind deconvolution stage of the image, among many priors, the super Laplacian prior strikes a balance between accuracy and speed, so the continuous method is used to solve the following cost function:

[0111]

[0112] in and are the high-frequency components of the blurred image in the x and y directions, respectively, and τ and β are adjustable weights.

[0113] Fig. 9 The clear de-shaking blurred image reconstructed after non-blind deconvolution of the final image.

[0114] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A natural image deblurring method based on structure extraction, characterized in that: After performing denoising preprocessing on the image, the blur kernel is estimated, and then the image non-blind deconvolution step is performed to obtain the deblurred image; the steps include: Step S1, performing denoising preprocessing on the original blurred image P1 to obtain P2; Step S2, performing intermediate latent image estimation on the preprocessed image P2 to obtain image P3; Step S3, extracting the main structure of the intermediate latent image P3, and obtaining the image P4 after some adjustments; Step S4, using image P4 to perform kernel estimation; Step S5, performing non-blind deconvolution to obtain a deblurred image P5; Step S2 estimates the intermediate latent image of the preprocessed image P2 to obtain the image P3, which specifically includes the following steps: The intermediate latent image estimation model is defined as: in is the gradient of image L, k is the blur kernel, B is the blurred image, ψ1 and ψ2 are weights, represents the element-wise multiplication operator, J is an all-1 matrix, and M is a 2D binary mask. To solve (1), k is fixed to optimize L, and a semi-quadratic penalty method is used to solve highly non-convex functions. An auxiliary variable v is introduced to replace Transform (1) into the following optimization problem: Where ψ3 is the weight. When it approaches ∞, the solution of (2) converges to the solution of (1). We solve (2) by iteratively updating v and L. By fixing all variables except v, (2) simplifies to: v is obtained by the Newton-Paphson method; then v is fixed and L is optimized by minimizing the following formula: Use fast Fourier transform to calculate L; based on Paseval's theorem, the solution of L is expressed as: Among them, F and F -1 are the fast Fourier transform and its inverse operation, is a conjugate operator; F(B), F(k), F(v), They are B, k, v, Fourier transform of In the step S3, the main structure of the intermediate latent image P3 is first extracted, and the image P4 is obtained after adjustment, which specifically includes the following steps: Step S31, extracting the main structure of the image; the main structure is extracted by minimizing the following objective function: Where S is the structure map to be obtained, B is the channel of the brightness or log brightness of the input blurred image, i is the pixel index in the two-dimensional image, and S i , B i are the pixel values ​​with index i in the structure graph S and the input blurred image B respectively; the second item is the RTV regularization term; ξ is a small positive number; where T is the pixel-level windowed total variation, expressed as follows (7), the L term is used to distinguish the salient structure from the texture elements, it captures the overall spatial variation, expressed as follows (9); where j belongs to Ω(i), i.e., the rectangular area centered at pixel i; T x (i) and T y (i) are the total windowed changes of pixel i in the x and y directions, respectively, and the absolute spatial difference within the window Ω(i) is calculated; w i,j is a weight function defined according to spatial attraction, expressed as: Among them, x i ,y i is the horizontal and vertical coordinates of i, x j ,y j is the horizontal and vertical coordinates of j, and σ controls the spatial size of the window; where j belongs to Ω(i), i.e., the rectangular area centered on pixel i; L x (i) L y (i) are the windowed intrinsic variations of pixel i in the x and y directions, respectively; Step S32, enhancing the main structure image S; using an impact filter to enhance the extracted structure image, which reduces the ringing artifacts caused by blur; in S x , S xx is the first and second derivative of S in the x direction, S xy S is derived once in the x direction and then twice in the y direction. y , S yy is the first and second derivative of S in the y direction; Step S33: extracting image High frequency components; significant structural diagram after the above enhancement Then, use the discrete filter and Generate high frequency images 2. The natural image deblurring method based on structure extraction according to claim 1, characterized in that: The denoising method used in step S1 is bilateral filtering.

3. The natural image deblurring method based on structure extraction according to claim 1, characterized in that: Step S4 is specifically implemented as follows: Subject to the constraints k≥0,∑k=1; here is the structural diagram of the unknown clear image in the high-frequency space, k is the unknown blur kernel, η1 and η2 are the adjustable weights for controlling the sparsity and smoothness of the blur kernel, respectively. is a 2D convolution operator; (11) consists of three items; the first item is the data fidelity item; the second and third items are Gaussian regularization items composed of image intensity and gradient, respectively.

4. The natural image deblurring method based on structure extraction according to claim 1, characterized in that: Step S5 is specifically implemented as follows: In the non-blind deconvolution stage of the image, among many priors, the super Laplacian prior strikes a balance between accuracy and speed, so the continuous method is used to solve the following cost function: in and are the high-frequency components of the blurred image in the x and y directions, respectively, and τ and β are adjustable weights.

Images