A defocused image deblurring method based on boundary neighborhood gradient difference

Abstract

The present invention discloses a defocused image deblurring method based on boundary neighborhood gradient difference, which belongs to the technical field of computer vision. In view of the fact that the existing defocused image deblurring method cannot accurately obtain the blur amount of the boundary position of the defocused image for static scenes with multiple depth layers, the present invention makes full use of the relationship between the boundary neighborhood gradient difference and the blur amount to accurately obtain the blur amount of the boundary position of the defocused image, thereby solving the problem of boundary ringing artifacts appearing in the deblurring result; in view of the problem that the ability of the non-blind deconvolution algorithm to retain image detail information is not strong enough, resulting in the loss of detail information in the deblurring result, the present invention combines the discrete blur amount selection strategy and the sparse prior to design a non-blind deconvolution algorithm to enhance the ability of the non-blind deconvolution algorithm to retain image detail information, solves the problem of the loss of detail information in the deblurring result, and can be used for defocused image deblurring processing in static scenes with multiple depth layers.

Description

Technical Field

[0001] The invention relates to a defocused image deblurring method based on boundary neighborhood gradient difference, and belongs to the field of computer vision. Background Art

[0002] Defocused images are ubiquitous in real life. Defocused images become unclear and blurred, making it difficult to see objects and losing detailed information. This limits the further application of images in fields such as precision instrument flaw detection and augmented reality.

[0003] To improve the efficiency of defocused images, it's necessary to remove the defocus from them to sharpen them. Traditional deblurring methods estimate the blur level of pixels and calculate the blur kernel using the point spread function (a Gaussian function) and the blur value. Finally, the blur kernel and the defocused image are fed into a non-blind deconvolution algorithm to perform the deblurring operation.

[0004] For static scenes with multiple depth layers, existing defocused image deblurring methods cannot accurately obtain the amount of blur at the boundaries of the defocused image. The blur amount represents the degree of blur of the pixel points. Inaccurate estimation of the amount of defocused blur at the boundaries will lead to boundary ringing artifacts in the deblurred results. Boundary ringing artifacts refer to oscillations caused by drastic changes in grayscale. In addition, the prior knowledge of the non-blind deconvolution algorithm in the defocused image deblurring method is not strong enough. Prior knowledge refers to some known features and characteristics in the image. It can be used to constrain the non-blind deconvolution algorithm so that the deblurred result can continuously approach the real clear image. The lack of prior knowledge will lead to the non-blind deconvolution algorithm's ability to recover image detail information being insufficient, resulting in the problem of loss of detail information in the final deblurred result.

[0005] In short, the existence of boundary ringing artifacts and the lack of prior knowledge will affect the deblurring process of defocused images, ultimately leading to poor deblurring effects and insufficient image clarity. Summary of the Invention

[0006] In order to solve the problems of poor deblurring effect and insufficient image clarity in current defocused image deblurring methods for static scenes with multiple depth layers, the present invention provides a defocused image deblurring method, which includes:

[0007] Step 1: Obtaining a boundary for the defocused image, and obtaining a gradient difference of a neighborhood of the boundary at the boundary position;

[0008] Step 2: using the gradient difference of the boundary neighborhood obtained in step 1, the blur amount at the boundary position is obtained, thereby obtaining a sparse blur map;

[0009] Step 3: interpolating the sparse fuzzy image obtained in step 2 to obtain an interpolated fuzzy image;

[0010] Step 4: using the interpolated blur map obtained in step 3 to perform blur detection on the defocused image and calculate a blur ratio; when the blur ratio is greater than a preset blur ratio threshold, deblurring the defocused image;

[0011] Step 5: For the image that needs to be deblurred in step 4, the interpolated blur map in step 3 is used to obtain a blur kernel, and then a deblurring operation is performed in combination with a non-blind deconvolution algorithm.

[0012] Optionally, step 1 includes:

[0013] A scale-consistent boundary detection algorithm is used to find the boundary of the defocused image. After obtaining the boundary position, the gradient in the neighborhood centered on each boundary position is calculated in the original image, and then the maximum gradient and the minimum gradient in the neighborhood are subtracted to obtain the gradient difference at the boundary position.

[0014] Optionally, step 2 includes:

[0015] Step 2.1: Get the relationship between the boundary neighborhood gradient difference and the blur amount:

[0016] The defocus process of the image is modeled as shown in formula (1):

[0017]

[0018] Wherein, I represents the defocused image, k represents the blur kernel, x represents the clear image, and n represents the noise. represents a convolution operation, wherein the blur kernel performs a convolution operation on the clear image and adds the noise to obtain the defocused image;

[0019] The blur kernel is obtained by a point spread function and a blur amount, and the point spread function is represented by a Gaussian function, as shown in formula (2):

[0020]

[0021] Among them, σ represents the blur amount, which represents the degree of blur of the pixel, and (x, y) represents the pixel coordinates;

[0022] The boundary l(x,y) of the clear image is modeled as shown in formula (3):

[0023] l(x,y)=a*u(x,y)+b (3)where a is the amplitude, b is the offset, and u(x,y) is the step function;

[0024] The boundary neighborhood gradient difference is defined as shown in formula (4):

[0025]

[0026] Wherein, GD(x,y) is the gradient difference of the boundary neighborhood, (x,y)′ represents the neighborhood centered at (x,y), and J(x,y) represents the defocused image boundary. represents the gradient in the boundary neighborhood, that is, the derivative of the boundary, and the derivative result is shown in formula (5):

[0027]

[0028] The boundary position is (x, y) = (0, 0), so the boundary neighborhood gradient difference is shown in formula (6):

[0029]

[0030] According to formula (6), the relationship between the boundary neighborhood gradient difference and the blur amount is derived as shown in formula (7):

[0031]

[0032] Step 2.2: According to the relationship between the boundary neighborhood gradient difference and the blur amount and the boundary neighborhood gradient difference obtained in step 1, the blur amount at the boundary is calculated, thereby obtaining a sparse blur map.

[0033] Optionally, the process of calculating the fuzzy ratio in step 4 includes:

[0034] Setting a blur threshold, determining whether the blur amount of each pixel in the interpolated blur image is greater than the blur threshold, and setting the pixel as blurred when the blur amount is greater than the threshold, otherwise setting the pixel as non-blurred;

[0035] The interpolation blur image is divided into a blur area and a non-blur area according to the segmentation result, and the ratio of the number of pixels in the blur area to the number of pixels in the entire image, that is, the blur ratio, is calculated.

[0036] Optionally, step 5 includes:

[0037] Step 5.1: Select discrete fuzzy values ​​for the interpolation fuzzy map obtained in step 3, and obtain n fuzzy values ​​σ1, σ2…σ with a step q. n ; Using these n blur amounts, a point spread function is used to obtain n blur kernels, wherein the point spread function is represented by a Gaussian function, and the standard deviation of the Gaussian function represents the blur amount;

[0038] Step 5.2: Performing n non-blind deconvolution operations on the defocused image using a non-blind deconvolution algorithm based on a sparse prior and n blur kernels to obtain n deblurred images; each blur kernel corresponds to a blur amount. After deblurring the defocused image using the nth blur kernel, first obtain the locations in the blur image where the blur amount is larger than the nth blur amount, and extract the pixels in the deblurred result corresponding to these locations to obtain the nth deblurred result.

[0039] The above operation is performed n times, and n deblurred images are finally obtained;

[0040] Step 5.3: Add the pixel values ​​of the corresponding positions of the n deblurred images in step 5.2, and finally divide each pixel value by 255 to obtain a fully focused image, that is, a completely clear image.

[0041] Optionally, step 3 uses a KNN matting interpolation algorithm to interpolate the sparse fuzzy image.

[0042] Optionally, the size of the neighborhood is 11×11.

[0043] Optionally, the image used for processing is at least one of the following: a blurred face image; a blurred person image; a blurred scenery image; a blurred vehicle image; a blurred animal image; and a blurred plant image.

[0044] The second object of the present invention is to provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the defocused image deblurring method described above.

[0045] A third object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned defocused image deblurring method.

[0046] The beneficial effects of the present invention are:

[0047] The defocused image deblurring method of the present invention fully utilizes the relationship between the gradient difference of the boundary neighborhood and the blur amount to accurately obtain the blur amount at the boundary position of the defocused image, thereby solving the problem of boundary ringing artifacts in the deblurred result. To address the problem that the non-blind deconvolution algorithm's ability to retain image detail information is insufficient, resulting in the loss of detail information in the deblurred result, the present invention combines a discrete blur amount selection strategy with a sparse prior to design a non-blind deconvolution algorithm to enhance the non-blind deconvolution algorithm's ability to retain image detail information, thus solving the problem of detail information loss in the deblurred result.

[0048] Compared with the existing defocused image deblurring method, the present invention can not only effectively solve the problem of boundary ringing artifacts, but also avoid the loss of image detail information, and effectively improve the clarity of the defocused image after deblurring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is the defocused image input in the second embodiment of the present invention.

[0051] Figure 2 The second embodiment of the present invention uses the scale-consistent boundary detection algorithm to detect the Figure 1 The boundary map obtained by performing boundary detection.

[0052] Figure 3 Demonstrates the use of the second embodiment of the present invention Figure 2 The boundary position in Figure 1 The boundary neighborhood gradient difference map obtained from the boundary in .

[0053] Figure 4 Demonstrates the use of the second embodiment of the present invention Figure 3 The blur amount map at the boundary position is obtained by the boundary neighborhood gradient difference in .

[0054] Figure 5 Demonstrates the use of the second embodiment of the present invention Figure 4 The blur amount at the boundary position in and the blur amount map of each pixel position obtained by the KNN matting interpolation algorithm.

[0055] Figure 6 The final deblurring result diagram obtained according to the obtained blur amount and the non-blind deconvolution algorithm in Example 2 of the present invention is shown. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0057] Example 1:

[0058] This embodiment provides a defocused image deblurring method, comprising:

[0059] Step 1: Obtaining a boundary for the defocused image, and obtaining a gradient difference of a neighborhood of the boundary at the boundary position;

[0060] Step 2: using the gradient difference of the boundary neighborhood obtained in step 1, the blur amount at the boundary position is obtained, thereby obtaining a sparse blur map;

[0061] Step 3: interpolating the sparse fuzzy image obtained in step 2 to obtain an interpolated fuzzy image;

[0062] Step 4: using the interpolated blur map obtained in step 3 to perform blur detection on the defocused image and calculate a blur ratio; when the blur ratio is greater than a preset blur ratio threshold, deblurring the defocused image;

[0063] Step 5: For the image that needs to be deblurred in step 4, the interpolated blur map in step 3 is used to obtain a blur kernel, and then a deblurring operation is performed in combination with a non-blind deconvolution algorithm.

[0064] Example 2:

[0065] This embodiment provides a defocused image deblurring method based on boundary neighborhood gradient difference, comprising the following steps:

[0066] Step 1: Use a scale-consistent boundary detection algorithm to find the boundaries of the defocused image. This scale-consistent boundary detection algorithm is derived from the article Edge-Based Defocus Blur Estimation With Adaptive Scale Selection. After obtaining the boundary location, calculate the gradient of an 11x11 neighborhood centered on each boundary location in the original image. Then, subtract the maximum and minimum gradients in the neighborhood to obtain the gradient difference at the boundary location.

[0067] Step 2: Use the boundary neighborhood gradient difference obtained in step 1 to calculate the blur amount at the boundary position, thereby obtaining a sparse blur map. The specific steps are as follows:

[0068] Step 2.1: Obtain the relationship between the boundary neighborhood gradient difference and the blur amount.

[0069] The defocus process of the image is modeled as shown in formula (1):

[0070]

[0071] Where I represents the defocused image, k represents the blur kernel, x represents the clear image, and n represents the noise. Represents the convolution operation. The blur kernel performs a convolution operation on the clear image and adds noise to obtain a defocused image. The blur kernel is obtained by the point spread function and the blur amount. The point spread function is represented by a Gaussian function, as shown in formula (2):

[0072]

[0073] Among them, σ represents the blur amount, the blur amount represents the degree of blur of the pixel, and (x, y) represents the pixel coordinates.

[0074] The boundary l(x,y) of the clear image is modeled as shown in formula (3):

[0075] l(x,y)=a*u(x,y)+b (3)

[0076] Where a is the amplitude, b is the offset, and u(x,y) is the step function.

[0077] The definition of the boundary neighborhood gradient difference is shown in formula (4):

[0078]

[0079] Through experiments, we found that the minimum gradient in the boundary neighborhood is very small and close to zero, so the minimum value is omitted in the derivation of the formula. Among them, GD(x,y) is the gradient difference of the boundary neighborhood, (x,y)′ represents the neighborhood centered at (x,y), and J(x,y) represents the defocused image boundary. represents the gradient in the neighborhood of the boundary, that is, the derivative of the boundary. The derivative result is shown in formula (5):

[0080]

[0081] The boundary position is (x, y) = (0, 0), so the boundary neighborhood gradient difference is shown in formula (6):

[0082]

[0083] According to formula (6), the relationship between the gradient difference of the boundary neighborhood and the blur amount can be derived, as shown in formula (7):

[0084]

[0085] Step 2.2: According to the relationship between the boundary neighborhood gradient difference and the blur amount and the boundary neighborhood gradient difference obtained in step 1, the blur amount at the boundary is calculated to obtain a sparse blur map.

[0086] Step 3: The sparse blur map only contains the blur amounts at the boundary positions. In order to obtain the blur amounts of the remaining pixels, this embodiment uses the KNN matting interpolation algorithm to interpolate the sparse blur map obtained in step 2, thereby obtaining an interpolated blur map containing the blur amounts of all pixels.

[0087] Step 4: Based on the interpolated blur map obtained in Step 3 and the set blur threshold, the defocused image is divided into blurred and unblurred areas. When the blur value of a pixel exceeds the blur threshold, the pixel is set to blurred; otherwise, it is unblurred. Performing the above operation for each pixel divides the image into blurred and unblurred areas. Calculate the ratio of the number of pixels in the blurred area to the total number of pixels in the entire image, i.e., the blur ratio. When the blur ratio is greater than a certain value, the image is considered blurred and requires deblurring; otherwise, it is not.

[0088] Step 5: For the image that needs to be deblurred in step 4, use the interpolated blur map in step 3 to obtain the blur kernel, and then perform the deblurring operation in combination with the non-blind deconvolution algorithm. The specific steps are as follows:

[0089] Step 5.1: Perform discrete blur selection on the interpolated blur map obtained in Step 3 to obtain n blur values. This discrete blur selection strategy is derived from the paper Spatially-Varying Out-Of-Focus Image Deblurring With L1-2 Optimization And A Guided Blur Map. These n blur values ​​are used to generate n blur kernels using a point spread function. In this embodiment, the point spread function is represented by a Gaussian function, and the standard deviation of the Gaussian function represents the blur value.

[0090] Step 5.2: Perform n non-blind deconvolution operations on the defocused image using a non-blind deconvolution algorithm based on a sparse prior and n blur kernels, yielding n deblurred images. The non-blind deconvolution algorithm is derived from the article "Image and Depth from a Conventional Camera with a Coded Aperture." Each blur kernel corresponds to a blur value. After deblurring the defocused image using the nth blur kernel, first obtain the pixel locations in the blur map with blur values ​​greater than the nth blur value. The pixel values ​​at these locations in the nth deblurred image are extracted, and the remaining pixels are discarded to obtain the nth deblurred image. This deblurred image contains only the values ​​of a subset of pixel locations, with the remaining pixel values ​​set to 0. The blur map is a grayscale image, and the value at each pixel location is the blur value. This blur map corresponds to the defocused image, with each pixel location in the defocused image corresponding to a pixel location in the blur map. The blur value at a pixel location in the blur map is the blur value at the corresponding pixel location in the defocused image. Perform this operation n times, ultimately yielding n deblurred images.

[0091] Step 5.3: Add the pixel values ​​of the n deblurred images from step 5.2, and finally divide each pixel value by 255 to obtain a fully focused image, that is, a completely clear image.

[0092] The environment in which the system is implemented in this embodiment is as follows:

[0093] Table 1: System hardware configuration table

[0094]

[0095] Table 2: System software configuration table

[0096] software Related information operating system Windows 10 64-bit Matlab Matlab R2021a

[0097] The experimental results are shown in the attached figure.

[0098] Figure 1 : The input defocused image, that is, the image that needs to be processed to make it sharp. As can be seen in the figure, the degree of blur is the highest on the left, and the blur decreases from left to right.

[0099] Figure 2 : The boundary map obtained using the scale-consistent boundary detection algorithm is a binary image, that is, the pixel values ​​are only 0 and 1, 1 represents white and 0 represents black. White pixels represent the boundary locations.

[0100] Figure 3 : The boundary neighborhood gradient difference map represents the gradient difference of each boundary position. In the method of this embodiment, it is found that the boundary neighborhood gradient difference is inversely proportional to the blur amount, that is, the more blurred the boundary position, the smaller the gradient difference, and the clearer the boundary position, the larger the gradient difference. Figure 1 The input defocused image is also described in Figure 1. The left side of the input image is the most blurry, with the blur decreasing from left to right. The boundary neighborhood gradient difference graph shows that the gradient difference is smaller at the left boundary and larger at the right boundary. This also verifies the method of this embodiment. Brighter areas in the graph represent larger values, while darker areas have smaller values.

[0101] Figure 4 : Sparse blur map, which only contains the blur amount at the boundary position. The blur amount represents the degree of blur of the pixel. It can be seen from the figure that the blur amount at the boundary position on the left is larger, and the blur amount at the boundary position on the right is smaller, which is also consistent with Figure 1 The brighter the area in the figure, the larger the value, and the darker the area, the smaller the value.

[0102] Figure 5 : Blur map, which represents the amount of blur for each pixel in the defocused image. It is a grayscale image, where the brighter the area, the higher the blur, and the darker the area, the lower the blur.

[0103] Figure 6 : Deblurred image, that is, a clearer image. It can be seen from the figure that compared with Figure 1 , the right side of the image is obviously clearer, and there is no boundary ringing artifact as a whole, and the image details are fully preserved.

[0104] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A defocused image deblurring method, characterized in that: The defocused image deblurring method comprises: Step 1: Obtaining a boundary for the defocused image, and obtaining a gradient difference of a neighborhood of the boundary at the boundary position; Step 2: using the gradient difference of the boundary neighborhood obtained in step 1, the blur amount at the boundary position is obtained, thereby obtaining a sparse blur map; Step 3: interpolating the sparse fuzzy image obtained in step 2 to obtain an interpolated fuzzy image; Step 4: using the interpolated blur map obtained in step 3 to perform blur detection on the defocused image and calculate a blur ratio; when the blur ratio is greater than a preset blur ratio threshold, deblurring the defocused image; Step 5: For the image that needs to be deblurred in step 4, use the interpolated blur map in step 3 to obtain a blur kernel, and then perform a deblurring operation in combination with a non-blind deconvolution algorithm; The step 5 comprises: Step 5.1: Select discrete fuzzy values ​​for the interpolation fuzzy map obtained in step 3, and obtain n fuzzy values ​​σ1, σ2…σ with a step q. n ; Using these n blur amounts, a point spread function is used to obtain n blur kernels, wherein the point spread function is represented by a Gaussian function, and the standard deviation of the Gaussian function represents the blur amount; Step 5.2: Performing n non-blind deconvolution operations on the defocused image using a non-blind deconvolution algorithm based on a sparse prior and n blur kernels to obtain n deblurred images; each blur kernel corresponds to a blur amount. After deblurring the defocused image using the nth blur kernel, first obtain the locations in the blur image where the blur amount is larger than the nth blur amount, and extract the pixels in the deblurred result corresponding to these locations to obtain the nth deblurred result. The above operation is performed n times, and n deblurred images are finally obtained; Step 5.3: Add the pixel values ​​of the corresponding positions of the n deblurred images in step 5.2, and finally divide each pixel value by 255 to obtain a fully focused image, that is, a completely clear image.

2. The defocused image deblurring method according to claim 1, wherein: The step 1 comprises: A scale-consistent boundary detection algorithm is used to find the boundary of the defocused image. After obtaining the boundary position, the gradient in the neighborhood centered on each boundary position is calculated in the original image, and then the maximum gradient and the minimum gradient in the neighborhood are subtracted to obtain the gradient difference at the boundary position.

3. The defocused image deblurring method according to claim 1, wherein: The step 2 includes: Step 2.1: Get the relationship between the boundary neighborhood gradient difference and the blur amount: The defocus process of the image is modeled as shown in formula (1): Wherein, I represents the defocused image, k represents the blur kernel, x represents the clear image, and n represents the noise. represents a convolution operation, wherein the blur kernel performs a convolution operation on the clear image and adds the noise to obtain the defocused image; The blur kernel is obtained by a point spread function and a blur amount, and the point spread function is represented by a Gaussian function, as shown in formula (2): Among them, σ represents the blur amount, which represents the degree of blur of the pixel, and (x, y) represents the pixel coordinates; The boundary l(x,y) of the clear image is modeled as shown in formula (3): l(x,y)=a*u(x,y)+b (3) Where a is the amplitude, b is the offset, and u(x,y) is the step function; The boundary neighborhood gradient difference is defined as shown in formula (4): Wherein, GD(x,y) is the gradient difference of the boundary neighborhood, (x,y)′ represents the neighborhood centered at (x,y), and J(x,y) represents the defocused image boundary. represents the gradient in the boundary neighborhood, that is, the derivative of the boundary, and the derivative result is shown in formula (5): The boundary position is (x, y) = (0, 0), so the boundary neighborhood gradient difference is shown in formula (6): According to formula (6), the relationship between the boundary neighborhood gradient difference and the blur amount is derived as shown in formula (7): Step 2.2: According to the relationship between the boundary neighborhood gradient difference and the blur amount and the boundary neighborhood gradient difference obtained in step 1, the blur amount at the boundary is calculated, thereby obtaining a sparse blur map.

4. The defocused image deblurring method according to claim 1, wherein: The process of calculating the fuzzy ratio in step 4 includes: Setting a blur threshold, determining whether the blur amount of each pixel in the interpolated blur image is greater than the blur threshold, and setting the pixel as blurred when the blur amount is greater than the threshold, otherwise setting the pixel as non-blurred; The interpolation blur image is divided into a blur area and a non-blur area according to the judgment result, and the ratio of the number of pixels in the blur area to the number of pixels in the entire image, that is, the blur ratio, is calculated.

5. The defocused image deblurring method according to claim 1, wherein: The step 3 uses the KNNmatting interpolation algorithm to interpolate the sparse fuzzy image.

6. The defocused image deblurring method according to claim 2, wherein: The size of the neighborhood is 11×11.

7. The defocused image deblurring method according to any one of claims 1 to 6, wherein the image to be processed is at least one of the following: a blurred face image; a blurred person image; a blurred scenery image; a blurred vehicle image; a blurred animal image; and a blurred plant image.

8. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the defocused image deblurring method according to any one of claims 1 to 7. 9 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the defocused image deblurring method according to claim 1 .

Images