Image blind deblurring method based on image gradient inner product prior
By optimizing the image deblurring algorithm using image gradient inner product prior and fast Fourier transform, the problems of large error and long computation time in existing technologies are solved, achieving more efficient image restoration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN UNIV
- Filing Date
- 2023-08-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing blind image deblurring methods suffer from large errors and long computation times when dealing with complex blur kernels, especially due to the low computational efficiency caused by inverting large matrices.
We employ a priori method based on image gradient inner product. By calculating the inner product of the gradients of each point in the image with its neighboring points, and combining Fast Fourier Transform and gradient ascent, we avoid large matrix inversion and optimize the image deblurring process.
It improves the accuracy and computational efficiency of image deblurring, reduces mathematical approximation errors, significantly shortens computation time, and achieves better restoration results on a variety of image datasets.
Smart Images
Figure CN117036196B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and specifically to a blind deblurring method for images based on the prior of the inner product of image gradients. Background Technology
[0002] During camera capture, unavoidable camera shake leads to motion blur. Extensive work has been dedicated to addressing motion blur, even restoring some visually indistinguishable images to high quality. However, restoring sharp images with complex blur kernels remains challenging. Traditional blind image deblurring methods fall into two categories: Variational Bayes Method and Maximum aposterior Method (MAP). While existing techniques have proposed numerous priors that effectively overcome the ill-posed problem of blind deblurring, these methods approximate the proposed priors to apply them within the MAP framework, inevitably introducing errors. Furthermore, most priors only use a first-order term of the blurred image, while the relationships between pixels in the blurred image contain blur information. Moreover, for ease of calculation, these methods often neglect the relationship between the G matrix and image i, treating it as a constant matrix for derivation and updating the G matrix accordingly. Such an approximation is not mathematically rigorous and can lead to errors. The prior method used in the image reconstruction process employs a semi-quadratic splitting method, which requires frequently calculating the inverse of the G matrix. The G matrix is usually a square matrix with the number of rows and columns equal to the number of pixels in the image. Calculating the inverse of such a large matrix will result in a long computation time. Summary of the Invention
[0003] To address the above problems, this invention provides a blind image deblurring method based on an image gradient inner product prior, which is a novel image local gradient inner product prior. This prior not only provides a more accurate approximation but also avoids the inversion process of large matrices, improving computational speed and solution efficiency.
[0004] Embodiments of the present invention provide a blind image deblurring method based on the prior of image gradient inner product, characterized in that it includes:
[0005] The formula for calculating the gradient inner product of a single point in an image is to perform a dot product operation between the gradient of each point in the image and the gradient of its neighboring points. The mathematical expression of the inner product of the local gradient of a point and the gradient of its eight neighboring pixels is as follows: Formula (1):
[0006] , (1)
[0007] The algorithm using the following formula (2) is used to calculate the local gradient inner product of all points in the image.
[0008] , (2)
[0009] Among them, ▽ x 、▽ y Let I be the matrix form of the gradient operators in the x and y directions, respectively; let P be the column vector of the image with length m, width n, and total pixels s = m × n; let P be the summation matrix of the eight neighborhoods; and let · be the vector inner product operator.
[0010] By incorporating the L1 norm of the local gradient inner product into the image deblurring framework, a new deblurring energy function is obtained:
[0011] , (3)
[0012] Since the image and blur kernel are estimated alternately during image deblurring, the energy function needs to be decomposed into two parts related to I and K:
[0013] , (4)
[0014] During the restoration process, I and K are optimized alternately;
[0015] The prior of LGP can be approximately expressed as the following formula (5):
[0016] , (5)
[0017] Where P is the matrix form corresponding to the eight-neighbor summation convolution kernel;
[0018] Taking the derivative of this prior term, we obtain the derivative of the LGP prior:
[0019] , (6)
[0020] After obtaining the derivative of the LGP prior, the energy function is differentiated, and the reconstructed image of the current round is obtained at the point where the derivative is zero. The column vector I corresponding to the image is differentiated and set to zero to obtain...
[0021] (7)
[0022] The following Fast Fourier Transform (FFT) algorithm is used to estimate the sharp image, obtain a stable deblurring result, and obtain the column vector I corresponding to the image:
[0023] (8)
[0024] Furthermore, the method also includes:
[0025] The gradient ascent method is used to apply the prior of this image to the original image deblurring framework. Specifically, the LGP term is separated, and the gradient ascent method is used to maximize the LGP term, thereby minimizing the energy function.
[0026] , (9)
[0027] Maximize the prior terms using gradient ascent.
[0028] Taking the derivative of the first term and setting it to 0, we use the FFT method to solve for the potentially sharp image I, obtaining the column vector I corresponding to the image:
[0029] , (10)
[0030] The result of g calculation is:
[0031] , (11)
[0032] The second term uses gradient ascent to optimize the latent image using the LGP term:
[0033] (12)
[0034] Where ε is the coefficient of the corresponding term in the energy function, It is the column vector corresponding to the optimized potential image.
[0035] Furthermore, in the process of estimating the fuzzy kernel K, the defuzzification energy function described in formula (3) is rewritten as follows:
[0036] (13)
[0037] Compared with the prior art, the present invention achieves the following technical effects:
[0038] 1. The blind deblurring algorithm of this invention can effectively distinguish between clear and blurred images;
[0039] 2. This invention eliminates the semi-quadratic splitting method in previous image priors, thus avoiding the process of inverting large matrices;
[0040] 3. The algorithm of this invention uses the Fast Fourier Transform method, which can greatly reduce the computational cost of adding image priors;
[0041] 4. This invention also uses the gradient ascent method, and the image prior used has a better restoration effect on blurred image datasets compared with some traditional deblurring algorithms. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of image gradients according to an embodiment of the present invention;
[0043] Figure 2 These are comparison images showing the results of restoring blurred images of real-world scenes according to an embodiment of the present invention. Detailed Implementation
[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0045] Since the gradient value of a clear image is larger and the gradient directions are closer than those of a blurry image, the gradient inner product is significantly larger than that of a blurry image. Based on this characteristic, this invention proposes a new prior for the local gradient inner product of images. This prior not only has a more accurate approximation, but also uses the relationship between image pixels in the gradient inner product term. By utilizing mathematical properties, it avoids the inversion of large matrices, thereby improving the solution efficiency.
[0046] This embodiment provides a blind image deblurring method based on the prior of image gradient inner product, including:
[0047] First, the Local Gradient Product (LGP) prior is expressed mathematically:
[0048] like Figure 1 The diagram shows the gradient vectors of each point in the image. The dots represent pixels, the arrows represent the gradient vectors of those pixels, and the angle θ represents the dot product of the vectors of points (i,j) and (i,j+1).
[0049] The formula for calculating the gradient inner product of a single point in an image is to perform a dot product operation between the gradient of each point in the image and the gradient of its neighboring points. The mathematical expression of the inner product of the local gradient of a point and the gradient of its eight neighboring pixels is as follows: Formula (1):
[0050] , (1)
[0051] The algorithm using the following formula (2) is used to calculate the local gradient inner product of all points in the image.
[0052] , (2)
[0053] Among them, ▽ x 、▽ y Let I be the matrix form of the gradient operators in the x and y directions, respectively. Let I be the column vector corresponding to the image with length m, width n, and total pixels s = m × n. Let P be the summation matrix of the eight neighborhoods, and · be the vector inner product operator.
[0054] The aforementioned Local Gradient Inner Product (LGP) prior is used to express the sharpness and continuity of image boundaries. Formula (2) above calculates the local gradient inner product for all points in the image. Compared to previous image priors, which were usually expressed as a linear product of the prior matrix and the latent image (a first-order term), this prior uses the autocorrelation information in the image, i.e., the quadratic term of the image.
[0055] The image prior proposed in this invention has a more complex and accurate mathematical expression than previous priors. To facilitate application, the L1 norm of the local gradient inner product is added to the image deblurring framework to obtain a new deblurring energy function. At the same time, we also use the semi-quadratic splitting method for the L0 term of the image gradient, and the variable g is used to replace the image gradient term.
[0056] , (3)
[0057] Except for replacing LGP prior in the second term, the other parameters are the same as in formula (2). Since the image and blur kernel are estimated alternately during the image deblurring process, the energy function needs to be decomposed into two parts about I and K:
[0058] , (4)
[0059] During the restoration process, I and K are optimized alternately.
[0060] Solve for the derivative of the LGP prior:
[0061] After analyzing the dataset images, it was found that the L1 norm of the LGP term in almost all images is positive, and all negative values are approximately equal to 0. Therefore, the LGP prior can be approximated by the following formula:
[0062] ; (5)
[0063] Where I is the column vector corresponding to an image with length m, width n, and total pixels s = m × n, and P is the matrix form corresponding to the eight-neighbor summation convolution kernel.
[0064] This transforms the original L1 norm problem into the trace of a matrix product, facilitating mathematical operations on the prior terms. To maximize the LGP value of the image, the derivative of the prior term is taken, according to:
[0065] ,
[0066] The derivative of the LGP prior can be obtained:
[0067] , (6)
[0068] LGP prior using Fast Fourier Transform (FFT):
[0069] After obtaining the derivative of the LGP prior, the energy function is differentiated, and the reconstructed image of the current round is obtained at the point where the derivative is zero. The column vector I corresponding to the image is differentiated and set to zero to obtain...
[0070] (7)
[0071] The following algorithm for estimating sharp images using Fast Fourier Transform (FFT) is used to obtain stable deblurring results:
[0072] In the experiment, β was gradually increased to make the variable g as close as possible to... In each iteration, the initial value of β is very small, and then it gradually increases. This invention gradually increases the coefficient of the LGP derivative term to a set value during optimization; experiments have shown that this approach yields stable deblurring results. The algorithm for estimating the sharp image using FFT is as follows:
[0073] ; (8)
[0074] LGP priors using gradient ascent:
[0075] Another way to introduce the prior of this invention is to apply the prior of this image to the original image deblurring framework using the gradient ascent method. Specifically, the LGP term is separated out, and the gradient ascent method is used to maximize the LGP term, thereby minimizing the energy function.
[0076]
[0077] Since the coefficients of the LGP prior are negative, the denominator may be less than 0 when using FFT optimization. This invention considers using the LGP term independently for deblurring, and the denominator of the FFT expression does not include the prior derivative. Compared to traditional image priors, the prior term proposed in this invention has a more accurate derivative, and maximizing the prior term using gradient ascent can also leverage the role of the LGP prior.
[0078] , (9)
[0079] Taking the derivative of the first term until it is zero, we use the FFT method to solve for the latent sharp image, obtaining the column vector I corresponding to the image:
[0080]
[0081]
[0082] , (10)
[0083] The result of g calculation is:
[0084] , (11)
[0085] The second term uses gradient ascent to optimize the latent image using the LGP term:
[0086] (12)
[0087] Where ε is the coefficient of the corresponding term in the energy function, It is the column vector corresponding to the optimized potential image.
[0088] Estimate the fuzzy kernel k:
[0089] In the process of estimating the fuzzy kernel K, since the latent image and fuzzy image in formula (3) can be transformed into gradient terms to achieve better computational results, the defuzzification energy function described in formula (3) is rewritten as follows:
[0090] (9)
[0091] When solving for the blur kernel, the clear image and the blurry image have a clear convolution relationship, and the blur kernel can be solved relatively easily by using the FFT method in reverse.
[0092] A test experiment was conducted on the blind deblurring algorithm based on the inner product prior of image gradient proposed in this invention.
[0093] 1. Blur image restoration effect:
[0094] The test images were restored, including faces, text, oversaturated images, and low-light images. We compared our experimental results with several state-of-the-art results. We achieved good restoration results in face images, suppressing ringing artifacts and producing smoother, clearer images compared to other methods. See [link to relevant documentation]. Figure 2 The image shows a comparison of the results of restoring blurred images in real-world scenes, where (a) is the blurred image, (b) is the DCP, (c) is the ECP, (d) is the PMP, and (e) is the LGP. This algorithm achieves good restoration results when deblurring the Kohler dataset.
[0095] 2. Calculation time:
[0096] Referring to Table 1, compared to previous deblurring algorithms, the algorithm of this invention has a significant advantage in runtime. Most algorithms add complex processing to the FFT-reconstructed image, which often involves large matrix inversion operations. This invention not only avoids large matrix inversions but also adds almost no extra computational steps. Furthermore, by using gradient ascent to extract the independent derivative terms of the prior terms, it also avoids large matrix inversions. Although it adds computational steps, it still has an advantage in runtime compared to methods using complex image priors, especially large matrix inversions.
[0097] Table 1. Comparison of computation time for different blind deblurring algorithms (unit: seconds)
[0098]
[0099] This invention proposes a blind image deblurring method based on the image gradient inner product prior, taking into account the image differences between clear and blurred images. Experimental results show that this prior not only provides a more accurate approximation but also utilizes the relationships between image pixels in the gradient inner product term. Unlike existing priors, this invention's local gradient inner product prior avoids the inversion of large matrices, simplifying computation, improving calculation speed, and theoretically reducing errors caused by approximations in mathematical expression. The algorithm uses the Fast Fourier Transform (FFT) method to significantly reduce the computational cost of adding the image prior and employs the gradient ascent method. Experiments show that this blind deblurring algorithm achieves relatively clear restoration results on several mainstream deblurring datasets, demonstrating better restoration performance.
[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A blind image deblurring method based on the prior of the inner product of image gradients, characterized in that, include: The formula for calculating the gradient inner product of a single point in an image is to perform a dot product operation between the gradient of each point in the image and the gradient of its neighboring points. The mathematical expression of the inner product of the local gradient of a point and the gradient of its eight neighboring pixels is as follows: Formula (1): （1） x5 and y5 are the x and y coordinates of the gradient vector of the center point at index 5, respectively. i and y i These are the x and y coordinates of the gradient vector at the i-th point, respectively. The algorithm using the following formula (2) is used to calculate the local gradient inner product of all points in the image. ， （2） Among them, ▽ x 、▽ y Let I be the matrix form of the gradient operators in the x and y directions, respectively; let P be the column vector of the image with length m, width n, and total pixels s = m × n; let P be the summation matrix of the eight neighborhoods; and let · be the vector inner product operator. By incorporating the L1 norm of the local gradient inner product into the image deblurring framework, a new deblurring energy function is obtained: ， （3） Where K represents the matrix form of the blur kernel, and B represents the vector form of the blurred image. , and It is the coefficient of the corresponding term in the energy function; Since the image and blur kernel are estimated alternately during image deblurring, the energy function needs to be decomposed into two parts related to I and K: ， （4） During the restoration process, I and K are optimized alternately; The prior of LGP can be approximately expressed as the following formula (5): ， （5） Where tr represents finding the trace of a matrix; Where P is the matrix form corresponding to the eight-neighbor summation convolution kernel; Taking the derivative of this prior term, we obtain the derivative of the LGP prior: ， （6） After obtaining the derivative of the LGP prior, the energy function is differentiated, and the reconstructed image of the current round is obtained at the point where the derivative is zero. The column vector I corresponding to the image is differentiated and set to zero to obtain... ，（7） in, It is the coefficient of the corresponding term in the energy function, where g represents... Alternative variables; The following Fast Fourier Transform (FFT) algorithm is used to estimate the sharp image, obtain a stable deblurring result, and obtain the column vector I corresponding to the image: （8） Where p is a matrix; Indicates Fourier transform, denoted as Inverse Fourier Transform; b is the matrix form of the blurred image, and its vector form is B; k is the matrix form of the blur kernel, and its vector form is uppercase K.
2. The image blind deblurring method based on image gradient inner product prior as described in claim 1, characterized in that, The method further includes: The gradient ascent method is used to apply the image prior to the original image deblurring framework. Specifically, the LGP term is separated, and the gradient ascent method is used to maximize the LGP term, thereby minimizing the energy function. ， （9） in, It is the coefficient of the corresponding term in the energy function; Maximize the prior terms using gradient ascent. Taking the derivative of the first term until it is zero, we use the FFT method to solve for the potential sharp image, obtaining the column vector I corresponding to the image. ， （10） The result of g calculation is: ， （11） The second term uses gradient ascent to optimize the latent image using the LGP term: （12） Where ε is the coefficient of the corresponding term in the energy function, It is the column vector corresponding to the optimized potential image.
3. A blind image deblurring method based on image gradient inner product prior as described in claim 1 or 2, characterized in that, in, In the process of estimating the fuzzy kernel K, the defuzzification energy function described in formula (3) is rewritten as follows: （13） in, It is the coefficient of the corresponding term in the energy function.