System, apparatus, and method for removing blur from an image

Abstract

Systems, apparatuses, and methods for image deblurring are presented. One method includes receiving an image and estimating a blur of the image. The method also includes applying a deblurring filter to the image and reducing halos from the image.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 114,314, filed November 16, 2020, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Unless otherwise stated herein, the materials described in this section are not prior art to the claims of this application and are not acknowledged as prior art simply because they are included in this section.

[0004] Recent advancements in image capture technology have led to a proliferation of camera devices, resulting in a significant increase in the number of digital photos produced by users. However, many cameras, such as those in cell phones and smartphones, may lack sophisticated mechanical or electronic focusing and stabilization mechanisms. Consequently, many captured images may be blurry, leading to reduced user satisfaction.

[0005] Removing blur from images has been a long-standing problem in image processing and computational photography. Blur can be caused by a variety of factors, such as the camera not being properly focused (e.g., focus misalignment), objects appearing at different depths, or relative movement between the camera and the scene. Even under perfect conditions, unavoidable physical limitations can introduce blur. For example, light diffraction due to lens aperture limitations, light integration within the sensor, and other possible lens aberrations can introduce blur, resulting in loss of detail. Furthermore, other components of the image processing pipeline itself, such as de-mosaicing and noise reduction, can also introduce blur.

[0006] Image blurring can often be modeled as a linear operator acting on a sharp latent image. If the linear operator is shift-invariant, then the blurring operation is equivalent to a convolution with a blur kernel. It is usually assumed that the captured image contains additional noise in addition to the blur. This means that...

[0007] v = k * u + n

[0008] Where v is the captured image, u is the underlying sharp image, k is the convolution with an unknown blur kernel, and n is additive noise.

[0009] Image deblurring is typically addressed using blind deconvolution methods. Most blind deconvolution methods involve a two-step process. The first step is to estimate the blur kernel. This can be done by assuming a sharp image model using a variational framework. In the second step, a non-blind deconvolution algorithm is applied. Image noise can severely hinder both steps. Therefore, a significant problem with blind deconvolution methods is that even when the blur kernel is fully known, the presence of noise and model mismatch can lead to artifacts. Summary of the Invention

[0010] Embodiments of this disclosure provide systems, methods, and apparatus for removing blur from images. It should be understood that the systems, methods, and apparatus of this disclosure can be implemented in various ways. Several embodiments of this disclosure are described below.

[0011] On one hand, a method for deblurring an image is described. This method includes receiving an image and estimating the blur of the image. The method also includes applying a deblurring filter to the image and reducing halos from the image.

[0012] On the other hand, this application describes an apparatus including a memory and one or more processors. The one or more processors can be configured to receive an image and estimate the blur of the image. The one or more processors can also be configured to apply a deblurring filter to the image and reduce halos from the image.

[0013] In another aspect, a non-transitory computer-readable medium storing instructions is disclosed, which, when executed by one or more processors, cause the processors to perform operations. These operations may include receiving an image and estimating blur in the image. These operations may also include applying a deblurring filter to the image and reducing halos in the image.

[0014] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description and with appropriate reference to the accompanying drawings. Furthermore, it should be understood that the overview and other descriptions and drawings provided herein are intended to illustrate embodiments by way of example only, and similarly, many variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated, or otherwise altered while remaining within the scope of the claimed embodiments. Attached Figure Description

[0015] Now refer to the attached diagrams and flowcharts, which are not necessarily drawn to scale.

[0016] Figure 1 An illustration depicts a computing device according to an example embodiment;

[0017] Figure 2This is a simplified block diagram of an image processing system according to an example embodiment;

[0018] Figure 3 An illustrative flowchart depicts an example method according to an example embodiment; and

[0019] Figure 4 This is a schematic diagram of a computer program according to an example implementation. Detailed Implementation

[0020] This document describes exemplary methods, apparatus, and systems. It should be understood that the terms “exemplary” and “illustrative” as used herein mean “serving as an example, illustration, or description.” Unless stated otherwise, any embodiment or feature described herein as “exemplary,” “illustrative,” and / or “illustrative” is not necessarily to be construed as superior to or superior to other embodiments or features. Therefore, other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0021] Therefore, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that, as generally described herein and shown in the accompanying drawings, aspects of this disclosure can be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0022] Furthermore, unless the context otherwise requires, the features shown in each figure can be used in combination with each other. Therefore, the figures should generally be considered as aspects of one or more overall embodiments, and it should be understood that not all features shown are necessary for every embodiment.

[0023] Additionally, the enumeration of any element, block, or step in this specification or claims is for clarity purposes only. Therefore, such enumeration should not be construed as requiring or implying that these elements, blocks, or steps follow a particular arrangement or are performed in a particular order. Unless otherwise stated, the drawings are not to scale.

[0024] The following embodiments describe systems, apparatus, and methods for eliminating blur in images. Referring now to the accompanying drawings, Figure 1 An illustration depicts a computing device 100 for removing blur from an image according to an example embodiment. The computing device 100 may be a camera, mobile device, personal computer system, desktop computer, laptop computer, notebook computer, netbook computer, mainframe computer, handheld computer, workstation, network computer, set-top box, mobile phone, video game console, handheld video game device, application server, storage device, wearable computing device (e.g., glasses equipped with a camera, head-mounted display, and / or augmented reality display), peripheral devices such as switches, modems, routers, or generally any type of computing or electronic device, etc. Figure 1 As shown, the computing device 100 may include a communication interface 102, a user interface 104, a processor 106, and a data storage device 108, all of which can be communicatively linked or coupled together via a system bus, network, or other connection mechanism 110.

[0025] The communication interface 102 of the computing device 100 can be used to allow the computing device 100 to communicate with other devices, access networks, and / or transmission networks using analog or digital modulation. Therefore, the communication interface 102 can facilitate circuit-switched and / or packet-switched communications, such as POTS communications and / or IP or other packetized communications. For example, the communication interface 102 may include a chipset and antenna arranged for wireless communication with a radio access network or access point. Furthermore, the communication interface 102 can take the form of a wired interface, such as an Ethernet, token ring, or USB port. The communication interface 102 can also take the form of a wireless interface, such as Wi-Fi. Global Positioning System (GPS) or wide-area wireless interface (e.g., WiMAX or LTE). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols can be used on communication interface 102. Furthermore, communication interface 102 may include multiple physical communication interfaces (e.g., Wi-Fi interface, etc.). Interface and wide area wireless interface).

[0026] The user interface 104 of computing device 100 can be used to allow computing device 100 to interact with human or non-human users, such as receiving input from the user and providing output to the user. User interface 104 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and / or video camera. User interface 104 may also include one or more output components such as a display (e.g., which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, light bulb, and / or other similar devices now known or developed in the future. User interface 104 may also be configured to generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices now known or developed in the future. In some embodiments, user interface 104 may include software, circuitry, or another form of logic capable of sending data to and / or receiving data from external user input / output devices. Additionally or alternatively, computing device 100 may support remote access from another device via communication interface 102 or via another physical interface (not shown).

[0027] In some examples, the user interface 104 may also serve as a viewfinder for still camera and / or video camera functions supported by the computing device 100. Additionally, the user interface 104 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of camera functions and image capture. In some implementations, some or all of these buttons, switches, knobs, and / or dials are implemented as touch or proximity-sensitive panels.

[0028] The processor 106 of the computing device 100 may include one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating-point units (FPUs), network processors, or application-specific integrated circuits (ASICs)). The processor 106 is capable of executing program instructions 118 (e.g., compiled or uncompiled program logic and / or machine code) stored in the data storage device 108 to perform the various functions described herein.

[0029] The data storage device 108 of the computing device 100 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash memory, or organic storage devices, and may be integrated wholly or partially with the processor 106. The data storage device 108 may include removable and / or non-removable components. Furthermore, the data storage device 108 may include a non-transitory computer-readable medium storing program instructions thereon that, when executed by the computing device 100, cause the computing device 100 to perform any methods, processes, techniques, and / or functions disclosed in this specification and / or the accompanying drawings. The processor 106 executing the program instructions 118 may cause the processor 106 to use data 112.

[0030] For example, program instructions 118 may include an operating system 122 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more applications 120 (e.g., address book, email, web browsing, social networking, and / or game applications) installed on computing device 100. Similarly, data 112 may include application data 114 and operating system data 116. Operating system data 116 may be primarily accessed by the operating system 122, and application data 114 may be primarily accessed by one or more applications 120. Application data 114 may be located in a file system that is visible or hidden from the user of computing device 100.

[0031] Application 120 can communicate with operating system 122 through one or more application programming interfaces (APIs). These APIs facilitate, for example, application 120 reading and / or writing application data 114, sending or receiving information via communication interface 102, receiving or displaying information on user interface 104, etc.

[0032] In some terminology, application 120 may be simply referred to as an "app". Additionally, application 120 may be downloaded to computing device 100 through one or more online app stores or app markets. However, application 120 may also be installed on computing device 100 in other ways, such as via a web browser or through a physical interface on computing device 100 (e.g., a USB port).

[0033] The computing device 100 may also include a camera assembly (not shown). The camera assembly may include, but is not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or image sensor), lens, and / or shutter button. The camera assembly may be controlled at least in part by software executed by the processor 106. Furthermore, the camera assembly may include multiple camera systems, each including an aperture, shutter, recording surface lens, image sensor, processor, and / or shutter button.

[0034] Figure 2 A simplified example block diagram of an image processing system or engine 202 for performing a blur removal process according to one embodiment is shown. The image processing system 202 may be implemented or incorporated into a camera, mobile device, personal computer system, desktop computer, laptop computer, notebook computer, netbook computer, mainframe computer, handheld computer, workstation, network computer, set-top box, mobile phone, video game console, handheld video game device, application server, storage device, wearable computing device (e.g., glasses equipped with a camera, head-mounted display, and / or augmented reality display), peripherals such as switches, modems, routers, or generally any type of computing or electronic device. The image processing system 202 may include various components, including a pre-filter or preprocessing module 204 (e.g., a denoising engine) and a blur removal module 206. The pre-filter module 204 may be an optional component (as indicated by the dashed outline). For example, in some cases, the pre-filter module 204 (e.g., a denoising engine) may be included in or used by the image processing system 202, and in other cases, it may be omitted from or not used by the image processing system 202.

[0035] The components of the image processing system 202 may include and / or be implemented using electronic circuitry or other electronic hardware. This electronic circuitry or other electronic hardware may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. For example, the pre-filter module 204 and the blur removal module 206 may be stored in a computer-readable storage medium and executed by a processor of a computer system to perform the functions described herein. Although the image processing system 202 is shown to include certain components, those skilled in the art will understand that the image processing system 202 may include more than […]. Figure 2 The image processing system 202 may include more or fewer components. For example, in some cases, the image processing system 202 may also include... Figure 2 One or more memory devices (e.g., RAM, ROM, cache, buffer, etc.) and / or processing devices not shown.

[0036] Image processing system 202 can receive an input image. The input image can be processed by pre-filter module 204 to reduce or remove noise from the image. For example, pre-filter module 204 can denoise the input image to generate a denoised image. Pre-filter module 204 may include non-local filtering to reduce or remove noise from the image. In other embodiments, pre-filter module 204 may use a denoising neural network (e.g., a convolutional neural network or other suitable neural network) to reduce or remove noise from the image. In this case, the denoising neural network can be trained with multiple noisy versions of the input image and clean versions of the same image. Using a known input (noisy image) and a known output (clean image), the denoising neural network can adjust its parameters (e.g., weights, biases, etc.) to output a clean image (denoised image) from the noisy image input to the neural network. The resulting denoised image may be noise-free or may still have some noise, but the noise can be significantly reduced by the denoising neural network.

[0037] In some embodiments, the pre-filter module 204 may use other noise removal techniques to generate a denoised image. In some examples, noise removal techniques may include AIM engine denoising-up-down, median filtering, anisotropic diffusion, geometric mean filtering, local mean filtering, bilateral filtering, block matching and three-dimensional (3D) filtering, local pixel grouping, filtering using Wiener filters, any other suitable noise filtering techniques and / or any combination thereof.

[0038] Furthermore, the pre-filter module 204 can perform operations for normalizing the image and determining its dynamic range. For example, the pre-filter module 204 can normalize the brightness of the input image so that the processed image has a predefined contrast and standard deviation value. The pre-filter module 204 can also determine whether to perform blur detection processing based on the dynamic range and noise in the image. If the image has too much noise or too low a dynamic range, the pre-filter module 204 can be configured to discard the image. If the pre-filter module determines that blur detection processing can be performed, the blur removal module 206 can perform further processing steps to remove blur from the image.

[0039] The blur removal module 206 of the image processing system 202 estimates the blur of an image, removes the blur from the input image, and reduces halos. The blur removal module 206 may include a blur estimation module 208, a deblurring module 210, and a halo removal module 212. The blur estimation module 208 can be configured to estimate the blur of an image. The blur estimation module 208 can receive an input image or a denoised image from a pre-filter module 204. The blur estimation module 208 can analyze the received image to determine the degree of blur. If the received image has too much blur, the blur estimation module 208 can discard the received image, as further described below. However, if the received image has slight blur (e.g., blur below a threshold), the blur estimation module 208 can perform operations on the received image to estimate the blur.

[0040] In one embodiment, the blur estimation module 208 can model the blur of the received image based on the parameters of a Gaussian function. It can be assumed that the image blur is uniform across the entire image, and that the image blur can be modeled using a linear model v = k * u + n, where v is the captured image, u is the underlying sharp image, k is the convolution with the unknown blur kernel, and n is additive noise. Since the blur estimation module 208 can be designed to handle mild blur where the blur kernel is expected to be small (e.g., blur below a threshold), a parametric model can be used for the blur kernel. In one embodiment, the blur kernel can be modeled using an anisotropic Gaussian function, specified by three parameters: σ0, the standard deviation of the principal axis; ρ, the ratio between the standard deviations of the principal axis and the orthogonal axis; and θ, the angle between the principal axis and the (horizontal) x-axis. Therefore, the Gaussian blur kernel at pixel (x, y) can be calculated using the following equation:

[0041] k(x,y;[a,b;b,c])=Z exp(-(ax 2 +2bxy+cy 2 ))

[0042] in:

[0043]

[0044] Furthermore, Z is a normalization factor, so its area is equal to 1. Therefore, estimating the fuzziness is equivalent to estimating three parameters (σ0, ρ, and θ).

[0045] To estimate the blur kernel of a specific image using a blur kernel model, the following assumptions can be made about the image gradient distribution. For example, it can be assumed that the maximum value of the image gradient in any direction of a sharp, natural image is essentially constant and independent of the image. It can also be assumed that if a sharp image is affected by Gaussian blur, the blur level along the principal (orthogonal) axis can be linearly related to the reciprocal of the maximum image gradient in that (orthogonal) direction. Based on these assumptions, the maximum magnitude of the image derivative can be estimated in all possible directions, and its minimum can be determined among them. This leads to the following equation:

[0046]

[0047] Where f(x) is any input image. Then,

[0048]

[0049] Where c is a coefficient that controls the assumed linear relationship between gradient features and fuzziness.

[0050] Image gradient estimation for any given image pixel may introduce additional blur (because some form of interpolation may be required). This may be a supplement to the image blur already present in the image. Assuming the blur estimation introduces isotropic Gaussian blur of σb, then the total blur of the image will be approximately... (Due to the semigroup property of Gaussian functions). This leads to:

[0051]

[0052] c and σb are two parameters that need to be calibrated.

[0053] In one implementation, parameters c and σb can be calibrated as follows: Given a set of clear, high-quality images, multiple Gaussian blurred images (e.g., K = 1000) can be simulated by randomly sampling the blur space and the image set. The Gaussian blur kernel can be generated by sampling random values ​​of σ0 ∈ [0.3, 4] and ρ ∈ [0.15, 1]. Additive white Gaussian noise of σn = 3.0 can be added to each simulated blurred image. For each blurred image, gradient features, as well as maximum and minimum values, can be sampled according to equation (9) above. Parameters c and σb can be estimated using equation (10) above. In other embodiments, other fitting schemes (e.g., robust L1 fitting, Hubber loss, weighted least squares, etc.) can be used.

[0054] In one example, the fuzzy estimation module 208 can be implemented using an algorithm to perform the following Gaussian fuzzy estimation procedure: Let the image be u, q-quantile, n angles, (a,b), (σ) max ,σ min ), (ρ max ,ρ min ) is the input, and (σ0,ρ,θ) is the output.

[0055] 1. Normalize the input image.

[0056] u=(uu min ) / (u max -u min ), where u max and u min The robust maximum and minimum values ​​are calculated using quantiles q = 0.0001 and 1-q.

[0057] 2. Calculate gradient features.

[0058] Calculate image gradient (u) x ,u y Then calculate in n angles (6) The directional derivative of uniformly covering [0, pi). This can be calculated by u. ψ =u x cos(ψ)-u y This is accomplished using sin(ψ). For each u ψ (x,y), calculate f ψ =max|u ψ |

[0059] 3. Interpolation gradient characteristics.

[0060] Perform cubic interpolation (f) on the super-resolution grid (n samples). ψ1 ,ψ1)......(f ψn ,ψ n And estimate the minimum value and the corresponding angle. Let (f0, θ0) be the minimum value, and (f1, θ0) be the minimum value. 0⊥ Direction θ 0⊥ Orthogonal direction θ 0⊥ eigenvalues.

[0061] 4. Use a regression model (e.g., Equation 10) to estimate σ0 and σ1.

[0062] 5. Verify parameters.

[0063] a.σ0=clamp(σ0,σ min ,σ max ) and σ1=clamp(σ1,σmin ,σ max ).

[0064] b. Define ρ = σ1 / σ0

[0065] c.ρ=clamp(ρ,ρ min ,ρ max 6. Return (σ0,ρ,θ0).

[0066] Furthermore, the blur estimation module 208 can calculate a blur score of the received image, which indicates the degree of blur. For example, an image received by the blur estimation module 208 can be analyzed, and a blur score can be calculated. A higher blur score indicates more severe blur in the image. Given estimated Gaussian blur parameters, the blur score can be estimated or calculated in several ways. In one example, the blur score can be determined by calculating the standard deviation of an isotropic Gaussian blur that has the same determinant of its covariance matrix as the estimated Gaussian blur. This is equivalent to defining the radius of a circle with the same area as an ellipse determined by σ0 and σ1 as the blur score. That is,

[0067]

[0068] The blur score can be compared with a predetermined threshold. If the blur score is less than the predetermined threshold, the image can be determined to have good / acceptable quality and can be further processed by the deblurring module 210. On the other hand, if the blur score is determined to be greater than the threshold, the image can be discarded.

[0069] Still referencing Figure 2 The deblurring module 210 of the image processing module 202 receives an image from the blur estimation module 208 and deblurs the image. The deblurring module 210 can apply a deblurring filter (e.g., a linear restoration filter) to the image to reduce blur. The deblurring filter can be constructed based on the inverse of the blur estimated by the blur estimation module 208. By combining different iterations of the blur kernel on the blurred image, the inverse of the blur can be approximated as follows: Let K be a convolution operator with blur kernel k, then if ||IK|| < 1, under a certain matrix norm, K -1 There exists an inverse of K, and

[0070]

[0071] If we assume the convolution has circular boundary conditions, then the geometric series of the above equation may converge, and the eigenvectors of matrix K will be Fourier modes. This implies (IK) i =F H (ID) i F, where F is a Fourier basis, F HLet F be the Hermitian transpose, and D be a diagonal matrix with eigenvalues ​​of K. Therefore, if the eigenvalues ​​of K are in D = {z ∈ C: |z⁻¹| < 1}, the series converges. However, if the fuzziness is non-invertible, i.e., the eigenvalues ​​are zero, the power series may converge to a pseudo-inverse of K. A reasonable assumption, in the case that the fuzzy filter preserves the overall brightness, is that k(x) ≥ 0 and ∫k(x)dx = 1, which means...

[0072] In one example, a fuzzy kernel can be used as a basis to approximate the inverse through a polynomial filter. For instance, if we truncate a power series of order 3, the polynomial approximation inverse of K can be expressed as:

[0073]

[0074] In addition, a general polynomial with the following equation can be used:

[0075]

[0076] Wherein, the order d and the coefficients (a0,...,a) d It can be designed to amplify or attenuate differently depending on how the blur affects a particular component.

[0077] Assuming the fuzzy filter k(x) is symmetric with k(x) = k(-x) and has non-negative Fourier coefficients, the Fourier coefficients of filter k... It can be in [0,1]. The Fourier coefficients of a polynomial filter can be correlated with the coefficients from the fundamental filter through the same polynomial, that is,

[0078]

[0079] Polynomial filters can be analyzed in the Fourier domain. Based on the interval I = [0,1] mapped by the polynomial, it is possible to determine how the attenuated Fourier components are amplified or attenuated by the polynomial filter. If a polynomial filter is applied to an image v that has been affected by the same underlying blur k as that used for the polynomial filter, the equation becomes...

[0080] p(K)v=p(K)Ku+p(K)n, (6)

[0081] It is evident that there is a trade-off between finding the inverse of the blur, i.e., p(K)K≈Id, and avoiding the amplification of noise n, i.e., p(K)n≈0. Therefore, the polynomial can be designed to (i) find the inverse of the blur effect at frequencies that are not significantly attenuated, and (ii) avoid amplifying frequencies that are significantly affected by the blur.

[0082] In one example, a polynomial filter can be built on the following: p(K)K≈Id, where K is close to the identity matrix, or, in terms of the polynomial, p(x)≈1 / x if x≈1. Note that at x=1, the d-th order Taylor expansion of 1 / x causes the polynomial filter to truncate Equation 2 to the d-th order. Therefore, this polynomial is essentially an inverse, but it significantly amplifies noise, especially the most attenuated components. To approximate the inverse operation and control the amplification of noise, a polynomial of order n can be designed, i.e., d+1 coefficients can be defined. The inverse can be approximated by forcing the derivatives of the inverse function of the polynomial at x=1 to be equal. This continues up to the d-2 order.

[0083] p (i) (x=1)=(-1) i i! , (7)

[0084] i = 1, ..., d–2, with an additional constraint p(x = 1) = 1. The remaining two degrees of freedom can be retained as design parameters. This can be achieved by controlling p. (d-1) (x=1)=α amplifies the intermediate frequency, and noise at frequencies that are completely attenuated by blurring can be amplified by controlling p(x=0)=b. This example of a d+1 linear equation produces a (closed-form) family of polynomials pol-d-α-b. The values ​​of α and b should vary in the range close to the truncated power series, i.e., α=(-1). d d!, and b = d + 1. The family of polynomial filters can be represented using orders d = 3 as follows:

[0085] p 3,α,b (x)=(α / 2-b+2)x 3 +(3b-α-6)x 2 +(5-3b+α / 2)x+b. (8)

[0086] The above analysis assumes that the blur filter k(x) is symmetric and may also not have negative Fourier coefficients. One way to avoid this is to apply a correction filter ck(x) to the image, such that the total filter is h = ck * k, but now h has the desired properties (non-negative Fourier coefficients).

[0087] In one example, a flipped kernel can be used, i.e., ck(x) = k(-x), which results in h(x) having real nonnegative Fourier coefficients. However, correction filters can introduce additional blur into the image, making deblurring more difficult. An alternative approach is to design correction filters to compensate for phase without introducing any additional spectral attenuation. This can be accomplished with pure phase filters. Therefore, the process can be implemented as follows: given k and the blurred image v, vc = ck * v and h = ck * k can be calculated. Then, the blurred image vc can be deblurred using a polynomial filter with a basic filter h.

[0088] Refer again Figure 2 The halo removal module 212 can receive the deblurred image from the deblurring module 210 and reduce halos in the image (e.g., halo artifacts). Haloes may be generated due to misestimation of the blur or, more generally, due to model mismatch. Haloes may appear at pixels in the blurred and restored images that have opposite gradients (gradient inversion).

[0089] In one example, let v(x) be a blurred image. For deblurring an image, pixels with gradient inversion are... For positive pixels. A new image u(x) can be calculated, which can be formed from the input image v(x) and the deblurred image using the following equation. Convex combination per pixel:

[0090]

[0091] Furthermore, this example could be designed to avoid halos in the final image u(x) while preserving as much deblurring as possible. If z(x) changes too quickly, then

[0092]

[0093] To avoid halos on the final image, it may be necessary to... Then, for pixels, where M(x) > 0, the equation is:

[0094]

[0095] This will result in the new image potentially lacking any gradient-inverted halo introduced in the deblurring step. To preserve as much of the deblurred image as possible (i.e., z(x), ∈ [0,1] should be as small as possible), the final image is then generated using the following method.

[0096]

[0097] Convex combinations are given in Equation 12.

[0098] After reducing or removing halos from the image, noise can be added to the image at 214. For example, the noise removed by the pre-filter module 204 can be combined with the output of halo removal to generate an output image. The output image can be a deblurred image.

[0099] In one implementation method Figure 2 The image processing system 202 can be integrated into mobile devices and can deblur 12MP images in fractions of a second using the techniques described herein. For example, the image processing system 202 can be designed for x86 and ARM architectures using Halide's programming language. Halide can decouple the algorithm implementation from its execution plan (i.e., nested loops, parallelization, etc.) to facilitate the fusion of locality in the fusion pipeline stages and leverage Single Instruction Multiple Data (SIMD) computer architectures and thread parallelism. The algorithm can run in parallel, and the parallel implementation can fully utilize the parallel processing capabilities of any suitable GPU.

[0100] In this implementation, a pre-filtering step can be applied to separate the noise-like structures from the rest of the image. If the input image is very noisy or has compression artifacts, this pre-filtering step will prevent the blur removal process from amplifying artifacts present in the image. Since the residual image is added last, this step may not need to be performed by a prior art denoiser.

[0101] Image processing system 202 can estimate image blur. First, the image is normalized using quantiles (q = 0.0001 and 1-q) to be robust to outliers. From the image gradient (u... x ;u y This allows for the calculation of uniform coverage. n angles (usually n angles =6). The direction of maximum amplitude at each angle can be found. In n angles Among the maximum values, the minimum values ​​and the angles (f0; θ0) can be obtained from the (f0; θ0) values ​​in the super-resolution grid (n samples). Ψ1 ;Ψ1)……(f Ψn Ψ n We can find it using bicubic interpolation. Using Equation 10, we can calculate σ0 and σ1, and we can calculate ρ.

[0102] In this scenario, gradient features can be efficiently computed in parallel not only between different angles but also between each different pixel. Computing the maximum value can be represented as a clustering operation, which can be optimized using shared memory and block partitioning. In the pipeline, this represents approximately 40-53% of the computation, since the maximum magnitude might require computing n... angles Second-rate.

[0103] Furthermore, the deblurring filter of the image processing system 202 can have a closed form given by estimating the blur and the polynomial coefficients of Equation 8 above. In the case of a third-order polynomial filter, the recovery filter supports approximately three times the estimated base blur. For large blur kernels, convolution can be efficiently computed in the Fourier domain. If the Gaussian blur is separable, convolution can be efficiently computed in space. In this case, it may be impossible to compute a polynomial filter. Instead, repeated applications of the Gaussian blur can be directly applied and accumulated as follows:

[0104] v0 = a0u, v i =k*v i-1 +a i u. (11)

[0105] Inseparable Gaussian filtering can also be efficiently computed using two 1D Gaussian filters on non-orthogonal axes.

[0106] Please refer to the attached diagram again. Figure 3 Example aspects of a method 300 for removing blur from an image are shown. Method 300 represents an example method that may include one or more operations as depicted by one or more boxes 302-308, each operation being... Figure 1 and Figure 2 Any of the embodiments shown, and other possible embodiments, are performed. In the example implementation, a computing device or image processing system (e.g., Figure 1 Computing device 100 or Figure 2 The image processing system 200 performs the operations shown, although in other embodiments, one or more other systems may perform some or all of the operations.

[0107] Those skilled in the art will understand that the flowcharts or diagrams described herein illustrate the functionality and operation of certain embodiments of this disclosure. In this regard, each block of a flowchart may represent a module, segment, or portion of program code, comprising one or more instructions executable by one or more processors for implementing a specific logical function or step in the process. The program code may be stored on any type of computer-readable medium, such as storage devices including disks or hard disk drives.

[0108] Furthermore, each block may represent a circuit with interconnects to perform a specific logical function during the process. Alternative implementations are included within the scope of the exemplary embodiments of this application, wherein functions may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved, as those skilled in the art will understand. In the examples, any system may cause another system to perform one or more operations (or a portion thereof) described below.

[0109] For reference Figure 3 Method 300 can be used to improve images received from an image capture device, such as a camera. For example, method 300 performs a blur removal process to estimate the blur of the image and remove the blur from the image. In one example, such as Figure 2 The image processing system shown can be used to perform image processing tasks. Figure 3 The blur removal process is illustrated. Image processing systems can be implemented or incorporated into cameras, mobile devices, personal computer systems, desktop computers, laptop computers, notebooks, netbook computers, mainframe computers, handheld computers, workstations, network computers, set-top boxes, mobile phones, video game consoles, handheld video game devices, application servers, storage devices, wearable computing devices (e.g., glasses with cameras, head-mounted displays, and / or augmented reality displays), peripherals such as switches, modems, routers, or generally any type of computing or electronic equipment.

[0110] At block 302, the image processing system can receive an image. The input image can optionally be preprocessed. Preprocessing may include image brightness normalization and determination of dynamic range and noise estimation. Images with high noise levels may be rejected. In some examples, the input image is passed through a filter (e.g., a non-local mean filter) to reduce noise in the image. In other embodiments, the image processing system may use other noise removal techniques to generate a denoised image. In some examples, noise removal techniques may include AIM engine denoising-up-down, neural networks, median filtering, anisotropic diffusion, geometric mean filtering, local mean filtering, bilateral filtering, block matching and three-dimensional (3D) filtering, local pixel grouping, filtering using Wiener filters, any combination thereof, and / or any other suitable noise filtering technique.

[0111] Furthermore, the image processing system can perform operations for image normalization and determining the image's dynamic range. For example, the image processing system can normalize the brightness of the input image so that the processed image has a predefined contrast and standard deviation value. The image processing system can also determine whether to perform blur detection processing based on the dynamic range and noise in the image. If the image has too much noise or too low a dynamic range, the image processing system can be configured to discard the image. If the image processing system determines that blur detection processing can be performed, further processing steps can be executed to remove the blur from the image.

[0112] In block 304, the image processing system can estimate the blur of the image. In one example, the image processing system can normalize the input image, compute image gradient features, interpolate gradient features, and use a regression model as described above to estimate the parameters of the Gaussian function (e.g., σ0 and σ1). Furthermore, the image processing system can compute a blur score, which indicates the degree of blur. The blur score can be compared to a predetermined threshold. If the blur score is less than the predetermined threshold, the image can be determined to have good / acceptable quality, and the image processing system can continue processing the image. On the other hand, if the blur score is determined to be greater than the threshold, the image can be discarded.

[0113] In block 306, the image processing device can deblur an image based on the estimated blur. The image processing device can apply a deblurring filter (e.g., a linear restoration filter) to the image to reduce the blur. The deblurring filter can be constructed based on the inverse of the blur estimated by the image processing system. In one example, as described above, a blur kernel can be used as a basis, and a polynomial filter can be used to approximate the inverse.

[0114] At block 308, the image processing device can reduce halo from the image. The image processing device can reduce halo by using equation (14) above. After reducing or removing halo from the image, noise can be added to the image at 214. For example, the noise removed by the pre-filtering step is combined with the output of the halo removal step to generate an output image. The output image can be a deblurred image.

[0115] It should be understood that Figure 3 The steps shown are intended to illustrate a method according to an example embodiment. Various steps can be changed or modified, the order of certain steps can be altered, and additional steps can be added, while still achieving the overall desired operation. The method can be performed by a client device, a server, or a combination of a client device and a server. The method can be performed by any suitable computing device.

[0116] This document describes an illustrative embodiment by way of example. However, those skilled in the art will understand that changes and modifications can be made to this embodiment without departing from the true scope and spirit of the elements, products, and methods targeted by this embodiment as defined in the claims.

[0117] Figure 4 This is a schematic diagram of a computer program according to an example embodiment. In some embodiments, the disclosed method may be implemented as computer program instructions encoded in a machine-readable format on a non-transitory computer-readable storage medium, or as computer program instructions encoded on other non-transitory media or articles of art.

[0118] In an example implementation, a computer program product 400 is provided using a signal bearer medium 402, which may include one or more programming instructions 404 that, when executed by one or more processors, can provide the above-mentioned... Figure 1-3 The described functions or partial functions. In some examples, signal-bearing medium 402 may comprise a non-transitory computer-readable medium 406, such as, but not limited to, hard disk drives, CDs, DVDs, digital magnetic tapes, memory, components of remote storage (e.g., in the cloud), etc. In some embodiments, signal-bearing medium 402 may comprise a computer-recordable medium 408, such as, but not limited to, memory, read / write (R / W) CDs, R / W DVDs, etc. In some embodiments, signal-bearing medium 402 may comprise a communication medium 410, such as, but not limited to, digital and / or analog communication media (e.g., optical fibers, waveguides, wired communication links, wireless communication links, etc.). Similarly, signal-bearing medium 402 may correspond to a remote storage device (e.g., the cloud). The computing system may share information with the cloud, including sending or receiving information. For example, the computing system may receive additional information from the cloud to supplement information obtained from sensors or another entity. Therefore, for example, signal-bearing medium 402 may be transmitted via a wireless communication medium 410. One or more programming instructions 404 may be, for example, computer-executable and / or logically implemented instructions. In some examples, such as Figure 1 Computing device 100 and Figure 2 The computing device of the image processing system 202 can be configured to provide various operations, functions or actions in response to one or more programming instructions 404 transmitted to the computing device from a computer-readable medium 406, a computer-recordable medium 408 and / or a communication medium 410.

[0119] Non-transitory computer-readable media can also be distributed across multiple data storage devices and / or the cloud (e.g., remotely), which can be geographically distant from each other. The computing device that executes some or all of the stored instructions can be another computing device, such as a server.

[0120] This document describes example methods, systems, and apparatuses in relation to one or more aspects related to image deblurring. Generally, any components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods can be described in the general context of executable instructions stored on computer-readable storage memory local and / or remote on a computer processing system, and implementations can include software applications, programs, functions, etc. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), or complex programmable logic devices (CPLDs).

[0121] Furthermore, the foregoing detailed description, with reference to the accompanying drawings, illustrates various features and operations of the disclosed systems, apparatus, and methods. While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be readily apparent. The various aspects and embodiments disclosed herein are for illustrative purposes and not for limitation; the true scope is indicated by the appended claims.

Claims

1. A method for removing blur from an image, comprising: Images are received by the processing device; The blur of the image is estimated by the processing device; Apply a deblurring filter to the image; as well as Reduce halos in the image; The estimation of image blur includes: Calculate the gradient features of the image; Calculate the directional derivative of each of the multiple angles based on gradient features; Calculate the maximum magnitude of the directional derivative for each of the multiple angles; Choose the minimum value from the maximum magnitude of the directional derivative; and Image blur estimation based on minimum value.

2. The method of claim 1 further includes removing noise from the image.

3. The method according to claim 1, wherein, Estimating fuzziness also includes determining the fuzziness function.

4. The method according to claim 1, wherein, The fuzziness is estimated based on at least one parameter of the Gaussian function.

5. The method according to claim 4, wherein, The Gaussian function includes an isotropic Gaussian function.

6. The method according to claim 1 further includes normalizing the image after receiving the image.

7. The method of claim 1, further comprising estimating a blur score of the image, the blur score indicating the degree of blur in the image.

8. The method of claim 7, further comprising comparing the fuzzy score with a threshold.

9. The method of claim 1, further comprising combining multiple iterations of the fuzz to estimate the inverse of the fuzz.

10. The method of claim 1, further comprising using a polynomial based on a fuzzy kernel to estimate the inverse of the fuzz.

11. The method according to claim 1, wherein, The deblurring filter includes a polynomial filter.

12. The method according to claim 11, wherein, The polynomial filter is based on the inverse of the estimated fuzz.

13. The method according to claim 1, wherein, The deblurring filter includes a linear recovery filter constructed from the estimated blur.

14. An apparatus for removing blur from an image, comprising: Memory; as well as One or more processors are configured as follows: Receive image; Estimate the blur of the image; Apply a deblurring filter to the image; as well as Reduce halos in the image; The estimation of image blur includes: Calculate the gradient features of the image; Calculate the directional derivative of each of the multiple angles based on gradient features; Calculate the maximum magnitude of the directional derivative for each of the multiple angles; Choose the minimum value from the maximum magnitude of the directional derivative; and Image blur estimation based on minimum value.

15. A non-transitory computer-readable medium storing instructions executable by one or more processors to perform the following functions: Receive image; Estimate the blur of the image; Apply a deblurring filter to the image; as well as Reduce halos in the image; The estimation of image blur includes: Calculate the gradient features of the image; Calculate the directional derivative of each of the multiple angles based on gradient features; Calculate the maximum magnitude of the directional derivative for each of the multiple angles; Choose the minimum value from the maximum magnitude of the directional derivative; and Image blur estimation based on minimum value.

Images