SYSTEM, APPARATUS AND METHOD FOR REMOVING BLUER IN IMAGES - Patent application

Abstract

A system, apparatus, and method for deblurring an image are presented. One method includes receiving an image and estimating blur for the image. The method also includes applying a deblurring filter to the image and reducing halo from the image.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 114,314, filed November 16, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Unless otherwise specified herein, the material described in this section is not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] Recent developments in image capture technology have led to a proliferation of camera devices, thereby greatly increasing the number of digital photographs created by users. However, many cameras, such as those in mobile phones and smartphones, may not have advanced mechanical or electronic focusing and stabilization mechanisms. As a result, many of the captured images may be blurred, reducing user satisfaction with those images.

[0004] Removing blur from images has been a long-standing challenge in image processing and computational photography. Blur can occur due to a number of factors, such as when a camera is not properly focused (e.g., out of focus), when objects appear at different depths, or when there is relative motion between the camera and the scene. Even in perfect conditions, there may be unavoidable physical constraints that cause blur. For example, diffraction of light due to limited lens aperture, light accumulation at the sensor, and other possible lens aberrations can cause blurring, leading to loss of detail. Additionally, other components of the image processing pipeline itself, such as demosaicing and denoising, can cause blurring.

[0005] Image blurring can generally be modeled as a linear operator acting on a sharp latent image. If the linear operator is shift-invariant, the blurring operation becomes a convolution with a blur kernel. It is often assumed that the captured image will have additive noise on top of the blur. This is v=k*u+n where v is the captured image, u is the underlying sharp image, k is the convolution with an unknown blur kernel, and n is the additive noise.

[0006] Image deblurring has traditionally been addressed by blind deconvolution approaches. Most blind deconvolution methods involve a two-step process. In the first step, the blur kernel is estimated. This can be done by assuming a sharp image model using a variational framework. In the second step, a non-blind deconvolution algorithm is applied. Image noise can seriously impede both steps. Thus, a significant challenge of blind deconvolution approaches is that even if the blur kernel can be perfectly known, the presence of noise and model mismatch can cause artifacts. Summary of the Invention [Means for solving the problem]

[0007] The embodiments of the present disclosure provide a system, method, and apparatus for removing blur in an image. It should be understood that the systems, methods, and apparatus of the present disclosure can be implemented in numerous ways. Several embodiments of the present disclosure are described below.

[0008] In one aspect, a method for deblurring an image is described. The method includes receiving an image and estimating blur for the image. The method also includes applying a deblurring filter to the image and reducing halo from the image.

[0009] In another aspect, the present application describes an apparatus that includes a memory and one or more processors. The one or more processors can be configured to receive an image and estimate blur for the image. The one or more processors can also be configured to apply a deblurring filter to the image to reduce halo from the image.

[0010] In yet another aspect, a non-transitory computer-readable medium is disclosed that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations may include receiving an image and estimating blur for the image. The operations also include applying a deblurring filter to the image and reducing halo from the image.

[0011] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description, with reference, where appropriate, to the accompanying drawings. Moreover, it should be understood that this summary and the other descriptions and figures provided herein are intended to illustrate embodiments by way of example only, and thus, numerous modifications are possible. For example, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise altered while remaining within the scope of the claimed embodiments.

[0012] Reference is now made to the accompanying drawings and flow charts, which are not necessarily drawn to scale. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an illustration of a computing device in accordance with an example embodiment. [Diagram 2] FIG. 1 is a simplified block diagram of an image processing system in accordance with an example embodiment. [Diagram 3] 1 is an illustrative flow diagram of an example method according to an example embodiment. [Figure 4] FIG. 2 is a schematic diagram of a computer program according to an exemplary implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Exemplary methods, devices, and systems are described herein. As used herein, the terms "example" and "exemplary" should be understood to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example," "exemplary," and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features, unless so stated. Thus, other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0015] Accordingly, the exemplary embodiments described herein are not meant to be limiting, as it will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the Figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0016] Moreover, unless the context suggests otherwise, the features illustrated in each of the drawings may be used in combination with one another, and the drawings should thus generally be considered as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for each embodiment.

[0017] Additionally, any enumeration of elements, blocks, or steps in the specification or claims is for purposes of clarity, and such enumeration should not be construed as requiring or implying that those elements, blocks, or steps are limited to a particular arrangement or performed in a particular order. Unless otherwise noted, the drawings are not drawn to scale.

[0018] The following embodiments describe systems, apparatus, and methods for removing blur in images. Referring now to the drawings, FIG. 1 shows an illustration of a computing device 100 for removing blur in images according to an exemplary implementation. The computing device 100 may be a camera, a mobile device, a personal computer system, a desktop computer, a laptop, a notebook, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a set-top box, a mobile phone, a video game console, a handheld video game device, an application server, a storage device, a wearable computing device (e.g., camera glasses, a head-mounted display, and / or an augmented reality display), a switch, a modem, a router, or generally any type of computing or electronic device, etc. As shown in FIG. 1, the computing device 100 may include a communication interface 102, a user interface 104, a processor 106, and a data storage 108, all of which may be communicatively linked or coupled to each other by a system bus, a network, or other connection mechanism 110.

[0019] The communication interface 102 of the computing device 100 may function to enable the computing device 100 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 102 may facilitate circuit-switched and / or packet-switched communications, such as POTS communications and / or IP or other packetized communications. For example, the communication interface 102 may include a chipset and antenna arranged for wireless communication with a radio access network or access point. The communication interface 102 may also take the form of a wireline interface, such as an Ethernet, token ring, or USB port. The communication interface 102 may also take the form of a wireless interface, such as Wifi, BLUETOOTH, Global Positioning System (GPS), or a wide area wireless interface (e.g., WiMAX or LTE). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over the communication interface 102. Additionally, the communication interface 102 may comprise multiple physical communication interfaces (eg, a Wifi interface, a BLUETOOTH interface, and a wide area wireless interface).

[0020] The user interface 104 of the computing device 100 may function to enable the computing device 100 to interact with a human or non-human user, such as receiving input from the user and providing output to the user. The user interface 104 may include input components such as a keypad, keyboard, a touch-sensitive or presence-sensitive panel, a computer mouse, a trackball, a joystick, a microphone, a still camera, and / or a video camera. The user interface 104 may also include one or more output components, such as a display screen (e.g., which may be combined with a touch-sensitive panel), a display using CRT, LCD, LED, DLP technology, a printer, a light bulb, and / or other similar technologies now known or later developed. The user interface 104 may also be configured to generate audible output via a speaker, a speaker jack, an audio output port, an audio output device, an earphone, and / or other similar devices now known or later developed. In some embodiments, user interface 104 may include software, circuitry, or another form of logic capable of transmitting data to and / or receiving data from an external user via input / output devices. Additionally or alternatively, computing device 100 may support remote access from another device via communications interface 102 or via another physical interface (not shown).

[0021] In some examples, the user interface 104 may also function as a viewfinder for still camera and / or video camera functions supported by the computing device 100. Additionally, the user interface 104 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and focusing of the camera functions and capturing images. In some implementations, some or all of these buttons, switches, knobs, and / or dials are implemented as functions on a touch or proximity sensitive panel.

[0022] The processor 106 of the computing device 100 may comprise one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application specific integrated circuits (ASICs)). The processor 106 may be capable of executing program instructions 118 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 108 to perform various functions described herein.

[0023] The data storage 108 of the computing device 100 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integral in whole or in part with the processor 106. The data storage 108 may include removable and / or non-removable components. Additionally, the data storage 108 may include a non-transitory computer-readable medium having program instructions stored thereon that, when executed by the computing device 100, cause the computing device 100 to perform any of the methods, processes, techniques, and / or functions disclosed herein and / or in the accompanying drawings. Execution of the program instructions 118 by the processor 106 results in the processor 106 using the data 112.

[0024] By way of example, program instructions 118 may include an operating system 122 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more application programs 120 (e.g., address book, email, web browsing, social networking, and / or gaming applications) installed on computing device 100. Similarly, data 112 may include application data 114 and operating system data 116. The operating system data 116 may be primarily accessible to the operating system 122, and the application data 114 may be primarily accessible to one or more of the application programs 120. The application data 114 may be located in a file system that is visible or hidden from a user of computing device 100.

[0025] Application programs 120 may communicate with operating system 122 via one or more application programming interfaces (APIs). These APIs may, for example, facilitate application programs 120 reading and / or writing application data 114, sending or receiving information via communications interface 102, receiving or displaying information on user interface 104, etc.

[0026] In some cases, application program 120 may be referred to as an "app" for short. Additionally, application program 120 may be downloadable to computing device 100 through one or more online application stores or application marketplaces. However, application program 120 may also be installed on computing device 100 in other ways, for example, via a web browser or through a physical interface on computing device 100 (e.g., a USB port).

[0027] Computing device 100 may also include a camera component (not shown). The camera component may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, and / or a shutter button. The camera component may be controlled, at least in part, by software executed by processor 106. Additionally, the camera component may include multiple camera systems, each including an aperture, a shutter, a recording surface, a lens, an image sensor, a processor, and / or a shutter button.

[0028] 2 is an illustration of an exemplary simplified block diagram of an image processing system or engine 202 for performing a deblurring process, according to one embodiment. The image processing system 202 may be implemented by or integrated with a camera, a mobile device, a personal computer system, a desktop computer, a laptop, a notebook, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a set-top box, a mobile phone, a video game console, a handheld video game device, an application server, a storage device, a wearable computing device (e.g., camera-equipped glasses, a head-mounted display, and / or an augmented reality display), a peripheral device such as a switch, a modem, a router, or generally any type of computing or electronic device. The image processing system 202 may include various components, including a pre-filter or pre-processing module 204 (e.g., a denoising engine) and a deblurring module 206. The pre-filter module 204 may be an optional component (as indicated by the dashed outline). For example, the pre-filter module 204 (e.g., a noise reduction engine) may in some cases be included in or used by the image processing system 202 and in other cases may be excluded from or not used by the image processing system 202.

[0029] The components of the image processing system 202 may include and / or be implemented with electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits) and / or the components of the image processing system 202 may include and / or be implemented with computer software, firmware, or any combination thereof to perform various operations described herein. For example, the pre-filter module 204 and the deblur module 206 may be stored in a computer-readable storage medium and executed by a processor of a computer system to perform the functions described herein. Although the image processing system 202 is shown to include certain components, one skilled in the art will appreciate that the image processing system 202 may include more or fewer components than those shown in FIG. 2. For example, the image processing system 202 may in some cases include one or more memory devices (e.g., RAM, ROM, cache, buffers, and / or the like) and / or processing devices not shown in FIG. 2.

[0030] The image processing system 202 may receive an input image. The input image may be processed by the pre-filter module 204 to reduce or remove noise from the image. For example, the pre-filter module 204 may perform denoising of the input image to generate a noise-reduced image. The pre-filter module 204 may include non-local mean filtering to reduce or remove noise from the input image. In other embodiments, the pre-filter module 204 may use a denoising neural network (e.g., a conventional neural network or other suitable neural network) to remove or reduce noise from the input image. In such cases, the denoising neural network may be trained by inputting multiple noisy versions of some images along with clean versions of the same images. With known inputs (noisy images) and known outputs (clean images), the denoising neural network may tune its parameters (e.g., weights, biases, etc.) to output a clean image (noise-reduced image) from the noisy images input to the neural network. The resulting noise-reduced image may have no noise or may still have some noise, but the noise can be significantly reduced by the denoising neural network.

[0031] In some embodiments, the pre-filter module 204 may also use other denoising techniques to generate a noise-reduced image. In some examples, the denoising techniques may include denoising-upscale-downscale with an AIM engine, median filtering, anisotropic diffusion, geometric mean filtering, local mean filtering, bilateral filtering, block matching and three-dimensional (3D) filtering, local pixel grouping, filtering with a Wiener filter, any other suitable noise filtering technique, and / or any combination thereof.

[0032] Furthermore, the pre-filter module 204 may perform operations to normalize the image to determine the dynamic range of the image. For example, the pre-filter module 204 may normalize the luminance of the input image so that the processed image has a predefined contrast and standard deviation value. The pre-filter module 204 may further determine whether to perform a blur detection process based on the dynamic range and noise in the image. If the image has too much noise or too low dynamic range, the pre-filter module 204 may be configured to discard the image. If the pre-filter module determines that a blur detection process can be performed, further processing steps may be performed by the deblur module 206 to remove blur from the image.

[0033] The deblurring module 206 of the image processing system 202 estimates the blur of an image and removes the blur of an input image to reduce halos. The deblurring module 206 may include a blur estimation module 208, a blur correction module 210, and a halo removal module 212 of the deblurring module 206. The blur estimation module 208 may be configured to estimate the blur of an image. The blur estimation module 208 may receive an input image or may receive a noise-reduced image from the prefilter module 204. The blur estimation module 208 may analyze the received image to determine the degree of blurriness. If the received image has too much blur, the blur estimation module 208 may discard the received image, as further described below. However, if the received image has mild blur (e.g., the blur is below a threshold), the blur estimation module 208 may perform an operation on the received image to estimate the blur.

[0034] In one embodiment, the blur estimation module 208 can model the blur of the received image based on the parameters of a Gaussian function. It may be assumed that the image blur is uniform throughout the image and can be modeled by a linear model v=k*u+n, where v is the captured image, u is the underlying sharp image, k is the convolution with the unknown blur kernel, and n is the additive noise. The blur estimation module 208 may be designed to handle mild blur (e.g., blur below a threshold) where the blur kernel is expected to be small and a parametric model can be used for the blur kernel. Therefore, in one implementation, the blur kernel may be modeled using an anisotropic Gaussian function specified with three parameters: σ0, the standard deviation of the main axis; ρ, the ratio between the standard deviations in the principal and orthogonal axes; and θ, the angle between the major axis and the (horizontal) x-axis. Thus, the Gaussian blur kernel at pixel (x,y) can be calculated by the following equation:

[0035]

number

[0036] Where:

[0037]

number

[0038] and Z is a normalization factor that makes the area equal to 1. Thus, estimating the blur reduces to estimating three parameters: σ0, ρ, and θ.

[0039] To estimate the blur kernel for a particular image using the blur kernel model, the following assumptions may be made regarding the image gradient distribution of the image. For example, it may be assumed that the maximum value of the image gradient in any direction for a clear natural image may be substantially constant and independent of the image. It may also be assumed that if a clear image is affected by Gaussian blur, the blur level in the direction of a principal (orthogonal) axis may be linearly related to the inverse of the maximum image gradient in that (orthogonal) direction. Based on these assumptions, the maximum magnitude of the image derivative may be estimated in all possible orientations, and the minimum among them may be determined. This results in the following equation:

[0040]

number

[0041] where f(x) is an arbitrary input image, and

[0042]

number

[0043] where c is a coefficient that controls the assumed linear relationship between the gradient feature and the level of blur.

[0044] For any given image pixel, the estimation of the image gradient may result in additional blurring (because some type of interpolation may be required). This may be in addition to the image blurring already contained in the image. b Assuming that the image is blurred with an isotropic Gaussian blur of

[0045]

number

[0046] (due to the semigroup property of the Gaussian function). This gives us

[0047]

number

[0048] where c and σ b are the two parameters that need to be calibrated.

[0049] In one implementation, the parameters c and σ b can be calibrated as follows: Given a set of sharp, high quality images, several Gaussian blurred images (e.g., K=1000) can be simulated by randomly sampling the blur space and image pairs. The Gaussian blur kernel can be generated by sampling random values ​​for σ0 in [0.3,4] and ρ in [0.15,1]. σ n Additive Gaussian white noise with a magnitude of σ = 3.0 may be added to each simulated blurred image. For each blurred image, the gradient feature and the maximum and minimum values ​​may be sampled according to equation (9) above. The parameters c and σ bmay be estimated by equation (10) above. In other embodiments, other fitting schemes may be used (e.g., robust L1 fitting, Hubber loss, weighted least squares, etc.).

[0050] In one example, the blur estimation module 204 may be implemented using an algorithm such as the following to perform a Gaussian blur estimation procedure: max ,σ min ), (ρ max ,ρ min ) is the input and the parameters (σ0,ρ,θ) are the output. 1.Normalize the input image. u=(uu min ) / (u max -u min ), where u max and u min is computed as the robust maximum and minimum taking quantiles q=0.0001 and 1-q. 2. Compute the gradient features. Image gradient (u x ,u y ), and then calculate n angles Calculate the directional derivative in (6). This is called u ψ = u x cos(ψ)-u y This can be done by calculating sin(ψ). ψ For (x,y), f ψ =max|u ψ Calculate |. 3. Interpolate the gradient features. On a super-resolution grid (n samples), (f ψ1 ,ψ1),...,(f ψn , ψ n ) is cubically interpolated to estimate the minimum value and each angle. Set (f0,θ0) to the minimum value and (f1,θ 0⊥ ) in the orthogonal direction θ 0⊥ Direction θ to the feature value in 0⊥ Let us assume that. 4. Estimate σ0 and σ1 using a regression model (e.g., equation (10)). 5. Enable the parameter. a.σ0 = clamp(σ0,σ min ,σ max ) and σ1=clamp(σ1,σ min ,σ max ) b.Define ρ=σ1 / σ0. c.ρ = clamp(ρ,ρ min ,ρ max ) 6. Return (σ0,ρ,θ0).

[0051] Further, the blur estimation module 208 may calculate a blur score indicating the degree of blur for the received image. For example, the received image may be analyzed by the blur estimation module 208, and the blur score is calculated. A higher blur score may indicate a greater degree of blur in the image. Given the estimated Gaussian blur parameters, the blur score may be estimated or calculated in several ways. In one example, the blur score may be determined by calculating the standard deviation of an isotropic Gaussian blur that has the same determinant of the covariance matrix as the estimated Gaussian. This amounts to defining the blur score as the radius of a circle that has the same area as the ellipse determined by σ0 and σ1. That is, as follows:

[0052]

number

[0053] The blur score may be compared to a predefined threshold, and if the blur score is less than the predefined threshold, it may be determined that the image is of good / acceptable image quality and may be further processed by the deblurring module 210. On the other hand, if the blur score is greater than the threshold, it may be determined that the image may be discarded.

[0054] Still referring to FIG. 2, the deblurring module 210 of the image processing system 202 receives the image from the blur estimation module 208 and deblurs the image. The deblurring module 210 may apply a deblurring filter (e.g., a linear restoration filter) to the image to reduce blur. The deblurring filter may be constructed based on the inverse of the blur estimated by the blur estimation module 208. By combining different iterations of the blur kernel on the blurred image, the inverse of the blur may be approximated as follows: Let K be the convolution operator with the blur kernel k, then if ||IK||<1 under some matrix norm, then K, the inverse of K, is -1 exists,

[0055]

number

[0056] The geometric series of the above equations can converge if we assume circular boundary conditions for the convolution, so that the eigenvectors of matrix K are Fourier modes. This gives us the Fourier modes (IK) i =F H (ID) i We obtain F, where F is the Fourier basis and F H is the Hermitian transpose of F, and D is a diagonal matrix using the eigenvalues ​​of K. Thus, this series converges if the eigenvalues ​​of K are in D={z∈C:|z-1|<1}. However, if the blur is not invertible, i.e., if the eigenvalues ​​are zero, then this power series may converge to the pseudoinverse of K. In the case of a blur filter that preserves overall luminosity, a reasonable assumption is that k(x) ≥ 0 and

[0057]

number

[0058] This results in the following:

[0059]

number

[0060] In one example, the inverse matrix can be approximated by a polynomial filter using the blurring kernel as a basis. For example, when truncating a power series of degree 3, the polynomial inverse of K can be expressed as:

[0061]

number

[0062] Additionally, general polynomials may be used, such as the following equation:

[0063]

number

[0064] Here, the degree d and the coefficients (a0,..., a d ) can be designed to amplify or attenuate differently depending on how the blur is affecting a particular component.

[0065] Assuming that the blurring filter k(x) is symmetric, i.e., k(x)=k(-x), and has non-negative Fourier coefficients, then the Fourier coefficients of filter k are

[0066]

number

[0067] can be in [0,1]. The Fourier coefficients of the polynomial filter can be related to the Fourier coefficients from the base filter through the same polynomial, i.e.

[0068]

number

[0069] A polynomial filter can be analyzed in the Fourier domain. Based on the interval I=[0,1] mapped by the polynomial, it can be determined how the attenuated Fourier components can be amplified or attenuated by the polynomial filter. If a polynomial filter is applied to an image v affected by the same base blur k used for the polynomial filter, the equation becomes: p(K)v=p(K)Ku+p(K)n (6) Here, it may become apparent that there is a trade-off between inverting the blur, i.e., p(K)K ≈ Id, and avoiding n-fold amplification of the noise, i.e., p(K)n ≈ 0. Thus, a polynomial may be designed to (i) invert the effect of blurring at frequencies that are not significantly attenuated, and (ii) avoid amplification of frequencies that are significantly affected by blurring.

[0070] In one example, a polynomial filter can be constructed based on p(K)K≈Id if K is close to the identity matrix, or as a polynomial p(x)≈1 / x if x≈1. Note that a Taylor expansion of order d of 1 / x at x=1 truncates the polynomial filter to order d in equation (2). In this way, this polynomial effectively approximates the inverse matrix, but significantly amplifies noise, especially in the most attenuated components. To approximate the inverse matrix and control the amplification of noise, a polynomial of order d can be designed, i.e., d+1 coefficients can be defined. The inverse matrix can be approximated by forcing the polynomial to have a derivative equal to the derivative of the inverse function at x=1. This can be approximated to order d-2, i.e., p (i) (x=1)=(-1) i i! (7) with i=1,...,d-2 and the additional constraint that p(x=1)=1. The remaining two degrees of freedom can be left as design parameters. The intermediate frequency is p (d-1)The noise can be amplified by controlling p(x=1)=α at frequencies that are completely attenuated by blurring, and the noise can be amplified by controlling p(x=0)=b at frequencies that are completely attenuated by blurring. This example of d+1 linear equations leads to a (closed form) family of polynomials, pol-d-α-b. The values ​​of α and b should vary within a range close to that given by the truncated power series, i.e., α=(-1) d d! and b=d+1. The family of polynomial filters can be presented as follows, of order d=3: p 3,α,b (x)=(α / 2-b+2)x 3 +(3b-α-6)x 2 +(5-3b+α / 2)x+b (8)

[0071] The above analysis assumes that the blurring filter k(x) is symmetric and furthermore cannot have negative Fourier coefficients. One way to get around this is to make the entire filter h=c k *k, but h has the desired properties (its Fourier coefficients are non-negative). k This may be by applying (x).

[0072] In one example, the flipped kernel, i.e., c k (x)=k(-x) can be used, which results in the actual non-negative Fourier coefficients

[0073]

number

[0074] However, the correction filter may introduce additional blurring into the image, making the deblurring task even more difficult. An alternative approach could be to design the correction filter to compensate for the phase but without introducing any additional attenuation into the spectrum. This is similar to a pure phase filter

[0075]

number

[0076] Thus, the procedure can be implemented as follows: given k and a blurred image v, c =c k *v and h=c k Then, the polynomial filter can be used together with the base filter h to compute the blurred image v c can be deblurred.

[0077] 2, the halo removal module 212 may receive the blur-corrected image from the blur correction module 210 and reduce halos (e.g., halo artifacts) in the image. Halos may be generated due to misestimation of blur or generally due to model mismatch. Halos may occur at pixels where the blurred and restored images have opposite gradients (reversal of gradient).

[0078] In one example, let v(x) be a blurred image, and v - Let (x) be the deblurred image. A pixel with gradient reversal is one that

[0079]

number

[0080] The input image v(x) and the deblurred image v(x) are pixels where - A new image u(x) can be formed as a pixel-wise convex combination with (x) using the equation

[0081]

number

[0082] It can be calculated by:

[0083] Furthermore, this example avoids halos in the final image u(x), but with deblurred

[0084]

number

[0085] can be designed to retain as much of z(x) as possible. If z(x) changes too fast, we get

[0086]

number

[0087] To avoid halos in the final image,

[0088]

number

[0089]

number

[0090] will result in a new image that cannot have any gradient reversal halo introduced in the deblurring step. To keep as much of the deblurred image (i.e., z(x)) as possible, ∈ [0, 1] should be as small as possible. Then, the final image is given by the convex combination given by equation (12):

[0091]

number

[0092] It is generated by employing

[0093] After the halo may be reduced or removed from the image, noise may be added to the image at 214. For example, the noise removed by the pre-filter 204 may be combined with the output of the halo removal to generate an output image. The output image may be a deblurred image.

[0094] In one implementation, the image processing system 202 of FIG. 2 may be integrated into a mobile device and may deblur a 12MP image in approximately a fraction of a second using the techniques described herein. For example, the image processing system 202 may be designed for X86 and ARM architectures using a programming language called Halide. Halide allows for local fuse pipeline stages to be fused more easily and may separate the implementation of an algorithm from its execution schedule (i.e., loop nesting, parallelization, etc.) to take advantage of single instruction stream multiple data stream (SIMD) computer architectures and thread parallelism. The algorithm may run in parallel, and the parallel implementation may fully utilize the parallel processing power of any suitable GPU.

[0095] In this implementation, a pre-filter step may be applied that separates noise-like structures from the rest of the image. If the input image is very noisy or has compression artifacts, this pre-filter step will avoid the deblurring process to amplify the artifacts present in the image. Since the residual image is added at the end, this step may not need to be performed by state-of-the-art denoising mechanisms.

[0096] The image processing system 202 can estimate the blur of an image. First, the image is normalized using quantiles (q=0:0001 and 1-q) to be robust to outliers. The image gradient (u x ;u y ) to find n angles (usually n angles= 6), the directional derivatives can be calculated. The direction of maximum magnitude for each angle can be found. angles Among the maxima, the minimum and angle (f0;_0) can be found through bicubic interpolation of (f1;1);:::;(fn;n) (n samples) in the super-resolution grid. Using equation (10), σ0 and σ1 can be calculated, and ρ can be calculated.

[0097] In this case, gradient features can be computed efficiently in parallel, not only across different angles, but also across each different pixel. Computing the maximum can be represented as an aggregation operation that can be optimized using shared memory and tiling. Inside the pipeline, this represents about 40-53% of the computation, since the maximum magnitude is n angles This is because it can be calculated multiple times.

[0098] Furthermore, the deblurring filter of the image processing system 202 may have a closed form given by the estimated blur and the polynomial coefficients of equation (8) above. In the case of a third order polynomial filter, the support of the reconstruction filter is roughly three times the support of the estimated base blur. In the case of a large blur kernel, the convolution can be efficiently calculated in the Fourier domain. In the case of a separable Gaussian blur, the convolution can be efficiently calculated in space. In this case, a polynomial filter cannot be calculated. Rather, repeated applications of the Gaussian blur can be performed directly and accumulated as follows: v0=a0u, v i =k*v i-1 +a i u (11)

[0099] Non-separable Gaussian filtering can also be efficiently computed by two 1D Gaussian filters in non-orthogonal axes.

[0100] Referring again to the drawings, Figure 3 illustrates an example aspect of a method 300 for removing blur from an image. Method 300 represents an example method that may include one or more operations illustrated by one or more blocks 302-308, each of which may be performed by any of the embodiments illustrated in Figures 1 and 2, among other possible embodiments. In an example implementation, a computing device or image processing system (e.g., computing device 100 of Figure 1 or image processing system 202 of Figure 2) performs the illustrated operations, although in other implementations, one or more other systems may perform some or all of these operations.

[0101] Those skilled in the art will appreciate that the flow charts or flowcharts described herein illustrate the functions and operations of certain implementations of the present disclosure. In this regard, each block of the flow chart may represent a module, segment, or portion of program code, and includes one or more instructions executable by one or more processors to implement a particular logical function or step in a process. The program code may be stored on any type of computer-readable medium, such as a storage device including, for example, a disk or hard drive.

[0102] Further, each block may represent circuitry that is hardwired to perform a particular logical function in the process. Alternative implementations are included within the scope of the exemplary embodiments of the present application, in which functions may be performed out of the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved, as would be understood by one of ordinary skill in the art. In some examples, either system may cause another system to perform one or more of the operations (or a portion of the operations) described below.

[0103] Referring now to Figure 3, a method 300 may be used to improve an image received from an image capture device such as a camera. For example, the method 300 estimates blur for an image and performs a deblurring process to remove the blur from the image. In one example, an image processing system such as the image processing system illustrated in Figure 2 may be used to perform the deblurring process illustrated in Figure 3. The image processing system may be implemented by or integrated with a camera, a mobile device, a personal computer system, a desktop computer, a laptop, a notebook, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a set-top box, a mobile phone, a video game console, a handheld video game device, an application server, a storage device, a wearable computing device (e.g., camera glasses, a head mounted display, and / or an augmented reality display), a peripheral device such as a switch, a modem, a router, or generally any type of computing or electronic device.

[0104] In block 302, an image may be received by an image processing system. The input image may be optionally pre-processed. Pre-processing may include normalizing image brightness and determining dynamic range and noise estimates. Images with a high degree of noise may be rejected. In some examples, the input image is passed through a filter (e.g., a non-local mean filter) to reduce noise in the image. In other embodiments, the image processing system may use other denoising techniques to generate a noise-reduced image. In some examples, the denoising techniques may include denoising-upscale-downscale with an AIM engine, neural networks, median filtering, anisotropic diffusion, geometric mean filtering, local mean filtering, bilateral filtering, block matching and three-dimensional (3D) filtering, local pixel grouping, filtering with a Wiener filter, any combination thereof, and / or any other suitable noise filtering techniques.

[0105] Furthermore, the image processing system may perform operations to normalize the image to determine the dynamic range of the image. For example, the image processing system may normalize the luminance of the input image so that the processed image has predefined contrast and standard deviation values. The image processing system may further determine whether to perform a blur detection process based on the dynamic range and noise in the image. If the image has too much noise or too low dynamic range, the image processing system may be configured to discard the image. If the image processing system determines that a blur detection process can be performed, further processing steps may be performed to remove blur from the image.

[0106] In block 304, the image processing system may estimate the blur of the image. In one example, the image processing system may normalize the input image, calculate image gradient features, interpolate the gradient features, and estimate the parameters of the Gaussian function (e.g., σ0 and σ1) using a regression model as described above. Additionally, the image processing system may calculate a blur score of the image indicating the degree of blurriness. The blur score may be compared to a predefined threshold. If the blur score is less than the predefined threshold, the image may be determined to have good / acceptable image quality, and the image processing system may continue to process the image. On the other hand, if the blur score is determined to be greater than the threshold, the image may be discarded.

[0107] In block 306, the image processing system may deblur the image based on the estimated blur. The image processing system may apply a deblurring filter (e.g., a linear restoration filter) to the image to reduce the blur. The deblurring filter may be constructed based on the inverse of the blur estimated by the image processing system. In one example, this inverse may be approximated by a polynomial filter using a blur kernel as a basis, as described above.

[0108] At block 308, an image processing system may reduce the halo from the image. The image processing system may reduce the halo by using equation (14) above. After the halo is reduced or removed from the image, noise may be added to the image at 214. For example, the noise removed by the pre-filter step may be combined with the output of the halo removal step to generate an output image. The output image may be a deblurred image.

[0109] It is understood that the steps shown in Figure 3 are intended to illustrate a method according to an exemplary embodiment. Thus, various steps may be altered or modified, the order of certain steps may be changed, and additional steps may be added, while still achieving the overall desired operation. The method may be performed by a client device or a server, or by a combination of a client device and a server. The method may be performed by any suitable computing device.

[0110] Illustrative embodiments have been described herein by way of example, but those skilled in the art will recognize that changes and modifications may be made to the embodiments without departing from the true scope and spirit of the elements, products, and methods to which the embodiments are directed and as defined by the claims.

[0111] 4 is a schematic diagram of a computer program according to an exemplary implementation. In some implementations, the disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a non-transitory computer-readable storage medium or other non-transitory medium or article of manufacture.

[0112] In an example implementation, the computer program product 400 is provided using a signal-bearing medium 402 that may include one or more programming instructions 404 that, when executed by one or more processors, may provide the functionality or a portion of the functionality described above with respect to FIGS. 1-3. In some examples, the signal-bearing medium 402 may include a non-transitory computer-readable medium 406, such as, but not limited to, a hard disk drive, a CD, a DVD, digital tape, memory, a remotely stored component (e.g., in the cloud), etc. In some implementations, the signal-bearing medium 402 may include a computer-recordable medium 408, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some implementations, the signal-bearing medium 402 may include a communication medium 410, such as, but not limited to, a digital and / or analog communication medium (e.g., fiber optic cable, a wave guide, a wired communication link, a wireless communication link, etc.). Similarly, the signal bearing medium 402 may correspond to a remote storage (e.g., a cloud). The computing system may share information with the cloud, including sending or receiving information. For example, the computing system may receive additional information from the cloud to augment information obtained from a sensor or other entity. Thus, for example, the signal bearing medium 402 may be carried in a wireless form of a communication medium 410. The one or more programming instructions 404 may be, for example, computer-executable and / or logic-implemented instructions. In some examples, a computing device, such as the computing device 100 of FIG. 1 and the image processing system 202 of FIG. 2, may be configured to provide various operations, functions, or actions in response to the programming instructions 404 carried to the computing device by one or more of the computer-readable medium 406, the computer-recordable medium 408, and / or the communication medium 410.

[0113] The non-transitory computer-readable medium can also be distributed (e.g., remotely) among multiple data storage elements and / or a cloud that may be located remotely from each other. The computing device that executes some or all of the stored instructions can also be another computing device, such as a server.

[0114] Exemplary methods, systems, and apparatuses are described herein according to one or more aspects related to image deblurring. In general, any of the components, modules, methods, and operations described herein may be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the exemplary methods may be described in the general context of executable instructions stored on computer-readable storage memory that is local and / or remote to a computer processing system, and implementations may include software applications, programs, functions, and the like. Alternatively, or in addition, any of the functions described herein may be performed, at least in part, by one or more hardware logic components, such as, but not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip (SoCs), complex programmable logic devices (CPLDs), and the like.

[0115] Furthermore, the above detailed description explains various features and operations of the disclosed systems, apparatus, and methods with reference to the accompanying drawings. Various aspects and embodiments have been disclosed herein. While other aspects and embodiments are apparent, the various aspects and embodiments disclosed herein are by way of illustration and are not intended to be limiting, with the true scope being indicated by the following claims. [Explanation of symbols]

[0116] 100 computing devices 102 Communication Interface 104 User Interface 106 processors 108 Data Storage 110 Connection Mechanism 112 Data 114 Application Data 116 Operating System Data 118 Program Instructions 120 Application Programs 122 Operating Systems 202 Image Processing System 204 Pre-filter module 206 Deblur Module 208 Blur Estimation Module 210 Blur Correction Module 212 Halo Removal Module 400 Computer Program Products 402 Signal-bearing media 404 Programming Instructions 406 Computer-readable medium 408 Computer recordable media 410 Communication media

Claims

1. 1. A computer-implemented method comprising: receiving, by a processing device, an image; calculating, by the processing device, gradient features of the image, the calculating gradient features comprising calculating a directional derivative at each of a plurality of angles; estimating, by the processing device, blur for the image based on the gradient features; applying, by the processing device, a deblurring filter to the image, the deblurring filter being constructed based on the estimated blur; reducing halo from the image by the processing device; The method includes:

2. The method of claim 1 , further comprising removing noise from the image.

3. The method of claim 1 , wherein estimating the blur further comprises determining a blur function.

4. The method of claim 1 , wherein the blur is estimated based on at least one parameter of a Gaussian function.

5. The method of claim 4 , wherein the Gaussian function comprises an isotropic Gaussian function.

6. The method of claim 1 , further comprising the step of normalizing the image.

7. The method of claim 1 , further comprising the step of calculating a maximum magnitude of the directional derivative for each of the plurality of angles.

8. The method of claim 7 further comprising the step of selecting a minimum value from the maximum magnitudes of the directional derivatives, and wherein the blur is estimated using the minimum value.

9. The method of claim 1 , further comprising estimating a blur score for the image, the blur score indicating a degree of blurriness of the image.

10. The method of claim 9 , further comprising comparing the blur score to a threshold.

11. The method of claim 1 , further comprising combining multiple iterations of the blurring to estimate an inverse of the blurring.

12. The method of claim 1 , further comprising estimating an inverse of the blur using a polynomial based on a blur kernel.

13. The method of claim 1 , wherein the deblurring filter comprises a polynomial filter.

14. The method of claim 13 , wherein the polynomial filter is based on the inverse of the estimated blur.

15. The method of claim 1 , wherein the deblurring filter comprises a linear restoration filter constructed from the estimated blur.

16. Memory, One or more processors an apparatus comprising: Receive the image, calculating gradient features of the image, where calculating the gradient features includes calculating a directional derivative at each of a plurality of angles; estimating blur for the image based on the gradient features; applying a deblurring filter to the image, where the deblurring filter is constructed based on the estimated blur; An apparatus configured to reduce halo from the image.

17. A computer-readable storage medium storing instructions, comprising: The instruction: Receiving the image; calculating gradient features of the image, the calculating gradient features including calculating a directional derivative at each of a plurality of angles; estimating blur for the image based on the gradient features; and applying a deblurring filter to the image, the deblurring filter being constructed based on the estimated blur; Reducing halo from the image.

2. A computer-readable storage medium executable by one or more processors to perform functions including:

Images