Unsupervised learning method for video deblurring

Abstract

An image processing method extracts successive input blurry frames from a video and generates sharp frames corresponding to the input blurry frames. An optical flow is determined between the sharp frames, and this flow is used to calculate a blur kernel for each pixel. The blur kernel is used to blur each of the sharp frames into a corresponding re-blurred frame. The re-blurred frame is used to fine-tune the deblurring mesh by minimizing the distance between the re-blurred frame and the input blurry frame.

Description

background

[0001] Motion blur is a fundamental problem in computer vision, as it degrades image quality and hinders inference. Conventional deblurring algorithms exploit the physics of the image formation model and use hand-crafted precursors as reference points. These algorithms typically produce results that accurately represent the underlying scene but exhibit artifacts. Newer learning-based methods implicitly extract the distribution of natural images directly from the inputs and use them to synthesize plausible images. Although these methods can sometimes produce impressively sharp outputs, they cannot always faithfully reproduce the content of the latent image. Brief description

[0002] Disclosed is a method for fine-tuning deblurring neural networks in a self-supervised manner by enforcing the output sharp frames, when deblurred based on the optical flow between consecutive frames, to match the input blurred frames within a minimum error distance. Brief description of the different views of the drawings

[0003] To make it easy to identify the discussion of any particular element or action, the leading number(s) in a reference sign refers to the number of the figure in which that element is first introduced. Fig. Figure 1 illustrates an embodiment of a deblurring training system 100. Fig. Figure 2 illustrates an embodiment of a deblurring training process 200. Fig. Figure 3 illustrates an embodiment of a pixel-wise uncertainty kernel estimation process 300. Fig. Figure 4 is a block diagram of a computer system 400 in which the GPU or method introduced herein can be embodied or carried out. Detailed description

[0004] With reference to Fig. 1, an image deblurring training system 100 includes a deblurring network 102, an optical flow network 104, a blur kernel estimator 106, a reblurring unit 108, a supervised loss comparator 110, a self-supervised loss comparator 112, and a loss optimizer 114.

[0005] The deblurring network 102 receives an input of a blurry image. The deblurring network 102 transforms the blurry image input into an estimated sharp image and sends the estimated sharp image to the optical flow network 104, the supervised loss comparator 110, and the self-supervised loss comparator 112.

[0006] The optical flow network 104 receives the estimated sharp images from the deblurring network 102. The optical flow network 104 uses the estimated sharp images to determine an estimated optical flow and sends the estimated optical flow to the blur kernel estimator 106.

[0007] The uncertainty kernel estimator 106 receives the estimated optical flux from the optical flux mesh 104. The uncertainty kernel estimator 106 generates pixel-wise uncertainty kernels from the estimated optical flux and sends the pixel-wise uncertainty kernels to the re-unsharpening unit 108.

[0008] The re-blur unit 108 receives the estimated sharp image from the deblurring network 102 and the pixel-wise blur kernels from the blur kernel estimator 106. The re-blur unit 108 generates an estimated blur image from the estimated sharp image and the pixel-wise blur kernels and sends the estimated blur image to the self-monitoring loss comparator 112.

[0009] The monitored loss comparator 110 receives the input of the ground truth image and the estimated sharp image from the deblurring network 102. The monitored loss comparator 110 compares the input of the ground truth image and the estimated sharp image to determine the monitored loss and sends the monitored loss to the loss optimizer 114.

[0010] The self-monitored loss comparator 112 receives the input of the blurred image and the estimated blurred image from the re-blurring unit 108. The self-monitored loss comparator 112 compares the input of the blurred image and the estimated blurred image to determine the self-monitored loss and sends the self-monitored loss to the loss optimizer 114.

[0011] The loss optimizer 114 receives the monitored loss from the monitored loss comparator 110 and the self-monitored loss from the self-monitored loss comparator 112. The loss optimizer 114 can also receive a weight coefficient to balance the monitored loss and the self-monitored loss. The loss optimizer 114 generates a deblurring mesh training signal to modify the deblurring mesh. This modification can be used to minimize the hybrid loss of the balanced monitored and self-monitored losses. The deblurring mesh training signal can modify the weights assigned to the deblurring mesh 102. The optical flow mesh 104 and the uncertainty kernel estimator 106 can also be modified by the deblurring mesh training signal.

[0012] In some embodiments, the supervised loss comparator 110 is not used. The loss optimizer 114 can receive the self-supervised loss and generate the deblurring network training signal without the supervised loss, although better results can be achieved if both the supervised and unsupervised losses are used to train the networks.

[0013] The deblurring training system 100 may be implemented in accordance with the Fig. 2 shown process. With reference to Fig. 2. A deblurring training process receives 200 input blurry images (block 202). The input blurry images can (for example) comprise three temporally consecutive images according to equation 1. IB=(IB(t−1),IB(t),IB(t+1))

[0014] The input blurred images are then deblurred to estimated sharp images (block 204). The deblurring process can be performed according to Equation 2. I^S(t)=d(IB(t);Θd) wherein Θ d are the weights of the deblurring network. A deblurring network such as DVD or DeblurGAN can be used. Estimated sharp images can be generated for each of the three inputs from consecutive blurred images.

[0015] The optical flow is then estimated (block 206). The optical flow can be estimated from the estimated sharp images for both the image preceding the current image and the image following the current image according to equations 3 and 4, respectively. Ft−1→t=f(I^S(t−1),I^S(t),Θf) Ft−1→t=f(I^S(t+1),I^S(t),Θf) wherein Θ fare the weights of the optical flow network. An optical flow network such as FlowNetS can be used. The deblurring network and the optical flow network can be considered two subnetworks of the same overall network, and the overall network is trained holistically instead of training the two subnetworks separately.

[0016] The optical fluxes are then used to generate pixel-wise blur kernels according to Equation 5 (Block 208). K(p)=k(Ft−1→t(p),Ft+1→t(p))

[0017] In preferred embodiments, equation 5 is differentiable. The pixel-wise blur kernels can also be calculated using the formula in Fig. The process shown in step 3 can be estimated because such an estimate is differentiable and can therefore be trained in a neural network. An estimated blurred image is generated by using the pixel-wise blur kernel to blur the current sharp image again according to Equation 6 (Block 210). I^B(t)=b(IS(t);K)

[0018] A distance function such as the mean squared error (MSE) is used to compare the estimated blurry image with the input blurry image, and a loss is determined according to Equation 7 (Block 212). LU(Θd)=ΣUh(I^B(t),IB(t)) wherein U={IB(t)}

[0019] A distance function such as MSE is used to compare the estimated sharp image with the ground truth image, and a loss is determined according to Equation 8 (Block 214). LS(Θd)=ΣSh(I^S,IS) wherein S={IB;IS}

[0020] A hybrid loss is determined according to equation 9 (block 216). L(Θd)=LS(Θd)+αLU(Θd)

[0021] A weight coefficient α can be obtained to balance the contribution of losses. The weight coefficient can be set to 0.1. The determined loss then modifies the deblurring mesh d (Block 218). The deblurring mesh or the weights of the deblurring mesh can be modified using this process until the loss is minimized.

[0022] In some embodiments, the loss determined by comparing the estimated sharp image with the ground truth image is not used to train the deblurring network.

[0023] With reference to Fig. 3, a pixel-wise blur kernel estimation process 300 receives an optical flow (block 302). The optical flow can be determined using equations 3 and 4, and equation 10 can be used to determine the blur kernel K. wherein R1:x∈[0,τut+1→t(p)],y∈[u,τvt+1→t(p)] R2:x∈[0,τut−1→t(p)],y∈[u,τvt−1→t(p)] and τ is the exposure time. The exposure time can be 1 in some embodiments.

[0024] Therefore, the pixel-wise uncertainty kernel estimation process maps 300 optical flows to an uncertainty kernel lookup table to determine weights ω. i to be used in bilinear interpolation (block 304). Bilinear interpolation is then used to determine the uncertainty kernel (block 306). Equation 11 can be used to perform bilinear interpolation. K(p)[x,y]=Σi=1Nωi(u,v)ki[x,y] where N is the area of ​​optical flow over which the blur kernel is to be calculated. The value of N can be set to 33 x 33, which uses from -16 to 16 pixels in both directions. The blur kernel is then used by the de-blurrer (block 308) to de-blur the sharpened images.

[0025] Fig. 4 is a block diagram of one embodiment of a computer system 400 in which one or more aspects of the invention may be implemented. Computer system 400 includes a system data bus 436, a CPU 426, input devices 430, system memory 404, a graphics processing system 402, and display devices 428. In alternative embodiments, CPU 426, portions of graphics processing system 402, system data bus 436, or any combination thereof may be integrated into a single processing unit. Furthermore, the functionality of graphics processing system 402 may be contained in a chipset or other type of special-purpose processing unit or coprocessor.

[0026] As shown, the system data bus 436 connects the CPU 426, the input devices 430, the system memory 404, and the graphics processing system 402. In alternative embodiments, the system memory 404 can be directly connected to the CPU 426. The CPU 426 receives user input from the input devices 430, executes programming instructions stored in the system memory 404, works with data stored in the system memory 404, and configures the graphics processing system 402 to perform specific tasks in the graphics pipeline. The system memory 404 typically contains dynamic random-access memory (DRAM) used to store programming instructions and data for processing by the CPU 426 and the graphics processing system 402. The graphics processing system 402 receives instructions sent by the CPU 426 and processes the instructions to render and display graphic images on the display devices 428.

[0027] As also shown, system memory 404 contains an application program 412, an API 418 (application programming interface), and a graphics processing unit (GPU) driver 422. The application program 412 makes calls to the API 418 to produce a desired set of results, for example, in the form of a sequence of images. The application program 412 also sends image processing instructions to the API 418 for processing within the GPU driver 422. The high-level shading programs are typically source code text of high-level programming instructions designed to operate the graphics processing system 402. The functionality of the API 418 is typically implemented within the GPU driver 422. The GPU driver 422 is configured to translate the high-level programs into machine code programs, which may be optimized for image processing.

[0028] The graphics processing system 402 includes a GPU 410 (graphics processing unit), an on-chip GPU memory 416, an on-chip GPU data bus 432, a local GPU memory 406, and a GPU data bus 434. The GPU 410 is configured to communicate with the on-chip GPU memory 416 via the on-chip GPU data bus 432 and with the local GPU memory 406 via the GPU data bus 434. The GPU 410 can receive instructions sent by the CPU 426, process instructions to render graphics data and images, and store these images in the local GPU memory 406. Subsequently, the GPU 410 can display certain graphics images stored in the local GPU memory 406 on the display devices 428.

[0029] When operating to sharpen images as described herein, the GPU 410 includes one or more logic blocks 414. The logic blocks 414 include one or more of the Fig.The processing functions described in Sections 1-3, particularly graphics-intensive functions such as generating a blur kernel, are supported. The CPU 426 can also enable the processing of logic blocks in conjunction with the GPU 410 to perform the image processing and training functions described herein. For example, the CPU 426 can computationally process more generic (non-graphics-intensive) features of the processes, such as aspects of mesh training.

[0030] The GPU 410 may be equipped with any amount of on-chip GPU memory 416 and local GPU memory 406, including none, and may use on-chip GPU memory 416, local GPU memory 406, and system memory 404 in any combination for memory operations.

[0031] The on-chip GPU memory 416 is configured to contain GPU programming (420) and on-chip buffers (424). GPU programming (420) can be sent from the graphics processor driver (422) to the on-chip GPU memory (416) via the system data bus (436). The on-chip buffers (424) can be used to store frame-by-frame data that requires fast access to reduce graphics pipeline latency. Because the on-chip GPU memory (416) occupies valuable die area, it is relatively expensive.

[0032] The local GPU memory 406 typically includes lower-cost off-chip dynamic random access memory (DRAM) and is also used to store data and programs used by the GPU 410. As shown, the local GPU memory 406 includes a frame buffer 408. The frame buffer 408 stores data for at least one two-dimensional surface that can be used to drive the display devices 428. Furthermore, the frame buffer 408 can contain more than one two-dimensional surface, so that the GPU 410 can render to one two-dimensional surface while using a second two-dimensional surface to drive the display devices 428.

[0033] The display devices 428 are one or more output devices that can output a visible image according to an input data signal. For example, a display device may be constructed using a liquid crystal display or any other suitable display system. The input data signals to the display devices 428 are typically generated by reading the contents of one or more frames of image data stored in the frame buffer 408. Implementations and interpretation

[0034] References herein to “a (numeral) embodiment” or “an embodiment” do not necessarily refer to the same embodiment, although they may. Unless the context clearly requires otherwise, the words “comprise,” “comprehensive,” and the like are to be interpreted in an inclusive sense as opposed to an exclusive or exhaustive one; that is, in the sense of “including but not limited to.” Words using the singular or plural form also include the plural or singular form unless they are expressly limited to one or more subject matters. Furthermore, the words “herein,” “above,” “below,” and words of similar meaning, when used in this application, refer to this application as a whole and not to specific parts thereof.When the claims use the word "or" in relation to a list of two or more items, this word covers all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of the items in the list, unless it is expressly limited to one or the other. Any terms not expressly defined herein shall have their conventional meaning as they are normally understood by those skilled in the art in the relevant technology.

[0035] "Logic" refers to machine memory circuits, non-volatile machine-readable media, and / or circuits that, by virtue of their material and / or material-energy configuration, comprise control and / or operating signals and / or settings and values ​​(such as resistance, impedance, capacitance, inductance, current / voltage values, etc.) that can be applied to influence the operation of a device. Magnetic media, electronic circuits, electrical and optical storage (both volatile and non-volatile), and firmware are examples of logic. Logic specifically excludes pure signals or software per se (but does not exclude machine memories that comprise software and thereby form material configurations).

[0036] In this context, "hardware" refers to logic embodied as analog or digital circuits.

[0037] "Software" in this context refers to logic implemented as processor-executable instructions in a machine memory (e.g., reading / writing volatile or non-volatile memory or media).

[0038] In this context, "firmware" refers to software logic embodied as processor-executable instructions stored in read-only memory or media.

[0039] The methods disclosed herein may be embodied in hardware, software, or firmware, or combinations thereof. For example, the methods may be implemented as logic of a computer system, automobile, mobile phone, camera, or other device.

Claims

[1] Image processing method, comprising: Extracting a plurality of consecutive input blurred frames from a video; generating a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurred frames; Determining an optical flow between the sharp individual images; Applying optical flow to calculate a blur kernel per pixel; and Applying the blur kernel to blur each of the sharp frames into a corresponding deblurred frame. [2] An image processing method according to claim 1, wherein: the generation of the sharp individual images is carried out with a first neural network; and the generation of the optical flow is carried out with a second neural network. [3] The image processing method according to claim 2, wherein the first neural network is trained using the re-blurred individual images. [4] The image processing method according to claim 3, wherein the training of the first neural network minimizes a distance between each of the input blurred image frames and a corresponding deblurred image frame. [5] An image processing method according to claim 3 or 4, wherein the training of the first neural network comprises backpropagation of weights obtained from both a determined supervised loss in the reblurred frames and a determined unsupervised loss in the reblurred frames. [6] The image processing method according to any one of claims 2 to 5, wherein the first neural network is one of DVD and DeblurGAN. [7] The image processing method according to any one of claims 2 to 6, wherein the second neural network is a FlowNetS network. [8] An image processing method according to any preceding claim, wherein the re-blurring is performed using a pre-computed look-up table that maps the optical flow to the blur kernel using bilinear interpolation. [9] An image processing method according to any preceding claim, wherein the re-blurring is performed using a per-pixel convolution with the blur kernel. [10] The image processing method of claim 9, wherein the blur kernel comprises a pair of piecewise linear line segments per pixel. [11] Image processing device comprising: a graphics processing unit; and Logic to configure the graphics processing unit to perform the following: Extracting a plurality of consecutive input blurred frames from a video; generating a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurred frames; Determining an optical flow between the sharp individual images; Applying optical flow to calculate a blur kernel per pixel; and Applying the blur kernel to blur each of the sharp frames into a corresponding deblurred frame. [12] The apparatus of claim 11, wherein the logic further configures the graphics processing unit to perform: Generating the sharp individual images with a first neural network; and Generating the optical flow with a second neural network. [13] The apparatus of claim 12, further comprising logic for training the first neural network using the re-blurred image frames. [14] The apparatus of claim 13, wherein training the first neural network minimizes a distance between each of the input blurred image frames and a corresponding deblurred image frame. [15] The apparatus of claim 13, wherein the logic configures the graphics processing unit to perform an image processing method according to any one of claims 1 to 10.

Images