Deep motion deblur for video

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for deblurring an input image of scene. In one aspect, a method comprises receiving the input image of the scene at a first time point for deblurring, obtaining one or more context images of the scene at respective different time points that are different than the first time point, generating, from each of the one or more context images, a respective reference image of the scene at the first time point, and processing the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image, where the predicted image is a deblurred version of the input image at the first time point.

Description

DEEP MOTION DEBLUR FOR VIDEOBACKGROUND[1] This specification relates to performing deblurring of one or more frames from a video depicting a scene.SUMMARY[2] This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs deblurring of a frame from a video depicting a scene. The frame corresponds to a first time point of multiple time points in the video, so that the video includes multiple frames corresponding to multiple different time points.[3] The system can receive an input image of a scene at a first time point for deblurring. As used in this specification, the input image can be a frame from a video depicting a scene at a first time point.[4] The system obtains one or more context images of the scene at respective different time points that are different than the first time point. As used in this specification, a context image is another frame of the video at a different time point than the first time point corresponding to the input image. That is, within the video, the context image can be a prior frame or a subsequent frame of the frame corresponding to the input image. In some examples, the context image can be relatively sharper than the input image based on a sharpness measure.[5] The system generates, from each of the one or more context images, a respective reference image of the scene at the first time point. As used in this specification, a reference image is a warped version of the respective context image that has been warped to appear to be an image at the first time point based on an optical flow prediction.[6] The system processes the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image. In particular, the predicted image is a deblurred version of the input image at the first time point (e.g., a deblurred frame).[7] In some implementations, the method includes generating a final reference image by concatenating the one or more respective reference images.[8] In some implementations, the trained deblurring neural network includes multiple resolution stages, and the trained deblurring neural network generates a deblurred output image of a respective resolution at each of the multiple resolution stages.[9] In some implementations, processing the input image and the one or more respective reference images using the trained deblurring neural network to generate the predicted image includes, for a first resolution stage of the multiple resolution stages, generating a first input from the input image, processing the first input using the trained deblurring neural network to generate a first output image, applying a high-pass filter to the first output image to generate a filtered first output image, and combing the filtered first output image and the input image to generate a first deblurred output image of a first respective resolution.

[0010] In some implementations, generating the first input from the input image includes concatenating the input image and a copy of the input image.[HI In some implementations, the method further includes, for at least one resolution stage subsequent to the first resolution stage of the multiple resolution stages, upsampling a respective deblurred output image of a previous resolution stage to be a respective resolution of the at least one resolution stage, combining the input image, the final reference image, and the deblurred output image of the previous resolution stage to generate a respective combined image, processing the respective combined image using the trained deblurring neural network to generate the respective output image, applying the high-pass filter to the respective output image to generate a filtered respective output image, and combining the filtered respective output image and the respective deblurred output image of the previous resolution stage to generate a respective deblurred output image of the respective resolution.

[0012] In some implementations, the method further includes, for a final resolution stage of the multiple resolution stages, upsampling a respective deblurred output image of the previous resolution stage to be a respective resolution of the final resolution stage, combining the final reference image and the respective deblurred output image of the previous resolution stage to generate a final combined image, processing the final combined image using the trained deblurring neural network to generate a final output image, and combining the final output image and the respective output image of the previous resolution stage to generate the predicted image, where the predicted image is of a final resolution that is the same resolution as the input image.

[0013] In some implementations, obtaining the one or more context images of the scene at respective different time points includes selecting the one or more context images based at least in part on a sharpness of the one or more context images, where the sharpness is based at least in part on a blur kernel magnitude.

[0014] In some implementations, obtaining the one or more context images of the scene at respective different time points includes selecting the one or more context images based at leastin part on how close the respective time point for the one or more context images are to the first time point.

[0015] In some implementations, the one or more context images include a first context image and a second context image, where the first context image corresponds to a second time point prior to the first time point, and where the second context image corresponds to a third time point subsequent to the first time point.

[0016] In some implementations, generating, from each of the one or more context images, a respective reference image of the scene at the first time point further includes, for each of the context images, processing the context image and the input image using an optical flow prediction engine to generate a predicted optical flow between the context image and the input image, and generating the respective reference image using the predicted optical flow.

[0017] In some implementations, generating the respective reference image using the predicted optical flow for each of the context images includes warping the context image using the predicted optical flow for the context image to generate a first warped context image that corresponds to the first time point.

[0018] In some implementations, warping the context image includes determining whether any pixels of the first warped context image are invalid based at least in part on the optical flow estimate, and, for each invalid pixel of the first warped context image, replacing the invalid pixel with a corresponding pixel of the input image to generate the respective reference image.

[0019] In some implementations, the trained deblurring neural network is trained by applying a mean-squared error loss between the respective output image for a respective resolution stage of the plurality of resolution stages and a ground-truth image.

[0020] In some implementations, the trained deblurring neural network is trained by applying a low light enhancement loss to a final resolution stage of the plurality of resolution stages.

[0021] In some implementations, the trained deblurring neural network is a cascaded convolutional neural network.

[0022] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0023] Capturing video using hand-held videography devices, such as a mobile phone, can result in motion blur. In particular, if a user shakes the device, the captured video may be blurry and of relatively low quality. Additionally, if a user uses the device to capture an environment with relatively low light, the device may increase a length of exposure time to compensate forthe low light in order to capture the video with lower noise level, which can also result in a blurred video with decreased quality.

[0024] Some conventional systems incorporate stabilizing devices for capturing video, such as a gimbal stabilizer. Gimbal stabilizers can prevent motion blur while capturing the video of the environment. However, using stabilizing hardware to prevent motion blurring can be generally inconvenient due to having to use a greater number of devices to capture video, and the stabilizing hardware cannot reduce motion blurring during post-processing of the video (e.g., after capturing the video).

[0025] Some other systems can perform motion deblurring during post-processing using deblurring techniques. For example, some systems can deblur one or more frames of a video using the deblurring techniques. However, these techniques may not be able to deblur relatively high levels of motion blurring in a frame, and these techniques do not leverage temporal information corresponding to the frames of the video.

[0026] In contrast, the described system uses relatively sharp frames of a video and the corresponding temporal information to deblur an input frame, which can result in increased efficiency and increased accuracy in deblurring the input frame.

[0027] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG. 1 is a block diagram of an example system.

[0029] FIG. 2 is an example diagram of a predicted image generation system.

[0030] FIG. 3 is an example diagram of a reference image generation system.

[0031] FIG. 4 is a flow diagram of an example process for performing deblurring of one or more frames from a video depicting a scene.

[0032] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0033] FIG. 1 shows an example system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0034] The system 100 is configured to performing deblurring of a frame from a video depicting a scene. The frame corresponds to a first time point of multiple time points in the video, and the video includes multiple frames corresponding to multiple different time points.

[0035] The system 100 includes a deblurring system 102, a user device 104, and a training system 108. The user device 104 can be a computer, and the user device 104 can provide the input image 114 to the deblurring system 102. The input image 114 is a frame of a video corresponding to a time point of the video.

[0036] The deblurring system 102 includes a reference image generation system 1 10, a training system 108, and a predicted image generation system 112.

[0037] The deblurring system 102 is configured to process the input image 114 and one or more context images 116 using the reference image generation system 110 and the predicted image generation system 112 to generate a predicted image 120 that is a deblurred version of the input image 114. The one or more context images 116 are images from the video at respective time points. The context images 116 correspond to respective time points that are different than the first time point corresponding to the input image 114.

[0038] In particular, the system uses the reference image generation system 110 to generate a final predicted image 118 by processing the input image 114 and the one or more context images 116, and the system can generate the predicted image 120 by processing the input image 114 and the final predicted image 118 using the predicted image generation system 112, as described in further detail below with reference to FIGs. 2 and 3.

[0039] During training, the dcblurring system 102 can use the training system 108 to train a neural network (e.g., a deblurring neural network) of the predicted image generation system 112. In particular, the deblurring system 102 uses the training system 108 to train the deblurring neural network by providing training data 106 including training examples 122 to the deblurring neural network, as described in further detail below with reference to FIG. 2.

[0040] In some examples, during training, the deblurring system 102 can use the training system 108 to train an optical flow prediction engine of the reference image generation system 110 by processing training examples 122. In particular, the deblurring system 102 uses the training system 108 to train the deblurring neural network by providing the training examples 122 to the optical flow prediction engine, as described in further detail below with reference to FIG. 3.

[0041] In particular, each of the training examples 122 include multiple images. Each training example 122 includes (i) a set of context images, (ii) a training image, and (iii) a ground-truth, deblurred version of the training image.

[0042] Overall, by leveraging the sharp reference images corresponding to different time points of a video, the deblurring system 102 can efficiently deblur an input image 114 using the reference image generation system 110 and the predicted image generation system 112. Additionally, by using previous predictions of the deblurred input image at multiple resolution stages, the deblurring system 102 generates a final deblurred image with increased efficiency and accuracy.

[0043] FIG. 2 shows an example predicted image generation system. The predicted image generation system 1 12 is an example of a system in which the systems, components, and techniques described below are implemented. For example, a system, e.g., the deblurring system 102 of FIG. 1, appropriately configured in accordance with this specification, can implement the predicted image generation system 112.

[0044] The predicted image generation system 112 can receive the input image 114 and the final reference image 118 generated by the reference image generation system 102 = as described in further detail below with reference to FIG. 3. In some cases, the final reference image is a combined, e.g., concatenated, version of reference image predictions that correspond to a time point prior to the input image and to a time point subsequent to the input image.

[0045] The predicted image generation system 112 can then process the input image 114 and the final reference image 118 using the predicted image generation system 112 to generate the predicted image 120. The predicted image 120 is a predicted deblurred version of the input image.

[0046] The predicted image generation system 1 12 includes multiple resolution stages, where each resolution stage processes an image using a deblurring neural network 204. In some examples, the resolution stage includes a high-pass filter 206.

[0047] In particular, the predicted image generation system 112 can generate an output image 212 having a respective resolution at each resolution stage. Generally, the resolution stages are arranged from coarse to fine. In other words, the resolution increases with each stage, i.e., with the first stage having the lowest resolution and the last stage having the highest.

[0048] For each stage other than the first stage, the predicted image generation system 112 can leverage the output image 212 of a previous resolution stage and the final reference image 118 in order to generate the output image 212 for the current resolution stage. At the final resolution stage, the predicted image generation system 112 can generate the predicted image 120.

[0049] In particular, at a first resolution stage, the predicted image generation system 112 can process the input image 114 using the deblurring neural network 204 to generate an output image 212-a. The predicted image generation system 112 can then apply a high-pass filter 206to the output image 212-a. In particular, the high-pass filter 206 filters out relatively lower frequency signals of the output image 212-a while allowing relatively high-pass frequency signals to pass through. Additionally, the predicted generation system 112 can combine (e.g., combining 208) the output image 212-a and the input image 114, and the predicted generation system 112 can perform upsampling 210 to the output image 114-a.

[0050] In some examples, the predicted image generation system 112 can perform a concatenation 202-a of the input image 114 and a copy of the input image 114, and the predicted image generation system 112 can process the concatenation result using the deblurring neural network 204 to generate the output image 212-a. In some examples, the deblurring neural network 204 is a cascaded convolutional neural network.

[0051] At each intermediate stage, i.e., each stage between the first stage and the last stage, the predicted image generation system 112 can process the input image 114, the final reference image 118, and the output image 212-a using the deblurring neural network 204 to generate an output image 212-b. The predicted image generation system 112 can then apply the high-pass filter 206 to the output image 212-b. Additionally, the predicted generation system 112 can combine (e.g., combining 208) the output image 212-b and the output image 212-a, and the predicted generation system 112 can perform upsampling 210 to the output image 212-b.

[0052] In some examples, the predicted image generation system 112 can perform a concatenation 202 -b of the input image 114, the final reference image 118, and the output image 212-a, and the predicted image generation system 112 can process the concatenation result using the deblurring neural network 204 to generate the output image 212-b.

[0053] Generally, the upsampling 210 performed at a given stage upsamples the input image 114 to the resolution for the next stage.

[0054] At the final resolution stage, the predicted image generation system 112 can process the final reference image 118 and the output image 212-b using the deblurring neural network 204 to generate the output image 212-c. The predicted image generation system 112 can then combine the output image 212-c and the output image 212-b to generate the predicted image 120, and the predicted image generation system 112 may provide the predicted image 120 to a user device.

[0055] In some examples, the predicted image generation system 112 can perform a concatenation 202-c of the final reference image 118 and the output image 212-b, and the predicted image generation system 112 can process the concatenation result using the deblurring neural network 204 to generate the output image 212-c. The output image 212-c is a residual image (e.g., a processed image) of the output image 212-b.

[0056] Prior to using the predicted image generation system 112, the training system 108 can train the deblurring neural network 204 on training data that includes the set of training examples 122, with each training example 122 including (i) the set of context images, (ii) the training image, and (iii) the ground-truth, deblurred version of the training image.

[0057] For example, the training system 108 can train the deblurring neural network 204 by applying for each stage, a mean-squared error loss between the predicted image of the respective resolution stage (e.g., output image 212) and a ground-truth image. The system generates the ground-truth image by downsampling the ground-truth image having the final resolution to have the corresponding resolution for the stage.

[0058] In some examples, the training system 108 can additionally train the deblurring neural network 204 by applying a saturation region emphasis loss to each resolution stage or to the final resolution stage.

[0059] In particular, the system can train the deblurring neural network 204 by transforming the input image 114 from a first particular color space to a second particular color space. For example, the deblurring neural network 204 can transform the input image 114 from an RGB color space to a YUV color space. The system can then generate a binary mask based on the Y channel of the input image 114. In this case, if the value of the Y channel is greater than a particular value, the system can set the max pixel value to 1 and other mask pixels to 0. The system can then compute the saturation region emphasis loss as shown in Equation 1 :(1 ) Losssat= (y - y) • Msatwhere y is the output image 212 for the particular resolution stage, y is the corresponding ground-truth image, and Msatis the binary mask. In some examples, the saturation region emphasis loss can include a customized loss weight.

[0060] The training system 108 can also train the deblurring neural network 204 by applying a low light enhancement loss to the final resolution stage.

[0061] In particular, the system can train the deblurring neural network 204 as shown in Equation 2:where yi / s is the output image 212 for the final resolution stage (e.g,. the downsampled predicted output image), y1 / gis the corresponding ground- truth image, Aperrepresents a perceptual loss between features of the output image 212 and the ground-truth image, and < > represents a pre-trained VGG19 neural network to generate perceptual features. The systemcan obtain the perceptual features of the output image 212 by processing the output image 212 and the ground-truth image using the VGG19 neural network.

[0062] FIG. 3 is an example diagram of a reference image generation system. For convenience, the reference image generation system 110 will be described as being implemented by a system. For example, a deblurring system, e.g., the deblurring system 102 of FIG. 1, appropriately configured in accordance with this specification, can implement the reference image generation system 110.

[0063] The reference image generation system 110 includes warping 302, an optical flow prediction engine 304, and a concatenation 306.

[0064] The reference image generation system 110 can obtain the input image 114 and one or more context images 308. The one or more context images 308 each correspond to a time point of the video that is different than the first time point.

[0065] In particular, the reference image generation system 110 can obtain a context image 208-a that corresponds to a time point prior to the first time point and another context image 208-b that corresponds to a time point subsequent to the first time point.

[0066] In some examples, the reference image generation system 110 selects each context image 308 according to a threshold sharpness, where the threshold sharpness is based on a blur kernel magnitude of the image. In particular, the reference image generation system 110 can select a context image 308 by selecting an image that is after the first time point in the video and has a sharpness that exceeds the threshold. The reference image generation system 110 also selects another context image 308 by selecting an image that is before the first time point in the video and has a sharpness that exceeds the threshold.

[0067] In some cases, if there is an image that exceeds the sharpness of any another image that is within a threshold number of frames from the first image (prior to the first image or subsequent to the first image), the reference image generation system 110 does not select the image that exceeds the sharpness of any another image within the threshold number of frames. Instead, the reference image generation system 110 can select the first image (e.g., the current blurry image) as the context image 308. In some other examples, the reference image generation system 110 may select an image that is within the threshold number of frames.

[0068] The reference image generation system 110 can generate a respective reference image 310 from each of the one or more context images 308. The reference image generation system 110 can concatenate the reference image 310-a and the reference image 310-b to generate the final reference image 118.

[0069] For example, for two context images (e.g., context image 308-a and context image 308- b), the reference image generation system 110 can process each context image 308 by applying warping 302-a, where each reference image 310 is a warped version of the context image 308 that corresponds to the first time point. In particular, the reference image generation system 110 can process each context image 308 and the input image 114 using the optical flow prediction engine 304 to generate the respective reference image 310. In some examples, the optical flow prediction engine is an optical flow neural network.

[0070] In particular, the reference image generation system 102 can apply warping 302-a to the context image 308-a in a first direction (e.g., forward optical flow prediction) to generate the reference image 310-a corresponding to the first time point.

[0071] The reference image generation system 110 can then warp the reference image 310-a in a second direction (e.g., backward optical flow prediction) to generate a warped version of the reference image 310-a corresponding to the time point of the context image 308-a. In particular, to “warp” an image is to adjust an image (e.g., pixels of an image) to produce another image by distorting or manipulating shapes of the image. In this case, the system generates a warped version of the reference image 310-a by using optical flow (e.g., an optical flow prediction engine), where the warped version is based on an optical flow prediction, as described in further detail below with reference to FIG. 4.

[0072] The reference image generation system 110 can map each pixel in the context image 308-a to a pixel in the reference image 310-a and a pixel in the warped version of the reference image 210-a based on the forward optical flow prediction and the backward optical flow prediction generated by the optical flow prediction engine 304.

[0073] The reference image generation system 110 can determine whether any of the pixels of the warped version of the reference image 310-a are invalid based on comparing a distance between a corresponding pixel of the warped version of the reference image 310-a and a corresponding pixel of the context image 308-a. In this case, pixels are “invalid” if the distance between the corresponding pixel of the warped version of the reference image 310-a and the corresponding pixel of the context image 308-a exceeds a threshold distance, such that the pixel of the warped version is “too far” from the original pixel to be accurately mapped to the original pixel.

[0074] If the reference image generation system 110 determines that any of the pixels are invalid, the reference image generation system 110 can replace the invalid pixel with a corresponding pixel of the input image 114 to output the reference image 310-a.

[0075] The reference image generation system 110 can perform this process for the context image 308-b by performing warping 302-b to generate the reference image 310-b, and the reference image generation system 110 can concatenate the reference image 310-a and the reference image 310-b to generate the final reference image 118.

[0076] Prior to using the reference image generation system 110, for the case in which the optical flow prediction engine 304 is an optical flow neural network, the training system 108 can train the optical flow neural on training data that includes the set of training examples 122. with each training example 122 including (i) the set of context images, (ii) the training image, and (iii) corresponding ground-truth reference images.

[0077] For example, the training system 108 can train the optical flow neural network on a loss using the training image, the context images, and the ground-truth reference images.

[0078] FIG. 4 is a flow diagram of an example process for performing deblurring of one or more frames from a video depicting a scene. For convenience, the process 400 will be described as being performed by a system. For example, a system, e.g., the deblurring system 102 of FIG. 1, appropriately configured in accordance with this specification, can perform the process 400.

[0079] The system can receive an input image of a scene at a first time point for deblurring (402).

[0080] The system can obtain one or more context images of the scene at respective different time points that are different than the first time point (404).

[0081] The system selects the one or more context images based on a sharpness of the one or more context images. The sharpness is based on a blur kernel magnitude. The system can also select the one or more context images based on how close the respective time point for each context image is to the first time point.

[0082] In particular, the reference image generation system 110 can obtain a context image that corresponds to a time point prior to the first time point and another context image that corresponds to a time point subsequent to the first time point. The system can select a context image by selecting an image that is after the first time point in the video and has a sharpness that exceeds the threshold. The reference image generation system 110 also selects another context image 308 by selecting an image that is before the first time point in the video and has a sharpness that exceeds the threshold.

[0083] In some examples, the system obtains a first context image and a second context image. The first context image corresponds to a second time point prior to the first time point, and the second context image corresponds to a third time point after the first time point.

[0084] The system can generate a respective reference image of the scene at the first time point from each of the one or more context images (406). In particular, the system can generate a final reference image by concatenating the one or more respective reference images.

[0085] In particular, for each of the context images, the system processes the context image and the input image using the optical flow prediction engine to generate a predicted optical flow between the context image and the input image. The system can then generate the reference image using the predicted optical flow.

[0086] In some examples, the system warps the context image using the predicted optical flow for the context image to generate a first warped context image that corresponds to the first time point. In this case, the system determines whether any of the pixels of the first warped context image are invalid based on the optical flow estimate. For each invalid pixel, the system replaces the invalid pixel with a corresponding pixel of the input image to generate the reference image.

[0087] The system can process the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image (408). The trained deblurring neural network includes multiple resolution stages, and at each resolution stage, the trained deblurring neural network generates a deblurred output image of a respective resolution for the resolution stage. In some examples, the trained deblurring neural network is a cascaded convolutional neural network.

[0088] In particular, for the first resolution stage, the system generates a first input from the input image, and the system processes the first input using the trained deblurring neural network to generate a first output image. The system then applies a high-pass filter to the first output image to generate a filtered first output image, and the system combines the filtered first output image and the input image to generate a first deblurred output image. The first deblurred output image is of a first resolution corresponding to the first resolution stage.

[0089] In some examples, for the first resolution stage, the system generates the first input by concatenating the input image and a copy of the first image.

[0090] For each intermediate resolution stage (e.g., each resolution stage subsequent to the first resolution stage), the system upsamples the deblurred output image of the previous resolution stage to be a respective resolution of the intermediate resolution stage. For example, if the intermediate resolution stage is the second resolution stage, the system upsamples the first deblurred output image of the first resolution stage to be the resolution of the second resolution stage.

[0091] The system then combines the input image, the final reference image, and the deblurred output image of the previous resolution stage to generate a respective combined image, and thesystem processes the respective combined image using the trained deblurring neural network to generate the respective output image. For example, for the second resolution stage, the system combines the input image, the final reference image, and the first deblurred output image of the first resolution stage to generate the combined image.

[0092] The system then applies the high-pass filter to the respective output image to generate a filtered respective output image, and the system combines the filtered respective output image and the respective deblurred output image of the previous resolution stage to generate a respective deblurred output image of the respective resolution for the resolution stage. For example, for the second resolution stage, the system applies the high-pass filter to the second output image to generate a second filtered output image, and the system combines the second filtered output image and the first deblurred output image of the first resolution stage to generate a second deblurred output image of the second resolution.

[0093] For the final resolution stage, the system upsamples the respective deblurred output image of the previous resolution stage to be a respective resolution of the final resolution stage, and the system combines the final reference image and the respective deblurred output image of the previous resolution stage to generate a third combined image. The system can then process the final combined image using the trained deblurring neural network to generate a final output image, and the system combines the final output image and the respective output image of the previous resolution stage to generate the predicted image. In this case, the resolution of the predicted image is the same resolution as the resolution of the input image.

[0094] For example, in the case that there are three total resolution stages, for the third and final resolution stage, the system upsamples the second deblurred output image of the second resolution stage to be the resolution of the third resolution stage, and the sy stem combines the final reference image and the second deblurred output image of the second resolution stage to generate a third combined image. The system can then process the third combined image using the trained deblurring neural network to generate a third output image, and the system combines the third output image and the second output image of the second resolution stage to generate the predicted image. In this case, the resolution of the predicted image is the same resolution as the resolution of the input image.

[0095] The trained deblurring neural network can be trained by applying a mean-squared error loss between the respective output image for a respective resolution stage and a ground-truth image. Additionally, the trained deblurring neural network can be trained using the saturation region emphasis loss, the low-light enhancement loss, and the perceptual loss, as described above.

[0096] In some examples, the trained deblurring neural network is trained by applying a saturation region emphasis loss to each resolution stage or only to the final resolution stage.

[0097] In some examples, the trained deblurring neural network is trained by applying a low light enhancement loss to a final resolution stage. The low light enhancement loss is a loss between the prediction and the ground truth on the 1 / 8 to compute a loss of large-motion blur removal.

[0098] This specification uses the term “configured’7in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0099] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e.. one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0100] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or a combination of one or more of them.

[0101] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0102] In this specification the term ’ engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0103] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0104] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0105] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g.. EPROM. EEPROM, and flash memory devices; magnetic disks, e g., internal hard disks or removable disks; magneto-optical disks; and CD- ROM and DVD-ROM disks.

[0106] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0107] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0108] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.

[0109] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by any form or medium of digital data communication, e.g.. a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0110] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0111] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0112] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0113] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims canbe performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0114] What is claimed is:

Claims

CLAIMS1. A computer-implemented method, comprising: receiving an input image of a scene at a first time point for deblurring; obtaining one or more context images of the scene at respective different time points that are different than the first time point; generating, from each of the one or more context images, a respective reference image of the scene at the first time point; and processing the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image, wherein the predicted image is a deblurred version of the input image at the first time point.

2. The method of claim 1 , further comprising: generating a final reference image by concatenating the one or more respective reference images.

3. The method of any one of claims 1 or 2, wherein the trained deblurring neural network comprises a plurality of resolution stages, and wherein the trained deblurring neural network generates a deblurred output image of a respective resolution at each of the plurality of resolution stages.

4. The method of claim 3, wherein processing the input image and the one or more respective reference images using the trained deblurring neural network to generate the predicted image further comprises: for a first resolution stage of the plurality of resolution stages: generating a first input from the input image; processing the first input using the trained deblurring neural network to generate a first output image; applying a high-pass filter to the first output image to generate a filtered first output image; and combining the filtered first output image and the input image to generate a first deblurred output image of a first respective resolution.

5. The method of claim 4, wherein generating the first input from the input image further comprises concatenating the input image and a copy of the input image.

6. The method of any one of claims 3-5, further comprising: for at least one resolution stage subsequent to the first resolution stage of the plurality of resolution stages: upsampling a respective deblurred output image of a previous resolution stage to be a respective resolution of the at least one resolution stage; combining the input image, the final reference image, and the deblurred output image of the previous resolution stage to generate a respective combined image; processing the respective combined image using the trained deblurring neural network to generate the respective output image; applying the high-pass filter to the respective output image to generate a filtered respective output image; and combining the filtered respective output image and the respective deblurred output image of the previous resolution stage to generate a respective deblurred output image of the respective resolution.

7. The method of any one of claims 3-6, further comprising: for a final resolution stage of the plurality of resolution stages: upsampling a respective deblurred output image of the previous resolution stage to be a respective resolution of the final resolution stage; combining the final reference image and the respective deblurred output image of the previous resolution stage to generate a final combined image; processing the final combined image using the trained deblurring neural network to generate a final output image; and combining the final output image and the respective output image of the previous resolution stage to generate the predicted image, wherein the predicted image is of a final resolution that is the same resolution as the input image.

8. The method of any preceding claim, wherein obtaining the one or more context images of the scene at respective different time points further comprises: selecting the one or more context images based at least in part on a sharpness of the one or more context images, wherein the sharpness is based at least in part on a blur kernel magnitude.

9. The method of claim 7, wherein obtaining the one or more context images of the scene at respective different time points further comprises: selecting the one or more context images based at least in part on how close the respective time point for the one or more context images are to the first time point.

10. The method of any preceding claim, wherein the one or more context images comprise a first context image and a second context image, wherein the first context image corresponds to a second time point prior to the first time point, and wherein the second context image corresponds to a third time point subsequent to the first time point.

11. The method of any preceding claim, wherein generating, from each of the one or more context images, a respective reference image of the scene at the first time point further comprises: for each of the context images, processing the context image and the input image using an optical flow prediction engine to generate a predicted optical flow between the context image and the input image; and generating the respective reference image using the predicted optical flow.

12. The method of claim 10, wherein, for each of the context images, generating the respective reference image using the predicted optical flow comprises: warping the context image using the predicted optical flow for the context image to generate a first warped context image that corresponds to the first time point.

13. The method of claim 11, wherein warping the context image comprises: determining whether any pixels of the first warped context image are invalid based at least in part on the optical flow estimate; and for each invalid pixel of the first warped context image, replacing the invalid pixel with a corresponding pixel of the input image to generate the respective reference image.

14. The method of any preceding claim, when dependent on claim 3, wherein the trained deblurring neural network is trained by applying a mean-squared error loss between the respective output image for a respective resolution stage of the plurality of resolution stages and a ground-truth image.

15. The method of any preceding claim, when dependent on claim 3, wherein the trained deblurring neural network is trained by applying a saturation region emphasis loss to each resolution stage of the plurality of resolution stages or a final resolution stage of the plurality of resolution stages.

16. The method of any preceding claim, when dependent on claim 3, wherein the trained deblurring neural network is trained by applying a low light enhancement loss to a final resolution stage of the plurality of resolution stages.

17. The method of any preceding claim, wherein the trained deblurring neural network is a cascaded convolutional neural network.

18. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving an input image of a scene at a first time point for deblurring; obtaining one or more context images of the scene at respective different time points that are different than the first time point; generating, from each of the one or more context images, a respective reference image of the scene at the first time point; and processing the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image, wherein the predicted image is a deblurred version of the input image at the first time point.

19. The system of claim 18, the operations further comprising: generating a final reference image by concatenating the one or more respective reference images.

20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving an input image of a scene at a first time point for deblurring; obtaining one or more context images of the scene at respective different time points that are different than the first time point: generating, from each of the one or more context images, a respective reference image of the scene at the first time point; and processing the input image and the one or more respective reference images using a trained deblurring neural network to generate a predicted image, wherein the predicted image is a deblurred version of the input image at the first time point.

Images