Image enhancement based on elimination of image degradation by learning from multiple machine learning models

Abstract

A method includes receiving an input image. The method includes predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate image degradation associated with the input image, the training including (1) training a plurality of intermediate machine learning models to eliminate the image degradation, each of the plurality of intermediate machine learning models has been trained on a respective one of a plurality of training datasets corresponding to a respective plurality of degradation factors, each training dataset having a plurality of pairs of a clear image and a corresponding synthetic degraded version of the clear image, and (2) the image transformation model has been trained on an additional real image training dataset and has been learned from the plurality of intermediate machine learning models. The method includes providing the predicted transformed version.

Description

Background Art

[0001] Many modern computing devices, including mobile phones, personal computers, and tablet computers, include image capture devices, such as still cameras and / or video cameras. Image capture devices can capture images, such as images including people, animals, landscapes, and / or objects. Some image capture devices and / or computing devices can correct or otherwise modify captured images. For example, some image capture devices can provide "red eye" correction, which removes artifacts, such as red-appearing eyes of people and animals that may be present in images captured using bright lighting (such as flash lighting). After the captured image has been corrected, the corrected image can be saved, displayed, sent, printed to paper, and / or otherwise utilized. Summary of the invention

[0002] Eliminating blur, noise, and compression artifacts in images is a long-standing problem in computational photography. Image degradation can arise from multiple sources. When the photographer or autofocus system sets the focus incorrectly (out of focus), or when the relative motion between the camera and the scene is faster than the shutter speed (motion blur). In addition, even under ideal acquisition conditions, there can be inherent camera blur due to sensor resolution, light diffraction, lens aberrations, and anti-aliasing filters. Similarly, image noise is inherent to the capture of a discrete number of photons (shot noise) and analog-to-digital conversion and processing (readout noise). Typically, images are compressed before storage or transmission, for example using JPEG compression. Image compression can also degrade image quality.

[0003] With the support of the machine learning component system, the image capture device can be configured to enable the user to remove blur, noise, compression artifacts, etc. to create a clear image. In some aspects, the mobile device can be configured with these features so that the image can be enhanced in real time. In some cases, the mobile device can automatically enhance the image. In other aspects, the mobile phone user can non-destructively enhance the image to match their preferences. And for example, the pre-existing images in the user's image library can be enhanced based on the technology described herein.

[0004] In one aspect, a computer-implemented method is provided. The method includes receiving, by a computing device, a plurality of training data sets corresponding to a corresponding plurality of degradation factors, wherein each training data set includes a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image. The method also includes training a plurality of intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the plurality of intermediate machine learning models is trained on a corresponding training data set in the plurality of training data sets to eliminate the one or more image degradations associated with the given image based on the corresponding plurality of degradation factors. The method further includes training an image transformation model on an additional real image training data set to eliminate the one or more image degradations associated with the real image, wherein the image transformation model learns from the plurality of intermediate machine learning models. The method further includes outputting, by the computing device, a trained image transformation model for eliminating image degradation.

[0005] On the other hand, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by one or more processors, cause the computing device to perform functions. The functions include: receiving multiple training data sets corresponding to corresponding multiple degradation factors by the computing device, wherein each training data set includes multiple pairs of clear images and corresponding synthetic degraded versions of the clear images; training multiple intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each intermediate machine learning model in the multiple intermediate machine learning models is trained on a corresponding training data set in the multiple training data sets to eliminate the one or more image degradations associated with the given image based on the corresponding multiple degradation factors; training an image transformation model on an additional real image training data set to eliminate the one or more image degradations associated with the real image, wherein the image transformation model learns from the multiple intermediate machine learning models; and outputting the trained image transformation model by the computing device for eliminating image degradation.

[0006] On the other hand, a computer program is provided. The computer program includes instructions that, when executed by a computing device, cause the computing device to perform functions. The functions include: receiving, by the computing device, a plurality of training data sets corresponding to a corresponding plurality of degradation factors, wherein each training data set includes a plurality of pairs of a clear image and a corresponding synthetic degraded version of the clear image; training a plurality of intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the plurality of intermediate machine learning models is trained on a corresponding training data set in the plurality of training data sets to eliminate the one or more image degradations associated with the given image based on the corresponding plurality of degradation factors; training an image transformation model on an additional real image training data set to eliminate the one or more image degradations associated with the real image, wherein the image transformation model learns from the plurality of intermediate machine learning models; and outputting, by the computing device, a trained image transformation model for eliminating image degradation.

[0007] On the other hand, a product is provided. The product includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: receiving, by the computing device, a plurality of training data sets corresponding to a corresponding plurality of degradation factors, wherein each training data set includes a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image; training a plurality of intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the plurality of intermediate machine learning models is trained on a corresponding training data set in the plurality of training data sets to eliminate the one or more image degradations associated with the given image based on the corresponding plurality of degradation factors; training an image transformation model on an additional real image training data set to eliminate the one or more image degradations associated with the real image, wherein the image transformation model learns from the plurality of intermediate machine learning models; and outputting, by the computing device, the trained image transformation model for eliminating image degradation.

[0008] In another aspect, a system is provided. The system includes a component for receiving, by the computing device, a plurality of training data sets corresponding to a plurality of corresponding degradation factors, wherein each training data set includes a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image; a component for training a plurality of intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the plurality of intermediate machine learning models is trained on a corresponding training data set in the plurality of training data sets to eliminate the one or more image degradations associated with the given image based on the corresponding plurality of degradation factors; a component for training an image transformation model on an additional real image training data set to eliminate the one or more image degradations associated with the real image, wherein the image transformation model learns from the plurality of intermediate machine learning models; and a component for outputting, by the computing device, the trained image transformation model for use in eliminating image degradation.

[0009] In another aspect, a computer-implemented method is provided. The method includes receiving, by a computing device, an input image including one or more image degradations. The method also includes predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training data set in a plurality of training data sets corresponding to a corresponding plurality of degradation factors, each training data set having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training data set and having learned from the plurality of intermediate machine learning models. The method further includes providing, by the computing device, the predicted transformed version of the input image.

[0010] On the other hand, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by the one or more processors, cause the computing device to perform functions. The functions include: receiving an input image including one or more image degradations by the computing device; predicting a transformed version of the input image by an image transformation model, the image transformation model has been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training multiple intermediate machine learning models to eliminate the one or more image degradations, each of the multiple intermediate machine learning models having been trained on a corresponding training data set in a plurality of training data sets corresponding to a corresponding plurality of degradation factors, each training data set having included multiple pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model has been trained on an additional real image training data set and has been learned from the multiple intermediate machine learning models; and providing the predicted transformed version of the input image by the computing device.

[0011] In another aspect, a computer program is provided. The computer program includes instructions that, when executed by a computing device, cause the computing device to perform functions. The functions include: receiving an input image including one or more image degradations by the computing device; predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training data set in a plurality of training data sets corresponding to a corresponding plurality of degradation factors, each training data set having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training data set and having learned from the plurality of intermediate machine learning models; and providing the predicted transformed version of the input image by the computing device.

[0012] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: receiving, by the computing device, an input image including one or more image degradations; predicting, by an image transformation model, a transformed version of the input image, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training dataset in a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training dataset and having learned from the plurality of intermediate machine learning models; and providing, by the computing device, the predicted transformed version of the input image.

[0013] In another aspect, a system is provided. The system includes: a component for receiving, by a computing device, an input image including one or more image degradations; a component for predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training dataset in a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training dataset and having learned from the plurality of intermediate machine learning models; and a component for providing, by the computing device, the predicted transformed version of the input image.

[0014] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A is a diagram illustrating an example image enhancement process according to an example embodiment.

[0016] Figure 1B is a diagram illustrating an example network architecture according to an example embodiment.

[0017] Figure 2A is a diagram illustrating an example overview of a pipeline for generating synthetic data according to an example embodiment.

[0018] Figure 2B is a diagram illustrating example training of multiple intermediate models according to an example embodiment.

[0019] Figure 2C is a diagram illustrating an example pipeline for generating a curated dataset according to an example embodiment.

[0020] FIG. 2D is a diagram illustrating example training of an image transformation model according to an example embodiment.

[0021] Figure 3 is a diagram illustrating an example pipeline for generating synthetic data according to an example embodiment, according to an example embodiment.

[0022] Figure 4 Example motion blur kernels for different scale parameters are shown according to example embodiments.

[0023] Figure 5 Example input and output images corresponding to simulated motion blur are shown according to example embodiments.

[0024] Figure 6 An example histogram related to motion kernel length as a function of a scale parameter is shown according to an example embodiment.

[0025] Figure 7 An example generalized Gaussian blur kernel is shown according to an example embodiment.

[0026] Figure 8 Example input and output images corresponding to simulated compression artifacts according to example embodiments are shown.

[0027] Fig. 9 Example images corresponding to training data generation according to example embodiments are shown.

[0028] Fig.10 An example output image of a trained image transformation model according to an example embodiment is shown.

[0029] Fig.11 is a diagram illustrating the training phase and the inference phase of a machine learning model according to an example embodiment.

[0030] Fig.12 A distributed computing architecture according to an example embodiment is depicted.

[0031] Fig.13 is a block diagram of a computing device according to an example embodiment.

[0032] Fig.14 Depicted is a network of computing clusters arranged as a cloud-based server system according to an example embodiment.

[0033] Fig.15 is a flowchart of a method according to an example embodiment.

[0034] Fig.16 is a flow chart of another method according to an example embodiment. DETAILED DESCRIPTION

[0035] Image blur can often be modeled as a linear operator acting on a sharp latent image. For offset-invariant linear operators, the blur operation may be equivalent to convolution with a blur kernel. In practice, a common assumption is that the captured image includes additive noise and compression in addition to the blur. Therefore, the following relationship may apply:

[0036] v=C(S(u*k)+n), (Equation 1)

[0037] Where v is the captured image, u is the underlying clear image, k is the unknown blur kernel, * is the convolution operation, n is additive noise, S simulates the sensor nonlinear response (e.g., saturation), and C represents image compression. Some prior art techniques perform image deblurring by treating the problem as a "blind" deconvolution process. For example, in a first step, the blur kernel can be estimated. This can be achieved by assuming a clear image model, such as by using a variational framework, and in a second independent step, a "non-blind" deconvolution algorithm can be applied. However, image noise and artifacts caused by compression may have a negative impact on both steps. Even in cases where the blur kernel can be determined, "non-blind" deconvolution may be an ill-posed problem, and the presence of noise, compression, etc. may cause artifacts. A significant disadvantage of model-based deblurring is that the degradation model usually must have a high degree of accuracy. This may pose significant challenges in practice due to some unknown or partially known image transformations (e.g., unknown blur, unknown camera image signal processor (ISP), post-processing, compression, etc.).

[0038] The present application relates to enhancing images using machine learning techniques (such as but not limited to neural network techniques). When a mobile computing device user captures an image, the resulting image may have one or more image degradations due to motion blur, lens blur, pixel saturation, image compression, etc. Therefore, a technical problem related to image processing arises, which involves eliminating one or more image degradations to generate a clear image.

[0039] In order to eliminate one or more image degradations, the technology described herein applies a model based on a convolutional neural network to predict a clear image. The technology described herein includes receiving an input image and predicting an output image that is a clearer version of the input image using a convolutional neural network, and generating an output based on the output image. The input image and the output image can be high-resolution images, such as images of millions of pixels captured by a camera of a mobile computing device. The convolutional neural network can handle input images captured under various photographic conditions well, including different cameras, different objects and scenes, different lighting environments, and the like. In some examples, the trained model of the convolutional neural network can work on a variety of computing devices, including but not limited to mobile computing devices (e.g., smart phones, tablet computers, cellular phones, laptop computers), fixed computing devices (e.g., desktop computers), and server computing devices. The convolutional neural network can apply an image transformation model to the input image, thereby eliminating one or more image degradations and solving the technical problem of obtaining a clearer version of the image obtained (e.g., higher visual perception quality).

[0040] A neural network, such as a convolutional neural network, can be trained using a training data set of images to perform one or more aspects as described herein. In some examples, the neural network can be arranged as an encoder / decoder neural network.

[0041] As described in this paper, the Deep Motion, Defocus, and Degradation Enhancement (DeepMode) model can be applied to challenging situations where the blur level is moderate / large and the image exhibits other degradations such as noise or JPEG compression artifacts.

[0042] DeepMode can be configured as a supervised deep learning end-to-end solution to remove blur, noise, compression artifacts, etc. from images.

[0043] In one example, (a copy of) a trained neural network may reside on a mobile computing device. The mobile computing device may include a camera that may capture an input image. A user of the mobile computing device may view the input image and determine that the input image should be sharpened. The user may then provide the input image to the trained neural network resident on the mobile computing device. In response, the trained neural network may generate a predicted output image that is a sharp version of the input image, and then output the output image (e.g., provide the output image to be displayed by the mobile computing device). In other examples, the trained neural network does not reside on the mobile computing device; instead, the mobile computing device provides the input image to a remotely located trained neural network (e.g., via the Internet or another data network). The remotely located convolutional neural network may process the input image and provide the output image that is a sharp version of the input image to the mobile computing device. In other examples, a non-mobile computing device may also use the trained neural network to sharpen images, including images that are not captured by the camera of the computing device.

[0044] In some examples, the trained neural network can work in conjunction with other neural networks (or other software) and / or be trained to identify whether an input image has image degradation. Then, when it is determined that the input image has image degradation, the trained neural network described herein can apply the trained neural network to eliminate the image degradation in the input image.

[0045] Thus, the techniques described herein can improve images by removing image degradation, thereby enhancing the actual and / or perceived quality of the image. Enhancing the actual and / or perceived quality of images (including portraits of people) can provide emotional benefits to those who think their pictures look better. These techniques are flexible and can therefore be applied to images of human faces and other objects, scenes, etc.

[0046] Network Architecture

[0047] Figure 1A1 is a diagram illustrating an example image enhancement process 100A according to an example embodiment. In this example, the relationship between the synthetic data generation pipeline 105, the training phase 110, and the reasoning phase 115 is shown. Details of each of these different aspects are provided in more detail below. Some embodiments involve receiving a plurality of training data sets corresponding to a plurality of corresponding degradation factors by a computing device. Each training data set may include a plurality of pairs of a clear image and a corresponding synthetic degraded version of the clear image. For example, the synthetic data generation pipeline 105 may be configured to generate synthetic data, which includes an image with image degradation. For example, image degradation may be synthetically introduced into a clear image based on a plurality of degradation factors. For example, the synthetic data generation pipeline 105 may be configured to generate a plurality of training data sets 120-1, 120-2, ..., 120-N. In some embodiments, each training data set may correspond to a specific degradation factor (and different training data sets may correspond to different degradation factors). Each training data set may include a plurality of pairs of a clear image and a corresponding synthetic degraded version of the clear image.

[0048] As used herein, the term "degradation factor" generally refers to any factor that affects image clarity, such as, for example, the clarity of an image relative to quantitative image quality parameters such as contrast, focus, etc. In some embodiments, the plurality of degradation factors may include one or more of motion blur, lens blur, image noise, image compression artifacts, or artifacts caused by saturated pixels.

[0049] The term "motion blur" as used herein generally refers to a degradation factor that causes one or more objects in an image to appear blurred and / or less distinct due to the motion of the camera capturing the image, the motion of one or more objects, or a combination of both. In some examples, motion blur may be seen as streaks or smears in the image. The term "lens blur" as used herein generally refers to a degradation factor that causes an image to appear narrower than the depth of field of the captured scene. For example, some objects in an image may be in focus while other objects may appear out of focus.

[0050] The term "image noise" as used herein generally refers to degradation factors that cause an image to appear to have artifacts (e.g., spots, color dots, etc.) due to a low signal-to-noise ratio (SNR). For example, an SNR below a certain desired threshold may result in image noise. In some examples, image noise may occur due to circuitry in an image sensor or camera. The term "image compression artifacts" as used herein generally refers to degradation factors caused by lossy image compression. For example, image data may be lost during the compression process, resulting in visible artifacts in the decompressed version of the image.

[0051] As used herein, the term "saturated pixel" generally refers to a situation where a pixel is saturated with photons and then overflows to neighboring pixels. For example, a saturated pixel may be associated with an image intensity that is above a threshold intensity (e.g., above 245, or 255, etc.). The image intensity may correspond to the intensity of a grayscale, or the intensity of a color component in red, blue, or green (RGB). For example, a highly saturated pixel may appear brightly colored. Therefore, the overflow of photons from a saturated pixel to neighboring pixels may cause perceptual defects in the image (e.g., causing one or more neighboring pixels to be saturated, distorting the intensity of one or more neighboring pixels, etc.).

[0052] Generating different training data sets (e.g., multiple training data sets 120-1, 120-2, ..., 120-N) enables a customized approach to training machine learning models to eliminate specific image degradations. For example, during the training phase 110, multiple intermediate machine learning models 125-1, 125-2, ..., 125-N can be trained to eliminate one or more image degradations associated with a given image. For example, during the training phase 110, each of the multiple intermediate machine learning models 125-1, 125-2, ..., 125-N can be trained on a corresponding training data set in the multiple training data sets 120-1, 120-2, ..., 120-N to eliminate one or more image degradations associated with a given image based on a corresponding plurality of degradation factors.

[0053] Some embodiments relate to training an image transformation model on an additional real image training dataset to eliminate one or more image degradations associated with the real image, wherein the image transformation model learns from multiple intermediate machine learning models. For example, an image transformation model 135 can be trained based on an additional real image training dataset 130 by learning from multiple trained intermediate machine learning models 125-1, 125-2, ..., 125-N. The additional training dataset 130 may include real images with one or more image degradations caused by one or more degradation factors. Since each intermediate machine learning model has been trained to eliminate image degradation caused by a specific degradation factor, the image transformation model 135 can be trained to eliminate any image degradation in the real image from the additional training dataset 130 using multiple trained intermediate machine learning models 125-1, 125-2, ..., 125-N.

[0054] Some embodiments involve outputting a trained image transformation model by a computing device for use in removing image degradation. For example, after completing the training phase 110 , the trained image transformation model 135 may be provided for use in, for example, the inference phase 115 .

[0055] Some embodiments involve receiving, by a computing device, an input image including one or more image degradations. For example, during the inference phase 115, a degraded input image 135 may be received. The term "degraded image" as used herein generally refers to an image that has one or more image degradations due to one or more degradation factors.

[0056] Some embodiments relate to predicting a transformed version of an input image by an image transformation model, the image transformation model having been trained to eliminate one or more image degradations associated with the input image, the training having included (1) training multiple intermediate machine learning models to eliminate the one or more image degradations, each of the multiple intermediate machine learning models having been trained on a corresponding training dataset in a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included multiple pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training dataset and having learned from the multiple intermediate machine learning models.

[0057] For example, the trained transformation neural network 140 may take as input the degraded input image 135. As previously described, the trained transformation neural network 140 may have been trained during the training phase 110 to eliminate one or more image degradations associated with the degraded input image 135, and the training may have included training a plurality of intermediate machine learning models 125-1, 125-2, ..., 125-N to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models 125-1, 125-2, ..., 125-N having been trained on a corresponding training dataset in a plurality of training datasets 120-1, 120-2, ..., 120-N corresponding to a corresponding plurality of degradation factors, each training dataset having included a plurality of pairs of a clear image and a corresponding synthesized degraded version of the clear image.

[0058] Some embodiments involve providing, by a computing device, a predicted transformed version of an input image. For example, a trained transformation neural network 140 may transform a degraded input image 135 to predict a transformed version of an enhanced output image 145. Typically, the enhanced output image 145 represents a version of the degraded input image 135 in which one or more image degradations associated with the degraded input image 135 have been eliminated.

[0059] In some embodiments, multiple intermediate machine learning models may be teacher networks, wherein the image transformation model may be a student network, and wherein the image transformation model learns from multiple intermediate machine learning models based on knowledge distillation. In general, knowledge distillation is a teacher-student training process that can be used to train a smaller (or lighter) and generally more accurate network based on a heavier pre-trained teacher network. For example, a heavier pre-trained teacher network may reside on a cloud server (typically associated with larger computing resources), while a smaller (or lighter) network may be configured to operate on a mobile device (typically associated with limitations of computing resources). For example, multiple intermediate machine learning models 125-1, 125-2, ..., 125-N may be teacher networks, and the image transformation model 145 may be a student network trained on the corresponding outputs of multiple intermediate machine learning models 125-1, 125-2, ..., 125-N. In some embodiments, multiple intermediate machine learning models 125-1, 125-2, ..., 125-N may have different sizes, and the image transformation model 145 may be trained to optimize prediction quality with lower prediction runtime.

[0060] One or more of the multiple intermediate machine learning models or image transformation models can be a neural network. For example, deep neural network models such as convolutional neural networks (CNNs), generative adversarial networks (GANs), feature space enhancement and autoencoder models, meta-learning models, etc. can be used.

[0061] Figure 1B is a diagram showing an example network architecture 100B according to an example embodiment. In some embodiments, each of the plurality of intermediate machine learning models may be an encoder-decoder neural network with skip connections. In some embodiments, each encoder-decoder subnetwork may include three encoder stages and three decoder stages. In some embodiments, each encoder stage may include a feature extraction module and two non-linear transformation modules. In some embodiments, each decoder stage may include two non-linear transformation modules and a feature reconstruction module. In some embodiments, a given image may be processed at different scales, and the output of a given scale may be upsampled and concatenated with the input of a continuous scale.

[0062] For example, DeepMode can be a multi-scale encoder-decoder network with skip connections. The network can process the input image 150 at different scales, where the output of a given scale is upsampled and concatenated with the input of the next scale. For example, the output scale N of the Nth scale (indicated as 170-N) is upsampled and concatenated with the input scale N-1 (not shown in the figure), the output scale N-1 of the (N-1)th scale (not shown in the figure) is upsampled and concatenated with the input scale N-2 (not shown in the figure), and so on, until the output scale 2 of the 2nd scale (not shown in the figure) is upsampled and concatenated with the input scale 1 (indicated as 153-1).

[0063] At each scale, the input image 150 may be resized to generate a resized scale 1 labeled 152-1, all the way to a resized scale N labeled 152-N. In addition, at each scale, each encoder-decoder subnetwork may include three encoder stages (e.g., Enc 1 at scale 1 labeled 156-1 to Enc 1 at scale N labeled 156-N; Enc 2 at scale 1 labeled 158-1 to Enc 2 at scale N labeled 158-N; and Enc 3 at scale 1 labeled 158-1 to Enc 3 at scale N labeled 158-N). In some embodiments, each encoder stage may include a feature extraction module and two nonlinear transformation modules, as indicated in legend 176.

[0064] In addition, at each scale, each encoder-decoder subnetwork may include three decoder stages (e.g., Dec 1 at scale 1, labeled 166-1, to Dec 1 at scale N, labeled 166-N; Dec 2 at scale 1, labeled 164-1, to Dec 2 at scale N, labeled 164-N; and Dec 3 at scale 1, labeled 162-1, to Dec 3 at scale N, labeled 162-N). In some embodiments, each decoder stage may include one feature reconstruction module and two nonlinear transformation modules. In some embodiments, one or more of Dec 3 at scale 1, labeled 162-1, to Dec 3 at scale N, labeled 162-N, may include two nonlinear transformation modules without a feature reconstruction module.

[0065] Moreover, for example, at each scale, the neural network may include one or more skip connections. For example, a skip connection 172-1 may connect the last nonlinear transformation module of Enc 2 at scale 1, labeled 158-1, with the corresponding first nonlinear transformation module of Dec 2 at scale 1, labeled 164-1, and so on, until a skip connection 172-N connects the last nonlinear transformation module of Enc 2 at scale N, labeled 158-N, with the corresponding first nonlinear transformation module of Dec 2 at scale N, labeled 164-N.

[0066] Similarly, the skip connection 174-1 can connect the last nonlinear transformation module of Enc1 at scale 1 labeled 156-1 with the corresponding first nonlinear transformation module of Dec 1 at scale 1 labeled 166-1, and so on, until the skip connection 174-N connects the last nonlinear transformation module of Enc 1 at scale N labeled 156-N with the corresponding first nonlinear transformation module of Dec 1 at scale N labeled 166-N.

[0067] In some embodiments, each encoder / decoder block may include a high-order residual convolution block. This architecture can significantly improve computational efficiency and reduce memory usage.

[0068] In some embodiments, each intermediate machine learning model may include one or more space-to-depth (S2D) layers. For example, at each scale, S2D layers 154-1, ..., 154-N may be included. Each S2D layer may implement low-resolution processing by acquiring input, reducing the spatial resolution of the input, and increasing the number of channels to save information. Moreover, for example, each intermediate machine learning model may include one or more depth-to-space (D2S) layers corresponding to one or more S2D layers. For example, at each scale, D2S layers 168-1, ..., 168-N may be included. S2D layers 154-1, ..., 154-N and D2S layers 168-1, ..., 168-N may be configured to speed up the process and reduce memory usage. For example, the DeepMode architecture 100B may be configured to include a space-to-depth (depth-to-space) layer that acquires input and rearranges it into a tensor with a smaller spatial resolution but more channels (e.g., information is saved). Core processing can be performed at a low resolution (e.g., 2 to 4 times smaller than the original input resolution). Some embodiments may be configured without one or more of the s2d layers 154-1, ..., 154-N and the d2s layers 168-1, ..., 168-N.

[0069] In some embodiments, the upsampling convolution layer can be implemented as a transposed convolution, and / or by performing bilinear upsampling followed by regular convolution. In general, when the network is quantized to integers, bilinear upsampling followed by regular convolution can reduce the overall memory usage and the amount of (grid) artifacts.

[0070] In some embodiments, the modules in the first encoder stage (e.g., Enc 1 at scale 1 labeled 156-1 to Enc 1 at scale N labeled 156-N) and the first decoder stage (e.g., Dec 1 at scale 1 labeled 166-1 to Dec 1 at scale N labeled 166-N) can have filters of size 32.

[0071] In some embodiments, the modules in the second encoder stage (e.g., Enc 2 at scale 1 labeled 158-1 to Enc 2 at scale N labeled 158-N) and the second decoder stage (e.g., Dec 2 at scale 1 labeled 164-1 to Dec 2 at scale N labeled 164-N) can have filters of size 64.

[0072] In some embodiments, the modules in the third encoder stage (e.g., Enc 3 at scale 1 labeled 158-1 to Enc 3 at scale N labeled 158-N) and the third decoder stage (e.g., Dec 3 at scale 1 labeled 162-1 to Dec 3 at scale N labeled 162-N) can have filters of size 128.

[0073] In some embodiments, each encoder-decoder layer can be subjected to a 3×3 convolution followed by a Leaky ReLU activation. In some embodiments, the encoder can utilize a fuzzy pooling layer for downsampling, while the decoder can utilize bilinear resizing followed by a 3×3 convolution for upsampling.

[0074] In some embodiments, parameter sharing can be performed across different scales. For example, with reference to the encoder level, a scale recursive structure can be used to share the same parameters across scales. For example, at the first, second, and third encoder levels, the feature extraction module can share parameters across scales. Similarly, at the first, second, and third encoder levels, the first nonlinear transformation module can share parameters across scales. And, for example, at the first, second, and third encoder levels, the second nonlinear transformation module can share parameters across scales.

[0075] In some embodiments, the modified scale recursive structure can be used with an independent feature extraction module. In such embodiments, similar to the scale recursive structure, at the first, second, and third encoder levels, the first nonlinear transformation module can share parameters across scales. Moreover, for example, at the first, second, and third encoder levels, the second nonlinear transformation module can share parameters across scales.

[0076] In some embodiments, the modified scale-recursive structure may be further modified, wherein the feature extraction modules remain independent. However, both the first and second non-linear transformation modules may share parameters within a stage and also across scales.

[0077] In some embodiments, each of the multiple intermediate machine learning models can be associated with a corresponding number of filters and corresponding spatial-to-depth parameters. Various basic configurations spanning different model capacities and computing resources can be determined. For example, a light model can be configured, including 4 spatial-to-depth parameters, 28 or 32 filters, and a 5th-order residual convolution block, abbreviated as s4-f[28,32]-r5. In some embodiments, an image transformation model (e.g., image transformation model 140) can be configured as a light model. As another example, a base model can be configured, including 2 spatial-to-depth parameters, 16 filters, and a 5th-order residual convolution block, abbreviated as s2-f16-r5. Moreover, for example, a heavy model can be configured, including 2 spatial-to-depth parameters, 32 filters, and a 5th-order residual convolution block, abbreviated as s2-f32-r5. In some embodiments, multiple intermediate machine learning models (e.g., multiple intermediate machine learning models 125-1, 125-2, ..., 125-N) can be configured as heavy models.

[0078] In some embodiments, there may be benefits from using a "swish" or "hardswish" activation function compared to a "ReLu" or "ReLu6" activation function.

[0079] Table 1 shows basic performance information of different DeepMode configurations for example for a float32 model.

[0080]

[0081] Table 1

[0082] Figure 2A The training pipeline is shown in Figure 2D. The training can be performed in two steps: first, different DeepMode models can be trained and then fine-tuned on a curated dataset of carefully selected real images.

[0083] Figure 2A206-1, training data set 206-2, and so on, until training data set N is represented as 206-N.

[0084] Some embodiments involve training multiple intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the multiple intermediate machine learning models is trained on a corresponding training dataset in multiple training datasets to eliminate one or more image degradations associated with the given image based on a corresponding plurality of degradation factors.

[0085] Figure 2B 2 is a diagram illustrating an example training 200B of multiple intermediate models according to an example embodiment. For example, N different DeepModels may be trained, having different configurations (e.g., number of filters, spatial to depth parameters, etc.). For example, training of model 1, represented as 210-1, may be performed on training dataset 1, represented as 206-1, and model configuration 1, represented as 208-1, to generate a trained intermediate model 1, represented as 212-1. As another example, training of model 2, represented as 210-2, may be performed on training dataset 2, represented as 206-2, and model configuration 2, represented as 208-2, to generate a trained intermediate model 2, represented as 212-2. Moreover, for example, training of model N, represented as 210-N, may be performed on training dataset N, represented as 206-N, and model configuration N, represented as 208-N, to generate a trained intermediate model N, represented as 212-N. In some embodiments, the trained intermediate model 1 represented as 212-1, the trained intermediate model 2 represented as 212-2, ..., the trained intermediate model N represented as 212-N can act as teachers to refine and / or fine-tune the student model on a dataset of real images (with real degradations not simulated using a data generation pipeline).

[0086] Some embodiments involve applying multiple trained intermediate machine learning models to a given real image of an additional real image training dataset to generate corresponding output images. Such embodiments involve selecting the best transformed version of the given real image from the generated output images. Such embodiments also involve generating a selected dataset comprising pairs of real images and corresponding best transformed versions of the real images. In some embodiments, applying multiple trained intermediate machine learning models to a given real image involves applying each trained intermediate machine learning model at multiple resolutions.

[0087] Figure 2C is a diagram showing an example pipeline 200C for generating a curated data set according to an example embodiment. In some embodiments, a database of real blurred images 216 may be used. For example, the database of real blurred images 216 may include approximately 500 images with real degradation (e.g., motion blur, lens blur, noise, compression artifacts, etc.). These images may not have corresponding high-quality copies. In some embodiments, multiple trained intermediate machine learning models may be applied to a given real image of an additional real image training data set (e.g., from a database of real blurred images 216) to generate a corresponding output image. For example, a trained intermediate model 1 represented as 212-1, a trained intermediate model 2 represented as 212-2, ..., a trained intermediate model N represented as 212-N can be run at multiple resolutions, for example as a teacher model. For example, a database of real blurred images 216 can be used to perform model inference 1 represented as 214-1 with a trained intermediate model 1 represented as 212-1 to generate a prediction result 1 represented as 218-1. Likewise, model inference N, denoted as 214 -N, may be performed using the database of real blurred images 216 and the trained intermediate model N, denoted as 212 -N, to generate a prediction result N, denoted as 218 -N.

[0088] In some embodiments, the method may involve selecting the best transformed version of a given real image from the generated output images. For example, such operations may include performing a zoom-out operation, running a corresponding intermediate machine learning model, and performing an enlargement operation to achieve a target resolution. In some embodiments, for each image in the database of real blurred images 216, image quality and evaluation 220 may be performed on all corresponding results output by the corresponding trained intermediate model, and the best output may be selected from the results. Thus, each input image from the database of real blurred images 216 may be associated with an optimal output. For example, each low-quality input image in the database of real blurred images 216 may be associated with a restored image generated using one of the trained deep intermediate models. In some embodiments, image quality and evaluation 220 may be performed using a neural image assessment (NIMA) score and / or by manual selection.

[0089] The process generates a curated (or gold standard) dataset 224 of pairs of real degraded images (e.g., from the database of real blurred images 216), and their high-quality restored counterparts (e.g., the output of image quality and assessment 220) stored in the database of selected images 222. Thus, the curated dataset 224 includes pairs such as "(real image, selected image)".

[0090] 2D is a diagram illustrating an example training 200D of an image transformation model according to an example embodiment. As described herein, a curated dataset 224 may be generated based on a database of real blurry images 216 and a database of selected images 222. In some embodiments, the training of the image transformation model involves fine-tuning the image transformation model based on the curated dataset. An image transformation model 226 (e.g., a model with desired latency, memory usage, etc.) may be fine-tuned 228 based on the curated dataset 224.

[0091] In such an embodiment, fine-tuning 228 of the image transformation model 226 may involve quantization-aware training (QAT). Furthermore, for example, if desired, a trained image transformation model 230 may be generated and exported as, for example, a TensorFlowLite integer model.

[0092] In some embodiments, each base DeepMode model can be trained using an adaptive moment estimation (ADAM) optimizer with default parameters. In some embodiments, the learning rate can be set to 1e-4 and a polynomial decay (e.g., factor = 1.0) can be applied. Also, for example, in some embodiments, the complete optimization process can be run for 1.5M iterations. In some embodiments, a mini-batch size of 8 can be used.

[0093] In some embodiments, the model can be trained using an L1 loss function to penalize differences in pixel values ​​at multiple output scales. Also, for example, a penalty can be imposed on the difference between the target and reconstructed gradients. As another example, a penalized L1 norm of the gradient difference can be applied. Additional and / or alternative complex loss functions such as polarization dependent loss (PDL) can be applied.

[0094] In some embodiments, training the same model multiple times in the same way but with different random initializations can produce different results on real data. This may be the case even though the trained models achieve similar accuracy on the training and / or evaluation data (i.e., data from the synthetic data generation pipeline). Therefore, training and fine-tuning on real images may be required.

[0095] Training Data

[0096] In general, it can be challenging to obtain ground truth degraded clear paired data for supervised blind deblurring (and image restoration in general). For example, a complex setup may be required to acquire paired photos with, e.g., different exposure times, different camera configurations, etc.

[0097] Some existing techniques that can synthesize motion blur (e.g., due to a camera or an object) can be based on two procedures. For example, a longer exposure time can be simulated (virtually) so that the image looks naturally blurred. This can be performed by capturing a high frame rate video and then averaging a window of consecutive frames, where a larger time window controls the increase in exposure (and therefore causes more blur in the capture). However, this technique has certain limitations. For example, in order to accurately simulate a longer exposure, frames need to be captured throughout the inter-frame time (e.g., the camera always captures light in a continuous manner). However, this is not the case in reality. Therefore, when the motion is greater than one pixel, the direct averaging of consecutive frames may result in ghosting-type artifacts. To avoid this problem, a typical solution may be to interpolate multiple intermediate frames before calculating the temporal average. Unfortunately, this may involve solving the technically challenging problem of frame time interpolation.

[0098] A different procedure may be to simulate motion blur and other degradations on high-quality images. This approach may be challenging because different objects and / or different parts of the scene may experience different motions and thus may result in different blurs. Also, for example, camera shake may result in non-offset-invariant blurs.

[0099] Therefore, as described below, various degradations can be synthetically generated on a reference high-quality frame.

[0100] Data Generation Pipeline

[0101] Figure 3 300 for generating synthetic data according to an example embodiment. In some embodiments, a random crop of an image and an associated random segmentation (e.g., a binary mask) may be generated, as shown at block 305. Gamma correction values ​​may then be sampled, and a corresponding inverse gamma correction may be applied 310. A brightness offset 315 may be applied by randomly offsetting pixel values ​​by a constant factor (e.g., for data augmentation). Subsequently, saturated or nearly or close to saturated pixels may be randomly extrapolated to desaturate 320. Motion blur 325 and generalized Gaussian blur 330 may be simulated. For example, these operations may be performed for each segment or for the entire crop. Also, for example, noise 335 (e.g., Poisson noise, Gaussian noise, white noise, colored noise, etc.) may be added, and gamma correction 340 may be applied. In some embodiments, JPEG compression 345 may be applied to the blurred image (e.g., using a random quality factor sampled from a predefined distribution) to generate a blurred crop 350.

[0102] Random Motion Blur Kernel

[0103] In some embodiments, the multiple degradation factors may include motion blur. Such an embodiment involves generating one or more motion kernels that simulate camera shake due to hand shaking, wherein the amount of motion blur is associated with a scale parameter related to the exposure time. For example, a motion kernel can be simulated by imitating camera shake due to hand shaking. The amount of blur (e.g., expected value) can be controlled by a parameter related to the exposure time (referred to as "scale" herein). In some embodiments, the motion kernel can be generated from a discretized two-dimensional (2D) random walk. For example, in order to synthesize a motion kernel, a small Gaussian kernel (e.g., standard deviation = 0.35) can be rendered and accumulated at each discrete position of the random walk. In some embodiments, the kernel can be non-negative in construction and can be normalized to have an area of ​​one (e.g., light conservation).

[0104] In some embodiments, the intensity of the blur can be controlled by a random walk property. To render a kernel that is generally smooth but may also result in some variation, the following procedure can be applied:

[0105] Let N be the length of the random walk. Then, the horizontal and vertical components of the random walk can be generated by sampling 2×N independent and identically distributed Gaussian variables from a normal distribution N(0,Id). Such random variables usually represent the acceleration at each discrete time step.

[0106] In some embodiments, the 2xN vector may be integrated twice ("cumsum") along each dimension to generate a sufficiently smooth random walk.

[0107] In some embodiments, to represent different blur strengths, the values ​​in the random walk vector can be multiplied by a factor (scale / N). Generally speaking, a larger factor corresponds to a larger motion. In some embodiments, this factor can be a random value sampled from a gamma distribution.

[0108] In some embodiments, to avoid temporal issues, the first half of the random walk can be eliminated. Also, the remainder can be centered, for example, by subtracting the mean of each dimension. Thus, a kernel with a centered first-order moment can be generated (i.e., the motion kernel does not shift the image).

[0109] In some embodiments, the random kernel may be generated by rasterizing the random walk as described above.

[0110] In practice, fixing N to a large value (eg, N=300), and controlling a single parameter 'scale' can produce a practical way to simulate blurs of varying strengths, while keeping the number of adjustable parameters to a minimum.

[0111] The support of the kernel can be selected to be large enough to include the entire kernel. In some embodiments, the kernel can then be pruned to a minimum value that includes the support to speed up the filtering process.

[0112] Approximate kernel length (motion blur strength)

[0113] The kernel length can be calculated as the inverse of the kernel's L2 norm. This may be motivated by the need for a complete uniform motion kernel of length L.

[0114]

[0115] In some embodiments, the L2 norm of the kernel measures the spread of the non-negative blur kernel:

[0116]

[0118] When the kernel is Delta (sharpest), the limiting case L2(k)=1 occurs, and as the kernel becomes larger, L2(k)→0 occurs.

[0119] Figure 4 Example motion blur kernels for different scale parameters are shown according to example embodiments. For example, the images in column 4C1 correspond to scale parameter s=0.5, the images in column 4C2 correspond to scale parameter s=1.0, the images in column 4C3 correspond to scale parameter s=1.5, the images in column 4C4 correspond to scale parameter s=2.0, the images in column 4C5 correspond to scale parameter s=2.5, and the images in column 4C6 correspond to scale parameter S=3.0.

[0120] The statistics of the generated kernels as a function of the "scale" parameter are summarized for different motion levels in Table 2. In some embodiments, the results may be generated using approximately 1000 sampled kernels per scale.

[0121] scale Average(length) Median (length) Maximum (length) Standard(Length) 0.5 6.38 6.00 18.84 3.12 1.0 12.35 11.47 36.69 6.17 1.5 18.09 16.96 57.26 9.02 2.0 23.82 22.53 77.27 12.02 2.5 29.57 27.88 86.24 14.80 3.0 35.07 32.43 99.47 18.00

[0122] Table 2

[0123] Figure 5 Example input and output images corresponding to simulated motion blur according to example embodiments are shown. For example, the images in column 5C1 correspond to the sharp images in column 5C2, modified by simulated motion blur, as described herein. For example, the pretzel image in row 5R1 and column 5C1 corresponds to a synthetically blurred version of the corresponding image in row 5R1 and column 5C2.

[0124] Figure 6An example histogram as a function of a scale parameter associated with motion kernel length according to an example embodiment is shown. In some embodiments, the histogram shown may be based on the results generated using approximately 1000 sampled kernels per scale. For different "scale" parameters, the distribution of the (approximate) lengths of the generated kernels may be a known quantity. In some embodiments, the distribution of the generated motion blur kernels may be of great significance for deblurring images during the learning process. The generated distribution may include images with different blur levels (e.g., clear, moderately blurred, and very blurred, etc.) so that the network understands the deblurring problem. As shown, histogram 605 corresponds to a scale parameter s=0.5, histogram 610 corresponds to a scale parameter s=1.0, histogram 615 corresponds to a scale parameter s=1.5, histogram 620 corresponds to a scale parameter s=2.0, histogram 625 corresponds to a scale parameter s=2.5, and histogram 630 corresponds to a scale parameter s=3.0.

[0125] Generalized Gaussian blur

[0126] In some embodiments, the plurality of degradation factors may include lens blur. Such embodiments involve generating one or more generalized Gaussian blur kernels that simulate lens blur. Generalized Gaussian blur can be used to simulate lens blur and / or slight out-of-focus effects. In some embodiments, a parametric generalized Gaussian blur kernel may be used. The shape and size of the parametric generalized Gaussian blur kernel may be controlled by four parameters: σ0, ρ, θ, γ.

[0127] k(x,y;[a,b,c,d,γ])=Zexp(-(ax 2 +2bxy+cy 2 ) γ )(Equation 4)

[0128] in,

[0129]

[0130] As mentioned above, this may be a fairly general model for mild lens blur. However, it may not capture bokeh effects that vary from camera to camera, as the kernel of such bokeh effects corresponds to a donut-shaped kernel.

[0131] Figure 7Example generalized Gaussian blur kernels according to example embodiments are shown. The generalized Gaussian kernels shown in different columns correspond to different gamma values. For example, the kernel in column 7C1 corresponds to a gamma value of γ=0.3, the kernel in column 7C2 corresponds to a gamma value of γ=0.5, the kernel in column 7C3 corresponds to a gamma value of γ=1.0, the kernel in column 7C4 corresponds to a gamma value of γ=1.5, and the kernel in column 7C5 corresponds to a gamma value of γ=2.0. Random orientations (theta, θ) and non-monistic eccentricities (rho, ρ) are also shown, as well as increasing sizes from left to right in each row.

[0132] Image noise / JPEG compression artifacts

[0133] In some embodiments, the plurality of degradation factors include one or more of additive noise, signal-dependent noise, or colored noise. Image noise in the RGB space may be very complex because it may be signal-dependent and signal-dependent (e.g., colored noise). Moreover, for example, the noise fingerprint may significantly depend on the camera sensor, and / or camera ISP, and other post-processing (e.g., denoising, sharpening, resizing, etc.).

[0134] In general, DeepMode can be configured to transform arbitrary images. Thus, generic noise models can be simulated for training purposes.

[0135] In some embodiments, additive noise and signal dependent noise may be simulated, which may mimic read noise and shot noise.

[0136] In some embodiments, colored (eg, correlated) noise can be simulated by filtering the noise layer with a Gaussian blur kernel of a random standard deviation and adding it back to the noise layer (eg, after multiplication by a random factor).

[0137] In some embodiments, the plurality of degradation factors include one or more image compression artifacts. In general, JPEG compression may be used to compress shared images. There are several methods for implementing the JPEG standard. Specifically, the JPEG quality factor may affect the quantization of discrete cosine transform (DCT) coefficients, and / or may affect how chroma subsampling is performed. Therefore, in a synthetic data generation pipeline, when a quality factor equal to or greater than 90 is used, greater control over image generation may be achieved without performing chroma subsampling. Alternatively, standard 420 subsampling may be used.

[0138] Figure 8Example input and output images corresponding to simulated compression artifacts according to example embodiments are shown. JPEG compression may affect the image by introducing artifacts, but JPEG compression may also introduce correlations in the noise. A reference image is shown in column 8C1, a noisy image is shown in column 8C2, and an image with colored noise is shown in column 8C3. The last row 8R3 shows the difference between the first row 8R1 and the second row 8R2 (magnified 5 times).

[0139] The synthetic data generation pipeline can be configured to enable compressing blurry images using a random quality factor sampled from a uniform distribution, where the two extreme values ​​are configurable parameters. In addition, JPEG compression may introduce perturbations to the employed noise distribution.

[0140] Desaturation

[0141] In some embodiments, the plurality of degradation factors include one or more artifacts caused by saturated pixels. Saturated pixels are problematic because blurring effects may spill over to adjacent pixels. However, a reference frame with saturated pixels does not provide an indication of the actual radiance of the saturated pixels in the reference frame. Some embodiments involve determining whether a pixel value exceeds a saturation threshold. For example, a saturation threshold may be determined. Such embodiments also involve multiplying the pixel value by a random factor based on a determination that the pixel value exceeds the saturation threshold. Pixel values ​​above the threshold may be multiplied by a random factor (sampled from a predetermined uniform interval).

[0142] Therefore, when applying blur to a sharp, desaturated image that may later be clipped to mimic sensor saturation, the contribution of saturated pixels is greater than the saturated values, resulting in a more realistic blur simulation.

[0143] In general, it may be desirable to use the saturation threshold conservatively (e.g., not too far from the maximum value (i.e., 1.0 or 255)). With a larger margin, the simulated training data can produce a model with a thinned saturation region.

[0144] In some embodiments, to synthetically generate degraded sharp image pairs, images from a database (e.g., the Open Image Dataset (OID)) can be used. To use high-quality images as input, images can be filtered out based on blurriness and NIMA scores. Images from OID include object segmentation, which can be used to render different blurs to different segments. To simplify the synthetic data generation pipeline, images from OID (e.g., .jpg and / or .png files) can be used directly with their corresponding segmentation masks.

[0145] In some embodiments, approximately 7000 images from the OID dataset may be used, with NIMA scores greater than or equal to 7 and Polysharp blur scores less than 0.65. It may not be necessary to perform semantically aware sampling.

[0146] A large number of (visually) diverse training datasets can be designed and evaluated, covering different trade-offs between blur strength, noise distribution, and many other parameters. The visual quality of the results depends on the capacity of the model. Since the training datasets are synthetically generated, different models with different capacities can use different training datasets to produce the best results.

[0147] result

[0148] Fig. 9 Example images corresponding to training data generation according to an example embodiment are shown. The image in column 9C1 is a synthetically degraded version of the image in column 9C2. Similarly, the image in column 9C3 is a synthetically degraded version of the image in column 9C4.

[0149] Fig.10 Example output images of a trained image transformation model according to an example embodiment are shown.The images in column 10C2 are transformed (predicted) versions of the images in column 10C1 that were input to the trained image transformation model described herein.

[0150] As described in this paper, DeepMode is an end-to-end solution for removing image degradations such as blur, noise, compression artifacts, etc. in images trained using synthetically generated degraded and clean image pairs.

[0151] Train machine learning models to generate inferences / predictions

[0152] Fig.11 A diagram 1100 is shown illustrating a training phase 1102 and an inference phase 1104 of a trained machine learning model 1132, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Fig.11A training phase 1102 is shown, in which one or more machine learning algorithms 1120 are trained on training data 1110 to become a trained machine learning model 1132. Then, during an inference phase 1104, the trained machine learning model 1132 may receive input data 1130 and one or more inference / prediction requests 1140 (possibly as part of the input data 1130), and responsively provide one or more inferences and / or predictions 1150 as output.

[0153] Therefore, the trained machine learning model 1132 may include one or more models of one or more machine learning algorithms 1120. The machine learning algorithm 1120 may include, but is not limited to: an artificial neural network (e.g., a convolutional neural network, a recursive neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein). The machine learning algorithm 1120 may be supervised or unsupervised, and may implement any suitable combination of online learning and offline learning.

[0154] In some examples, the machine learning algorithm 1120 and / or the trained machine learning model 1132 can be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such an on-device coprocessor can be used to accelerate the machine learning algorithm 1120 and / or the trained machine learning model 1132. In some examples, the trained machine learning model 1132 can be trained, resident, and executed to provide reasoning on a specific computing device, and / or can otherwise make reasoning for a specific computing device.

[0155] During the training phase 1102, the machine learning algorithm 1120 may be trained using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques by providing at least the training data 1110 as training input. Unsupervised learning involves providing a portion (or all) of the training data 1110 to the machine learning algorithm 1120, and the machine learning algorithm 1120 determining one or more output inferences based on the portion (or all) of the provided training data 1110. Supervised learning involves providing a portion of the training data 1110 to the machine learning algorithm 1120, wherein the machine learning algorithm 1120 determines one or more output inferences based on the portion of the provided training data 1110, and accepting or correcting the output inferences based on correct results associated with the training data 1110. In some examples, supervised learning of the machine learning algorithm 1120 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 1120.

[0156] Semi-supervised learning involves obtaining the correct result for a portion (but not all) of the training data 1110. During semi-supervised learning, supervised learning is used for the portion of the training data 1110 that has the correct result, and unsupervised learning is used for the portion of the training data 1110 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 1120 receiving a reward signal about a previous reasoning, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 1120 can output an inference and receive a reward signal in response, where the machine learning algorithm 1120 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm 1120 and / or the trained machine learning model 1132 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0157] In some examples, the machine learning algorithm 1120 and / or the trained machine learning model 1132 may use transfer learning techniques. For example, the transfer learning technique may involve the trained machine learning model 1132 being pre-trained on a set of data and additionally trained using the training data 1110. More specifically, the machine learning algorithm 1120 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to the computing device CD1, where CD1 intends to execute the trained machine learning model during the inference phase 1104. Then, during the training phase 1102, the pre-trained machine learning model may be additionally trained using the training data 1110, where the training data 1110 may be derived from the core data and non-core data of the computing device CD1. Such further training of the machine learning algorithm 1120 and / or the pre-trained machine learning model using the training data 1110 of the CD1 data may be performed using supervised learning or unsupervised learning. Once the machine learning algorithm 1120 and / or the pre-trained machine learning model has been trained at least on the training data 1110, the training phase 1102 may be completed. The resulting trained machine learning model can be used as at least one of the trained machine learning models 1132.

[0158] In particular, once the training phase 1102 has been completed, the trained machine learning model 1132 can be provided to the computing device (if not already on the computing device). After the trained machine learning model 1132 is provided to the computing device CD1, the reasoning phase 1104 can begin.

[0159] During the inference phase 1104, the trained machine learning model 1132 may receive input data 1130 and generate and output one or more corresponding inferences and / or predictions 1150 about the input data 1130. Thus, the input data 1130 may be used as an input to the trained machine learning model 1132 for providing corresponding inferences and / or predictions 1150 to core components and non-core components. For example, the trained machine learning model 1132 may generate inferences and / or predictions 1150 in response to one or more inference / prediction requests 1140. In some examples, the trained machine learning model 1132 may be executed by portions of other software. For example, the trained machine learning model 1132 may be executed by an inference or prediction daemon to easily provide inferences and / or predictions upon request. The input data 1130 may include data from the computing device CD1 executing the trained machine learning model 1132 and / or input data from one or more computing devices other than CD1.

[0160] Input data 1130 may include training data described herein, such as real blurred images, synthetically generated images, images from a curated dataset, etc. Other types of input data are also possible. For example, the training data may include data collected for training an image transformation model.

[0161] Inferences and / or predictions 1150 may include task outputs, values, and / or other output data produced by a trained machine learning model 1132 operating on input data 1130 (and training data 1110). In some examples, the trained machine learning model 1132 may use the output inferences and / or predictions 1150 as input feedback 1160. The trained machine learning model 1132 may also rely on past inferences as input for generating new inferences.

[0162] After training, the trained version of the neural network can be an instance of the trained machine learning model 1132. In this approach, an instance of one or more inference / prediction requests 1140 can be a request to predict a transformed (e.g., deblurred, denoised, etc.) image, and a corresponding instance of inference and / or prediction 1150 can be the predicted transformed (e.g., deblurred, denoised, etc.) image.

[0163] In some examples, a computing device CD_SOLO may include a trained version of the neural network (perhaps after training). The computing device CD_SOLO may then receive a request to transform (e.g., deblur, denoise, etc.) an image and use the trained version of the neural network to predict the transformed (e.g., deblur, denoise, etc.) image.

[0164] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide output; for example, a first computing device CD_CLI may generate a request to a second computing device CD_SRV to predict a transformed (e.g., deblurred, denoised, etc.) image. CD_SRV may then use a trained version of a neural network to predict a transformed (e.g., deblurred, denoised, etc.) image and respond to the request from CD_CLI. CD_CLI may then, upon receiving a response to the request, provide the requested output (e.g., using a user interface and / or display, a printed copy, an electronic communication, etc.).

[0165] Sample Data Network

[0166] Fig.12A distributed computing architecture 1200 is depicted according to an example embodiment. The distributed computing architecture 1200 includes server devices 1208, 1210 configured to communicate with programmable devices 1204a, 1204b, 1204c, 1204d, 1204e via a network 1206. The network 1206 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 1206 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0167] although Fig.12 Only five programmable devices are shown, but the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. In addition, programmable devices 1204a, 1204b, 1204c, 1204d, 1204e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head mounted device (HMD), a network terminal, a mobile computing device, and the like. In some examples, such as shown by programmable devices 1204a, 1204b, 1204c, 1204e, the programmable device can be directly connected to the network 1206. In other examples, such as shown by programmable device 1204d, the programmable device can be indirectly connected to the network 1206 via an associated computing device such as programmable device 1204c. In this example, programmable device 1204c can act as an associated computing device for passing electronic communications between programmable device 1204d and network 1206. In other examples, such as shown by programmable device 1204e, the computing device may be part of and / or located within a vehicle (such as a car, truck, bus, boat or ship, airplane, etc.). Fig.12 In other examples not shown, the programmable device can be connected to the network 1206 either directly or indirectly.

[0168] The server devices 1208, 1210 may be configured to perform one or more services as requested by the programmable devices 1204a-1204e. For example, the server devices 1208 and / or 1210 may provide content to the programmable devices 1204a-1204e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0169] As another example, server devices 1208 and / or 1210 may provide programmable devices 1204a-1204e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0170] Computing device architecture

[0171] Fig.13 is a block diagram of an example computing device 1300 according to an example embodiment. Specifically, Fig.13 The illustrated computing device 1300 may be configured to perform at least one function of and / or related to the neural networks and / or methods 1500 , 1600 described herein.

[0172] The computing device 1300 may include a user interface module 1301, a network communication module 1302, one or more processors 1303, a data storage device 1304, one or more cameras 1318, one or more sensors 1320 and a power system 1322, all of which may be linked together via a system bus, network or other connection mechanism 1305.

[0173] The user interface module 1301 may be operable to send data to and / or receive data from an external user input / output device. For example, the user interface module 1301 may be configured to send data to and / or receive data from a user input device such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 1301 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether known now or developed later. The user interface module 1301 may also be configured to generate sound output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 1301 may further be configured with one or more tactile devices that may generate tactile outputs such as vibrations and / or other outputs that may be detected by touch and / or physical contact with the computing device 1300. In some examples, user interface module 1301 may be used to provide a graphical user interface (GUI) for utilizing computing device 1300 , such as, for example, a graphical user interface of a mobile phone device.

[0174] The network communication module 1302 may include one or more devices that provide one or more wireless interfaces 1307 and / or one or more wired interfaces 1308 that may be configured to communicate via a network. The wireless interface 1307 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth. TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM Transceiver, LTE TM The wired interface 1308 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver that can be configured to communicate via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection to a wired network.

[0175] In some examples, the network communication module 1302 can be configured to provide reliable, protected and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., ensuring message delivery) can be provided, which may be part of a message header and / or message trailer (e.g., packet / message ordering information, encapsulation header and / or encapsulation trailer, size / time information, and transmission verification information (such as a cyclic redundancy check (CRC) and / or parity value)). One or more cryptographic protocols and / or algorithms can be used to ensure the security of communications (e.g., encode or encrypt) and / or decrypt / decode communications, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol (such as a secure socket layer (SSL) or a transport layer security (TLS)), and / or a digital signature algorithm (DSA). Other cryptographic protocols and / or algorithms besides those listed herein may be used to protect (and then decrypt / decode) communications.

[0176] The one or more processors 1303 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 1303 may be configured to execute computer readable instructions 1306 contained in the data storage device 1304 and / or other instructions as described herein.

[0177] The data storage device 1304 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1303. The one or more computer-readable storage media may include volatile and / or non-volatile storage components that may be fully or partially integrated with at least one of the one or more processors 1303, such as optical, magnetic, organic or other memory or disk storage devices. In some examples, the data storage device 1304 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other examples, the data storage device 1304 may be implemented using two or more physical devices.

[0178] Data storage 1304 may include computer-readable instructions 1306 and possibly additional data. In some examples, data storage 1304 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data storage 1304 may include storage for trained neural network models 1312 (e.g., trained neural networks, such as models of convolutional neural networks described herein). Specifically, in these examples, computer-readable instructions 1306 may include instructions that, when executed by one or more processors 1303, enable computing device 1300 to provide some or all of the functionality of trained neural network model 1312.

[0179] In some examples, computing device 1300 may include one or more cameras 1318. Cameras 1318 may include one or more image capture devices, such as still cameras and / or video cameras, that are configured to capture light and record the captured light in one or more images; that is, camera 1318 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in video imagery. Camera 1318 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0180] In some examples, computing device 1300 may include one or more sensors 1320. Sensors 1320 may be configured to measure conditions within computing device 1300 and / or conditions in the environment of computing device 1300 and provide data about these conditions. For example, sensors 1320 may include one or more of the following: (i) sensors for obtaining data about computing device 1300, such as, but not limited to, a thermometer for measuring the temperature of computing device 1300, a battery sensor for measuring the charge of one or more batteries of power system 1322, and / or other sensors that measure conditions of computing device 1300; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, where identification sensors may be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the location of computing device 1300; (iv) an environmental sensor for obtaining data indicative of the environment of computing device 1300, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or gravity) acting on computing device 1300, such as, but not limited to, one or more sensors measuring force, torque, ground force, friction in one or more dimensions, and / or a ZMP sensor that identifies a zero moment point (ZMP) and / or the location of the ZMP. Many other examples of sensors 1320 are also possible.

[0181] The power system 1322 may include one or more batteries 1324 and / or one or more external power interfaces 1326 for providing power to the computing device 1300. Each of the one or more batteries 1324 may act as a storage power source for the computing device 1300 when electrically coupled to the computing device 1300. The one or more batteries 1324 of the power system 1322 may be configured to be portable. Some or all of the one or more batteries 1324 may be able to be easily removed from the computing device 1300. In other examples, some or all of the one or more batteries 1324 may be located inside the computing device 1300 and therefore may not be easily removed from the computing device 1300. Some or all of the one or more batteries 1324 may be rechargeable. For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply device, such as by one or more power supply devices located outside the computing device 1300 and connected to the computing device 1300 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1324 may be non-rechargeable batteries.

[0182] The one or more external power interfaces 1326 of the power system 1322 may include one or more wired power interfaces, such as USB cables and / or power cords, that implement a wired power connection to one or more power supply devices located outside the computing device 1300. The one or more external power interfaces 1326 may include one or more wireless power interfaces, such as Qi wireless chargers, that implement wireless power connections to one or more external power supply devices. Once a power connection to an external power source is established using the one or more external power interfaces 1326, the computing device 1300 can draw power from the external power source through the established power connection. In some examples, the power system 1322 may include associated sensors, such as battery sensors associated with one or more batteries or other types of power sensors.

[0183] Cloud-based servers

[0184] Fig.14 A cloud-based server system is depicted according to an example embodiment. Figure 6, the functionality of the neural network and / or computing devices may be distributed among computing clusters 1409a, 1409b, 1409c. Computing cluster 1409a may include one or more computing devices 1400a, cluster storage array 1410a, and cluster router 1411a connected via a local cluster network 1412a. Similarly, computing cluster 1409b may include one or more computing devices 1400b, cluster storage array 1410b, and cluster router 1411b connected via a local cluster network 1412b. Likewise, computing cluster 1409c may include one or more computing devices 1400c, cluster storage array 610c, and cluster router 1411c connected via a local cluster network 1412c.

[0185] In some embodiments, computing clusters 1409a, 1409b, 1409c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 1409a, 1409b, 1409c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in different geographic locations. For example, Fig.14 Each of the computing clusters 1409a, 1409b, 1409c is depicted as residing in a different physical location.

[0186] In some embodiments, the data and services at the computing clusters 1409a, 1409b, 1409c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 1409a, 1409b, 1409c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located at one or more distinct geographic locations.

[0187] In some embodiments, each of computing clusters 1409a, 1409b, and 1409c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0188] For example, in computing cluster 1409a, computing device 1400a may be configured to perform various computing tasks of a conditional neural network and / or computing device based on axial self-attention. In one embodiment, various functionalities of a neural network and / or computing device may be distributed among one or more of computing devices 1400a, 1400b, 1400c. Computing devices 1400b and 1400c in respective computing clusters 1409b and 1409c may be configured similarly to computing device 1400a in computing cluster 1409a. On the other hand, in some embodiments, computing devices 1400a, 1400b, and 1400c may be configured to perform different functions.

[0189] In some embodiments, computing tasks and stored data associated with the neural network and / or computing devices may be distributed across computing devices 1400a, 1400b, and 1400c based at least in part on processing requirements of the neural network and / or computing devices, processing capabilities of computing devices 1400a, 1400b, 1400c, latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may contribute to cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0190] The cluster storage arrays 1410a, 1410b, 1410c of the computing clusters 1409a, 1409b, and 1409c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0191] Similar to the way that the functionality of the conditional neural network and / or computing device based on axial self-attention can be distributed across the computing devices 1400a, 1400b, 1400c of the computing clusters 1409a, 1409b, 1409c, the various active parts and / or backup parts of these components can be distributed across the cluster storage arrays 1410a, 1410b, 1410c. For example, some cluster storage arrays can be configured to store a first layer of the neural network and / or a portion of the data of the computing device, while other cluster storage arrays can store a second layer of the neural network and / or other portions of the data of the computing device. Moreover, for example, some cluster storage arrays can be configured to store data of the encoder of the neural network, while other cluster storage arrays can store data of the decoder of the neural network. In addition, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0192] The cluster routers 1411a, 1411b, 1411c in the computing clusters 1409a, 1409b, 1409c may include networking equipment configured to provide internal and external communications for the computing clusters. For example, the cluster router 1411a in the computing cluster 1409a may include one or more Internet switches and routing devices configured to (i) provide local area network communications between the computing device 1400a and the cluster storage array 1410a via a local cluster network 1412a, and (ii) provide wide area network communications between the computing cluster 1409a and the computing clusters 1409b and 1409c via a wide area network link 1413a to the network 1206. Cluster routers 1411b and 1411c may include similar network equipment as cluster router 1411a, and cluster routers 1411b and 1411c may perform networking functions for computing clusters 1409b and 1409b similar to the networking functions that cluster router 1411a performs for computing cluster 1409a.

[0193] In some embodiments, the configuration of the cluster routers 1411a, 1411b, 1411c can be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network equipment in the cluster routers 1411a, 1411b, 1411c, the latency and throughput of the local cluster network 1412a, 1412b, 1412c, the latency, throughput and cost of the wide area network links 1413a, 1413b, 1413c, and / or other factors that may help to adjust the cost, speed, fault tolerance, resilience, efficiency and / or other design criteria of the system architecture.

[0194] Example How-to

[0195] Fig.15 is a flow chart of a method 1500 according to an example embodiment. The method 1500 may be performed by a computing device, such as computing device 1300.

[0196] Method 1500 may begin at block 1510 , where the method involves receiving, by a computing device, a plurality of training data sets corresponding to a respective plurality of degradation factors, where each training data set includes a plurality of pairs of a sharp image and a corresponding synthetically degraded version of the sharp image.

[0197] In box 1520, the method involves training multiple intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each intermediate machine learning model of the multiple intermediate machine learning models is trained on a corresponding training data set in multiple training data sets to eliminate one or more image degradations associated with the given image based on a corresponding multiple degradation factors.

[0198] At box 1530, the method involves training an image transformation model on an additional real image training dataset to eliminate one or more image degradations associated with the real images, wherein the image transformation model learns from multiple intermediate machine learning models.

[0199] At block 1540 , the method involves outputting, by a computing device, the trained image transformation model for use in removing image degradation.

[0200] In some embodiments, each of the multiple intermediate machine learning models may include an encoder-decoder neural network with skip connections, where a given image can be processed at different scales, where the output at a given scale can be upsampled and concatenated with the input at a continuous scale.

[0201] In some embodiments, each intermediate machine learning model may include: (1) one or more spatial-to-depth (S2D) layers, wherein each S2D layer performs low-resolution processing by taking an input, reducing the spatial resolution of the input, and increasing the number of channels to preserve information; and (2) one or more depth-to-spatial (D2S) layers corresponding to the one or more S2D layers.

[0202] In some embodiments, each intermediate machine learning model of the plurality of intermediate machine learning models is associated with a corresponding number of filters and a corresponding spatial-to-depth parameter.

[0203] Some embodiments involve applying multiple trained intermediate machine learning models to a given real image of an additional real image training dataset to generate a corresponding output image. Such embodiments involve selecting an optimal transformed version of a given real image from the generated output images. Such embodiments also involve generating a selected dataset comprising pairs of real images and corresponding optimal transformed versions of the real images. In some embodiments, applying multiple trained intermediate machine learning models to a given real image involves applying each trained intermediate machine learning model at multiple resolutions. In some embodiments, training of the image transformation model involves fine-tuning the image transformation model based on the selected dataset. In such embodiments, fine-tuning of the image transformation model involves quantization-aware training (QAT).

[0204] Some embodiments are directed to generating a synthetically degraded version of a sharp image.

[0205] In some embodiments, the plurality of degradation factors include motion blur. Such embodiments involve generating one or more motion kernels that simulate camera shake due to hand shaking, wherein the amount of motion blur is associated with a scale parameter related to exposure time.

[0206] In some embodiments, the plurality of degradation factors includes lens blur.Such embodiments involve generating one or more generalized Gaussian blur kernels that simulate lens blur.

[0207] In some embodiments, the plurality of degradation factors include one or more of additive noise, signal-dependent noise, or colored noise.

[0208] In some embodiments, the plurality of degradation factors includes one or more image compression artifacts.

[0209] In some embodiments, the plurality of degradation factors include one or more artifacts caused by saturated pixels. Such embodiments involve determining whether a pixel value exceeds a saturation threshold. Such embodiments also involve multiplying the pixel value by a random factor based on a determination that the pixel value exceeds the saturation threshold.

[0210] In some embodiments, the plurality of intermediate machine learning models may be a teacher network, wherein the image transformation model may be a student network, and wherein the image transformation model learns from the plurality of intermediate machine learning models based on knowledge distillation.

[0211] Fig.16 is a flow chart of a method 1600 according to an example embodiment. The method 1600 may be performed by a computing device, such as computing device 1300.

[0212] Method 1600 may begin at block 1610 , where the method involves receiving, by a computing device, an input image including one or more image degradations.

[0213] At box 1620, the method involves predicting a transformed version of an input image by an image transformation model, the image transformation model has been trained to eliminate one or more image degradations associated with the input image, the training having included (1) training multiple intermediate machine learning models to eliminate the one or more image degradations, each of the multiple intermediate machine learning models having been trained on a corresponding training dataset in a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included multiple pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model has been trained on an additional real image training dataset and has learned from the multiple intermediate machine learning models.

[0214] At block 1630 , the method involves providing, by a computing device, a predicted transformed version of an input image.

[0215] In some embodiments, each of the multiple intermediate machine learning models may include an encoder-decoder neural network with skip connections, where a given image can be processed at different scales, where the output at a given scale can be upsampled and concatenated with the input at a continuous scale.

[0216] In some embodiments, the image transformation model includes an encoder-decoder neural network with skip connections, and the image transformation model further includes: (1) one or more spatial to depth (S2D) layers, wherein each S2D layer implements low-resolution processing by obtaining an input, reducing the spatial resolution of the input while increasing the number of channels to preserve information; and (2) one or more depth to spatial (D2S) layers corresponding to the one or more S2D layers.

[0217] In some embodiments, the plurality of degradation factors include one or more of motion blur, lens blur, image noise, image compression artifacts, or artifacts caused by saturated pixels.

[0218] The present disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its spirit and scope. In addition to those methods and devices enumerated herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0219] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit and scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0220] With respect to any or all ladder diagrams, scenarios and flow charts discussed in the accompanying drawings and herein, each frame and / or communication may represent the processing of information and / or the transmission of information according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, depending on the functions involved, the functions described as frames, transmissions, communications, requests, responses and / or messages may not be performed in the order shown or discussed, including substantially performing at the same time or in the reverse order. In addition, more or less frames and / or functions may be used together with any ladder diagram, scenario and flow chart discussed herein, and these ladder diagrams, scenarios and flow charts may be combined with each other in part or in their entirety.

[0221] The box representing the processing of information may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or in addition, the box representing the processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions that can be executed by a processor to implement a specific logical function or action in a method or technology. The program code and / or related data may be stored on any type of computer-readable medium, such as a storage device or other storage medium including a disk or hard drive.

[0222] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media for short-term storage of data, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media for long-term storage of program code and / or data, such as auxiliary or persistent long-term storage devices, for example, such as read-only memory (ROM), optical or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered, for example, as a computer readable storage medium or a tangible storage device.

[0223] In addition, the blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0224] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of illustration and are not intended to be limiting, with the true scope being associated with the following claims.

Claims

1. A computer-implemented method comprising: Receiving, by a computing device, a plurality of training data sets corresponding to a respective plurality of degradation factors, wherein each training data set includes a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image; training a plurality of intermediate machine learning models to eliminate one or more image degradations associated with a given image, wherein each of the plurality of intermediate machine learning models is trained on a corresponding training dataset in the plurality of training datasets to eliminate the one or more image degradations associated with the given image based on the corresponding plurality of degradation factors; training an image transformation model on an additional real image training dataset to remove the one or more image degradations associated with the real images, wherein the image transformation model learns from the plurality of intermediate machine learning models; as well as The trained image transformation model is outputted by the computing device for eliminating image degradation.

2. A computer-implemented method as described in claim 1, wherein each of the multiple intermediate machine learning models comprises an encoder-decoder neural network with skip connections, wherein the given image is processed at different scales, wherein the output at a given scale is upsampled and concatenated with the input at a continuous scale.

3. The computer-implemented method of claim 2, wherein each intermediate machine learning model further comprises: (1) one or more space-to-depth (S2D) layers, where each S2D layer performs low-resolution processing by taking an input and reducing the spatial resolution of the input while increasing the number of channels to preserve information; and (2) one or more depth-to-spatial d2s layers corresponding to the one or more s2d layers.

4. A computer-implemented method as described in claim 1, wherein each of the multiple intermediate machine learning models is associated with a corresponding number of filters and corresponding space-to-depth parameters.

5. The computer-implemented method of claim 1 , further comprising: applying the plurality of trained intermediate machine learning models to a given real image in the additional real image training dataset to generate a corresponding output image; selecting the best transformed version of the given real image from among the generated output images; as well as A curated dataset is generated comprising pairs of real images and corresponding optimally transformed versions of the real images.

6. A computer-implemented method as described in claim 5, wherein applying the multiple trained intermediate machine learning models to the given real image includes applying each trained intermediate machine learning model at multiple resolutions.

7. The computer-implemented method of claim 5, wherein the training of the image transformation model comprises fine-tuning the image transformation model based on the curated dataset.

8. The computer-implemented method of claim 7, wherein the fine-tuning of the image transformation model comprises quantization-aware training (QAT).

9. The computer-implemented method of claim 1, further comprising: The synthetically degraded version of the sharp image is generated.

10. The computer-implemented method of claim 9, wherein the plurality of degradation factors include motion blur, and the generating further comprises: One or more motion kernels simulating camera shake due to hand shaking are generated, wherein the amount of motion blur is associated with a scale parameter related to the exposure time.

11. The computer-implemented method of claim 9, wherein the plurality of degradation factors include lens blur, and the generating further comprises: One or more generalized Gaussian blur kernels simulating the lens blur are generated.

12. The computer-implemented method of claim 9, wherein the plurality of degradation factors comprises one or more of additive noise, signal-dependent noise, or colored noise.

13. The computer-implemented method of claim 9, wherein the plurality of degradation factors includes one or more image compression artifacts.

14. The computer-implemented method of claim 9, wherein the plurality of degradation factors include one or more artifacts caused by saturated pixels, and the generating further comprises: determining whether the pixel value exceeds a saturation threshold; as well as Based on a determination that the pixel value exceeds the saturation threshold, the pixel value is multiplied by a random factor.

15. A computer-implemented method as described in claim 1, wherein the multiple intermediate machine learning models are teacher networks, wherein the image transformation model is a student network, and wherein the image transformation model learns from the multiple intermediate machine learning models based on knowledge distillation.

16. A computer-implemented method comprising: Receiving, by a computing device, an input image comprising one or more image degradations; predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training dataset of a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training dataset and having learned from the plurality of intermediate machine learning models; A predicted transformed version of the input image is provided by the computing device.

17. A computer-implemented method as described in claim 16, wherein each of the multiple intermediate machine learning models comprises an encoder-decoder neural network with skip connections, wherein the given image is processed at different scales, wherein the output at a given scale is upsampled and concatenated with the input at a continuous scale.

18. The computer-implemented method of claim 16, wherein the image transformation model comprises an encoder-decoder neural network with skip connections, and the image transformation model further comprises: (1) one or more space-to-depth (S2D) layers, where each S2D layer performs low-resolution processing by taking an input and reducing the spatial resolution of the input while increasing the number of channels to preserve information; and (2) one or more depth-to-spatial d2s layers corresponding to the one or more s2d layers.

19. The computer-implemented method of claim 16, wherein the plurality of degradation factors comprises one or more of motion blur, lens blur, image noise, image compression artifacts, or artifacts caused by saturated pixels.

20. A computing device comprising: one or more processors; as well as A data storage device having computer executable instructions stored thereon, the computer executable instructions, when executed by the one or more processors, causing the computing device to perform operations comprising: receiving, by the computing device, an input image comprising one or more image degradations; predicting a transformed version of the input image by an image transformation model, the image transformation model having been trained to eliminate the one or more image degradations associated with the input image, the training having included (1) training a plurality of intermediate machine learning models to eliminate the one or more image degradations, each of the plurality of intermediate machine learning models having been trained on a corresponding training dataset of a plurality of training datasets corresponding to a corresponding plurality of degradation factors, each training dataset having included a plurality of pairs of a clear image and a corresponding synthetically degraded version of the clear image, and (2) the image transformation model having been trained on an additional real image training dataset and having learned from the plurality of intermediate machine learning models; A predicted transformed version of the input image is provided by the computing device.

Images