Method and system for deblurring a blurred image

Abstract

A method and system (100) for image deblurring are described. First, the weights of a deblurring network (101) are obtained by meta-training on a main deblurring task and an auxiliary reconstruction task. Then, based on an application-time blurred input image, application-time training of the deblurring network (101) is performed to obtain values ​​for the application-time training weights. Application-time training includes performing the auxiliary reconstruction task on the application-time blurred input image and updating the weights of the deblurring network (101) based on an auxiliary loss computed from the auxiliary reconstruction task. A deblurred output image is generated based on the application-time blurred input image using the application-time training weights in the deblurring network (101).

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Nonprovisional Application No. 17 / 098,605, filed November 16, 2020, entitled “Method and System for Deblurring Blurred Images,” the contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of computer vision, and more particularly to methods and systems for deblurring blurred digital images. Background Technology

[0004] Images captured by digital cameras often appear unsatisfactory due to unwanted blurring. Blur in digital camera images can be caused by factors such as movement of objects in the scene during capture, camera movement, and low light conditions in the scene. The task of processing blurry images (i.e., unclear images) to generate cleaner images (i.e., slightly blurred or clearer images) is called deblurring. Blurring in an image can be uniform (i.e., every part of the image is blurred in the same way, such as blurring due to camera shake) or non-uniform (i.e., different parts of the image are blurred in different ways, such as blurring due to movement of objects in the scene). Non-uniform blurring in images can also be called dynamic scene blur, because it is often caused by the dynamic movement of objects in the scene during image capture. Deblurring non-uniformly blurred images is often very challenging because the blur is caused by the dynamic movement of objects in the scene during image capture, and this movement can be irregular and variable.

[0005] Some existing solutions for dynamic scene image deblurring (often referred to as image deblurring) utilize neural networks, particularly those that have been trained end-to-end for image deblurring (i.e., neural networks trained to predict deblurred images from inputs of blurred images). Such solutions typically require large training datasets (e.g., large labeled training datasets where the labeled training data consists of blurred images paired with baseline, clean images). Obtaining realistic blurred-clean image pairs is difficult, if not impossible, as digital cameras often cannot capture both blurred and clean images simultaneously. Therefore, training datasets often include synthetic data simulating blurred images. For example, blurred images can be synthesized from a sequence of consecutive frames from a high-frame-rate video (e.g., 240 fps). Such techniques can simulate blur caused by long exposure times (e.g., when capturing images in low-light environments). The synthesized blurred image can then be paired with a single frame from the sequence as a baseline, clean image. However, the synthesized blurred image may not accurately reflect a realistic blurred image.

[0006] Another challenge facing existing image deblurring solutions is that, while not impossible, it's difficult to include all possible types of dynamic scene blur in the training data, which (if possible) would lead to model overfitting. Existing image deblurring solutions are often sensitive to the training data. That is, trained neural networks perform well on images similar to those in the training dataset, but poorly on images different from the training dataset (e.g., outside the statistical distribution). When a trained neural network is applied to a real-world blurred image different from the training dataset, the clean image output by the trained neural network may contain some unwanted ghosting. Traditionally trained neural networks cannot adapt to new blurred images.

[0007] Therefore, there is a need to provide a method and system for adaptive deblurring of blurred images. Summary of the Invention

[0008] In various examples, this disclosure describes methods and systems for deblurring blurred images using neural networks. The disclosed neural network is designed to simultaneously perform a primary task of image deblurring and an auxiliary task. The auxiliary task is defined as related to the primary task but designed to be easier to learn (e.g., requiring no additional collection of benchmark truth labels at test time). In the disclosed examples, the trained neural network is further trained for a predetermined number of iterations using a specific blurred input image (i.e., a specific blurred image input to the neural network) to update the weights of the trained neural network such that the weights are customized (i.e., adapted) to process the specific blurred input image. After the further training of the neural network is complete, the further trained neural network deblurrs the specific blurred input image to generate a clean image. The customization (i.e., adaptation) of the weights of the further trained neural network based on the specific blurred input image and the ability to deblur the specific blurred input image using the further trained neural network can be referred to herein as adaptive deblurring, and the further training of the trained neural network to adapt to the specific blurred input image can be referred to herein as application-time training.

[0009] In the examples disclosed herein, adaptive deblurring (e.g., for deblurring real-world images captured by a camera of an electronic device) can be implemented in a real-world context, and a neural network can be trained to deblur real-world blurred images at the application stage. This disclosure describes methods and systems capable of training neural networks relatively quickly at the application stage (e.g., requiring only a few training iterations), thereby enabling on-the-fly adaptation of the neural network weights to perform deblurring for each specific real-world image. The disclosed methods and systems provide the following technical advantages: a specific blurred input image can be used to further train a trained neural network (e.g., a neural network already trained for deblurring blurred input images) on-the-fly at the application stage. The neural network trained here achieves improved performance in deblurring specific blurred input images compared to using an existing trained neural network.

[0010] When applying a trained neural network, training can be performed only through auxiliary tasks (e.g., assisted reconstruction tasks), using relatively few iterations (e.g., ten or fewer iterations, or five or fewer iterations). This provides the following technical advantages: training can be performed on the fly during the application phase, without excessive use of memory resources and / or excessive computation time, thus enabling execution by resource-constrained systems, such as handheld or mobile devices (e.g., smartphones, tablets, or laptops).

[0011] Compared to some existing deblurring methods and systems, the examples disclosed herein enable the generation of higher quality clean output images after image deblurring. The disclosed methods and systems are capable of further training a trained neural network to adapt the weights of the trained neural network to deblur a specific blurred input image, requiring the baseline truth (i.e., a clean image) of that specific image.

[0012] In some exemplary aspects, this disclosure describes an image deblurring method. The method includes obtaining a deblurring neural network with meta-training weights, wherein the meta-training weights were previously obtained by meta-training the deblurring neural network on a primary deblurring task and an auxiliary reconstruction task. The method also includes obtaining an application-time blurred input image. The method further includes: applying-time training the deblurring neural network with the meta-training weights using the application-time blurred input image to obtain the application-time training weights of the deblurring neural network by: performing the auxiliary reconstruction task on the application-time blurred input image to predict a reconstructed blurred image; and updating the meta-training weights of the deblurring neural network based on an auxiliary loss calculated according to an auxiliary loss function, the application-time blurred input image, and the reconstructed blurred image. The method further includes, after application-time training is complete, generating a deblurred output image using the deblurring neural network with the application-time training weights, based on the application-time blurred input image.

[0013] In any of the above examples, the application-time training includes multiple iterations of the application-time blurred input image, which is a single blurred image, and each iteration may include performing the auxiliary reconstruction task and updating the weights of the deblurring neural network.

[0014] In any of the examples above, the training process involves a maximum of five iterations.

[0015] In any of the above examples, the auxiliary reconstruction task is performed using features passed from the main deblurring task.

[0016] In any of the above examples, the deblurring neural network may include: a shared subnetwork for processing a blurred input image during the application, wherein the shared subnetwork is coupled to a main subnetwork for performing the primary deblurring task and an auxiliary subnetwork for performing the auxiliary reconstruction task; the main subnetwork includes a main output neural network layer for performing the primary deblurring task, the main subnetwork processing the output from the shared subnetwork to generate the deblurred output image; and the auxiliary subnetwork includes an auxiliary output neural network layer for performing the auxiliary reconstruction task, the auxiliary subnetwork processing the output from the shared subnetwork to generate the reconstructed blurred image.

[0017] In any of the above examples, the features output by the main output neural network layer can be copied to the neural network layer of the auxiliary sub-network.

[0018] In any of the above examples, in the master sub-network, the output from the master output neural network layer may represent the residual between the deblurred output image and the applied blurred input image, and the residual may be added to the applied blurred input image to generate the deblurred output image.

[0019] In any of the above examples, the meta-training of the deblurred neural network can be performed based on a training dataset, wherein the training dataset includes input-output pairs labeled with training data.

[0020] In some exemplary aspects, this disclosure describes a method for training a deblurring neural network. The method includes initializing the weights of the deblurring neural network. The method further includes: performing one round of meta-training of the deblurring neural network to perform a primary deblurring task and an auxiliary reconstruction task, and obtaining the meta-training weights through the following steps: sampling batches of training data, wherein the sampling batch includes multiple blurred training images, each blurred training image being paired with a corresponding clean training image in the sampling batch; for each given blurred training image in the sampling batch, performing the auxiliary reconstruction task on the given blurred training image to predict a corresponding reconstructed blurred image, and calculating a corresponding temporary weight set based on an auxiliary loss calculated according to an auxiliary loss function, the given blurred training image, and the corresponding reconstructed blurred image; for each pair of given blurred training images and corresponding clean training images in the sampling batch, performing the primary deblurring task to predict a corresponding predicted clean image, and calculating a corresponding primary loss based on a primary loss function, the corresponding clean training image, the corresponding predicted clean image, and the corresponding temporary weight set; and updating the weights of the deblurring neural network by summing the gradients of the correspondingly calculated primary losses. The method further includes storing the meta-training weights after meta-training is completed, which can be used to further train the deblurring neural network with the meta-training weights for deblurring applications when using blurred input images.

[0021] In any of the above examples, the auxiliary reconstruction task is performed using features passed from the main deblurring task.

[0022] In any of the above examples, the deblurring neural network may include: a shared subnetwork for processing the blurred training image, wherein the shared subnetwork is coupled to a main subnetwork for performing the primary deblurring task and an auxiliary subnetwork for performing the auxiliary reconstruction task; the main subnetwork includes a main output neural network layer for performing the primary deblurring task, the main subnetwork processing the output from the shared subnetwork to generate the deblurred output image; and the auxiliary subnetwork includes an auxiliary output neural network layer for performing the auxiliary reconstruction task, the auxiliary subnetwork processing the output from the shared subnetwork to generate the reconstructed blurred image.

[0023] In any of the above examples, the features output by the main output neural network layer can be copied to the neural network layer of the auxiliary sub-network.

[0024] In any of the above examples, in the master sub-network, the output from the master output neural network layer may represent the residual between the deblurred output image and the applied blurred input image, and the residual may be added to the applied blurred input image to generate the deblurred output image.

[0025] In some exemplary aspects, this disclosure describes an apparatus including a processor for executing instructions to cause the apparatus to perform any of the methods described above.

[0026] In some exemplary aspects, this disclosure describes a computer-readable medium storing instructions that, when executed by a processor of a computing device, cause the computing device to perform any of the methods described above. Attached Figure Description

[0027] The accompanying drawings, by way of example, illustrate exemplary embodiments of this application, wherein:

[0028] Figure 1 This is a block diagram of an example system architecture that can be used for meta-training and application-time training according to some embodiments of this disclosure.

[0029] Figure 2 This is a block diagram of an example hardware structure of a neural network processor according to some embodiments of the present disclosure.

[0030] Figure 3 This is a block diagram of an example architecture of a deblurring network according to some embodiments of the present disclosure.

[0031] Figure 4 This is a flowchart of an example method for meta-training and application-time training of a deblurred network according to some embodiments of the present disclosure.

[0032] Figure 5This is a flowchart of an example method for meta-training and application-time training of a deblurred network according to some embodiments of the present disclosure.

[0033] Figure 6 For implementing some embodiments of this disclosure Figure 5 Example pseudocode for the example method.

[0034] Figure 7 This is a flowchart of an example method for meta-training and application-time training of a deblurred network according to some embodiments of the present disclosure.

[0035] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0036] The technical solution of this disclosure will now be described in conjunction with the accompanying drawings.

[0037] The image deblurring methods and systems described in this paper can be applied to scenarios involving the deblurring of blurred digital images. In the disclosed methods and systems, a neural network is first trained via meta-training to perform a primary task and an auxiliary task. The primary task is to generate a clean image based on a blurred input image. The auxiliary task is defined as a task related to the primary task but easier to learn. For example, the auxiliary task could be to reproduce the blurred input image. The result of meta-training is the meta-training weights of the neural network. When the neural network has already been meta-trained on a training device other than the application device (e.g., a computing system), the neural network and the meta-training weights can be provided to or deployed to the application device (e.g., a consumer system), such as a handheld device, smartphone, tablet, or digital camera. When the neural network is trained on the application device, the meta-training weights are stored in the application device's memory after training is complete. In the application phase, the neural network with the meta-training weights can be further trained using a specific blurred input image (which does not have a corresponding baseline truth output clean image) to update the meta-training weights to the application-time training weights. After further training the neural network in the application phase, the neural network with the application-time training weights can be used to deblur a specific blurred input image to output a clean image. It is important to note that after the neural network with application-time trained weights outputs a clean image based on a specific blurred input image, the application-time trained weights can be discarded. That is, starting from the meta-trained weights, the neural network can be retrained to deblur each specific blurred input image. Therefore, the disclosed method and system provide the following technical effect: a neural network further trained on-the-fly during the application phase is capable of deblurring blurred images. In this disclosure, the application phase refers to using the deblurring network for real-world applications (e.g., deblurring real-world captured images), and may also be referred to as the inference phase, prediction phase, or online phase.

[0038] Training a neural network to update the meta-trained weights to application-time trained weights can be called application-time training. Application-time training of the neural network is performed only on auxiliary tasks. This allows application-time training of the neural network to be performed with relatively few iterations (e.g., ten or fewer iterations, or five or fewer iterations), enabling application-time training to be performed on the fly during the application phase without excessive use of memory resources and / or excessive computation time. The technical effect is that, after further training is complete, a neural network trained with application-time trained weights can be used to generate high-quality deblurred images from realistically blurred input images, requiring only a relatively short computation time (e.g., less than one second) and can be performed on resource-constrained systems, such as handheld or mobile devices (e.g., smartphones, tablets, or laptops) and desktop devices (e.g., desktop computers or personal computing devices).

[0039] To facilitate understanding of this disclosure, some existing techniques for image restoration are discussed. Some of these techniques are conventional techniques (i.e., not based on machine learning). Conventional techniques include, for example, performing deconvolution on an estimated blur kernel based on the assumption that the entire image is uniformly blurred. Such techniques typically perform poorly when the blur is non-uniform. Therefore, deblurring dynamic scenes using conventional techniques is often challenging.

[0040] Several attempts have been made to apply machine learning-based techniques to image restoration. For example, meta-learning (especially model-agnostic meta-learning (MAML)) has been proposed as a solution for achieving single image super-resolution (SISR). The goal of SISR is to obtain a high-resolution output image from only a single low-resolution input image. With MAML, a neural network is quickly trained (e.g., within a few gradient steps, or training iterations) to adapt its learned weights, and the neural network with the adapted learned weights is used to predict a high-resolution output image based on a specific low-resolution input image. Further training of the neural network to adapt its learned weights and using the neural network with the adapted learned weights to predict a high-resolution output image based on a specific input image can be called test-time training, as the specific input image can be considered a fitness test for the neural network. Test-time training typically requires rapidly training the neural network (e.g., within a few gradient steps, or training iterations) to be practically useful, and often also requires pairing each input data (e.g., each test input image) with benchmark truth output data (e.g., the expected output image) to facilitate training the neural network. However, this test-time training method may not be suitable for image deblurring problems. This is because a baseline truth output image cannot be obtained for realistically blurred images.

[0041] Another machine learning-based technique to consider for image restoration is auxiliary learning. In auxiliary learning, the auxiliary task is defined as related to the primary task (e.g., image deblurring) but easier to learn. Certain layers (and weights) of the neural network are shared between the auxiliary and primary tasks, such that updating the weights to improve the performance of the neural network in the auxiliary task will affect the performance of the neural network in the primary task. An auxiliary task can be defined to support learning for the primary deblurring task, such that benchmark truth output data for each test input can be easily obtained during test-time training. However, this approach typically provides little or no improvement to the performance of the primary deblurring task because updates to the neural network weights only benefit the performance of the auxiliary task.

[0042] In various examples, this disclosure describes methods and systems for image deblurring, in which a neural network trained via a training technique (referred to herein as meta-assisted learning) is used, which combines the advantages of meta-learning and assisted learning. As discussed below, the disclosed methods and systems enable test-time training of a neural network using a specific blurred input image that lacks a corresponding benchmark truth output clean image for rapid adaptation (e.g., within a few training iterations) of the neural network's previous meta-trained weights to improve the network's performance in predicting clean output images based on the specific blurred input image. In the examples disclosed herein, test-time training of the deblurring network can be performed, enabling deblurring of real-world images at the application stage. Therefore, this disclosure may refer to test-time training of the deblurring network as application-time training. Application-time training is performed on an auxiliary task, but the prior training of the neural network via meta-learning and the architecture of the neural network are designed to ensure that the primary deblurring task also benefits from application-time training. Examples of the disclosed methods and systems can provide good performance even in the presence of data distribution differences, where the blurriness of the test image or real-world image may differ from the blurriness found in the training dataset.

[0043] In some examples, this disclosure describes example methods for training neural networks to learn image deblurring tasks. Specifically, this disclosure describes examples of training neural networks to adapt their weights to improve deblurring performance on a particular input image. The training methods involve computer vision processing. Specifically, the training methods can be applied to data processing methods such as data training, machine learning, or deep learning to perform symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., blurred image data in the context of this disclosure) to obtain a trained neural network. These will be discussed further below. Furthermore, this disclosure describes an example method for image deblurring that can be performed using the neural network trained as described above. In the examples discussed herein, input data (e.g., real-world blurred image data) is used to further train the trained neural network to obtain output data (e.g., a deblurred image). It should be noted that the neural network training method and the deblurring method described herein can be considered as being based on the same idea, or as two parts of a system or two stages of a whole process: for example, a model training stage and a model application stage.

[0044] Typically, the examples disclosed in this article involve a wide range of neural network applications. For ease of understanding, some concepts related to neural networks and some relevant terms that may be associated with the examples disclosed in this article are described below.

[0045] A neural network is composed of neurons. A neuron is a computational unit that uses x... sThe intercept of 1 is used as input. The output of the computation unit can be:

[0046]

[0047] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s It is x s The weights are denoted by b, the neuron's offset (i.e., bias), and f is the neuron's activation function, used to introduce non-linear features into the neural network to transform the neuron's input into its output. The output of the activation function can serve as the input to neurons in the next convolutional layer of the neural network. For example, the activation function could be a sigmoid function. The neural network is formed by connecting multiple neurons. In other words, the output of one neuron can be the input of another neuron. The input of each neuron can be associated with a local receptive region in the previous layer to extract features from that local receptive region. A local receptive region can be a region consisting of several neurons.

[0048] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks consisting of a first layer (usually called the input layer), multiple hidden layers, and a final layer (usually called the output layer). The term "multiple" here doesn't have a specific metric. A layer is considered fully connected when there is a full connection between two adjacent layers. Specifically, for two adjacent layers (e.g., layer i and layer (i+1)) to be fully connected, each neuron in layer i must be connected to every neuron in layer (i+1).

[0049] The processing at each layer of a DNN can be relatively simple. In short, the operations at each layer are represented by the following linear relationship: in, For the input vector, For the output vector, Let W be the offset vector, W be the weights (also called coefficients), and α(.) be the activation function. In each layer, for the input vector... Perform the operation to obtain the output vector.

[0050] Because DNNs have a large number of layers, they also have a large number of weights W and offset vectors. The parameters in a DNN are defined as follows, taking the weight W as an example. In this example, in a three-layer DNN (i.e., a DNN with three hidden layers), the linear weight from the fourth neuron in the second layer to the second neuron in the third layer is expressed as: The superscript 3 indicates the layer number of the weight W (i.e., the third layer in this example), and the subscript indicates that the output is at index two of layer three (i.e., the second neuron in the third layer), while the input is at index four of layer two (i.e., the fourth neuron in the second layer). Typically, the weights from the k-th neuron in layer (L-1) to the j-th neuron in layer L can be represented as... It should be noted that the input layer does not have a W parameter.

[0051] In deep neural networks (DNNs), more hidden layers allow the DNN to better model complex situations (e.g., real-world scenarios). Theoretically, DNNs with more parameters are more complex and have greater capacity (likely referring to the learned model's ability to adapt to various possible scenarios), indicating that DNNs can perform more complex learning tasks. Training a DNN is the process of learning its weight matrix. The goal of training is to obtain the trained weight matrix, which consists of the learned weights W from all layers of the DNN.

[0052] A convolutional neural network (CNN) is a DNN with a convolutional structure. A CNN consists of a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be viewed as a filter. The convolution process can be viewed as using a trainable filter to convolve a two-dimensional (2D) input image or a convolutional feature map.

[0053] A convolutional layer is a layer of neurons that performs convolutional processing on the input in a CNN. In a convolutional layer, a neuron may be connected to only a subset of neurons in adjacent layers (i.e., not all neurons). That is, a convolutional layer is typically not a fully connected layer. A convolutional layer usually consists of several feature maps, each composed of neurons arranged in a rectangle. Neurons in the same feature map share weights. These shared weights are collectively called the convolutional kernel. Typically, the convolutional kernel is a two-dimensional weight matrix. It should be understood that the convolutional kernel can be independent of the way and location of image information extraction. A principle underlying convolutional layers is that the statistical information of one part of an image is the same as that of another part. This means that image information learned from one part of an image can also be applied to another part. Multiple convolutional kernels can be used in the same convolutional layer to extract different image information. Generally, the larger the number of convolutional kernels, the richer the image information reflected by the convolutional operation.

[0054] Convolutional kernels can be initialized as two-dimensional matrices with random values. During CNN training, the weights of the convolutional kernels are learned. One advantage of using convolutional kernels to share weights among neurons in the same feature map is that (compared to fully connected layers) the connections between convolutional layers in a CNN are reduced, thus lowering the risk of overfitting.

[0055] During DNN training, the predicted values ​​output by the DNN can be compared with the desired target value (e.g., the ground truth). The weight vector of each layer of the DNN (a vector containing the weights W of a given layer) is updated based on the difference between the predicted and desired target values. For example, if the predicted value output by the DNN is too high, the weight vector of each layer can be adjusted to lower the predicted value. This comparison and adjustment can be performed iteratively until a convergence condition is met (e.g., a predefined maximum number of iterations has been performed, or the predicted value output by the DNN has sufficiently converged with the desired target value). A loss function or objective function is defined as a way to quantitatively represent how close the predicted value is to the target value. The objective function represents the quantity to be optimized (e.g., minimized or maximized) so that the predicted value is as close as possible to the target value. The loss function more specifically represents the difference between the predicted and target values, and the goal of training the DNN is to minimize the loss function.

[0056] Backpropagation is an algorithm for training a deep neural network (DNN). It's used to adjust (also called update) the values ​​of parameters (e.g., weights) in the DNN to reduce the error (or loss) in the output. For example, a defined loss function is computed based on the forward propagation from the DNN's input to its output. Backpropagation computes the gradient of the loss function based on the DNN's parameters, and gradient algorithms (e.g., gradient descent) are used to update the parameters to reduce the loss function. Backpropagation is performed iteratively, thus converging to or minimizing the loss function.

[0057] See Figure 1 , Figure 1System 100 according to an example embodiment of this disclosure is illustrated. The following description should not be construed as limiting any example of this disclosure. As shown in system 100, labeled training data may be stored in database 130. Database 130 may be located on a server or in a data center, or may be provided as a service by a cloud computing service provider. In the context of this disclosure, labeled training data refers to training data used to learn the meta-training weights of a deblurring neural network 101 (also referred to as deblurring network 101 for simplicity). Labeled training data includes input-output image pairs, where the input image is a blurred image and the paired output image (i.e., the expected output image) is a baseline truth clean (i.e., unblurred) image. Labeled training data differs from application-time training data, which may be unlabeled real-world data (e.g., real-world images captured by application device 110, discussed below) or unlabeled test data. Application-time data includes blurred images without paired output images (e.g., without a baseline truth clean image). As will be discussed further below, the application-time training of the deblurring network 101 can be performed using a single input real-world image to obtain the application-time training weights of the deblurring network 101. The deblurring network 101, including the application-time training weights, can be used to predict the corresponding single deblurred output image based on a single input real-world image.

[0058] Database 130 may contain, for example, previously collected labeled training data that is typically used to train models related to image tasks (e.g., image recognition). The input images for the labeled training data stored in database 130 may, or additionally, be images optionally collected from application device 110 (which may be a user device) (e.g., with user consent). For example, images captured by the camera of application device 110 and stored on application device 110 may be optionally anonymized and uploaded to database 130 for storage as input images for labeled training data. The labeled training data stored in database 130 may include input-output image pairs, where the input image is a synthetic blurred image based on a paired output image (e.g., a baseline clean image).

[0059] As will be discussed further below, the meta-training device 120 can be used to train the deblurring network 101 based on training data stored in the database 130. Alternatively, the meta-training device 120 can use training data obtained from other sources (e.g., distributed storage (or cloud storage platform)) to train the deblurring network 101. The meta-trained deblurring network 101 (i.e., the result of the meta-training of the meta-training device 120) has a set of meta-training weights. According to the examples disclosed herein, the application device 110 can further train the meta-trained deblurring network 101 to deblur specific blurred real-world images. The application-time training of the application device 110 can be performed using images (e.g., digital photographs) captured by the camera (not shown) of the application device 110. The application device 110 may not have access to the training data stored in the database 130.

[0060] In the examples disclosed herein, the meta-trained deblurring network 101 can be implemented in the processing unit 111 of the application device 110. For example, the deblurring network 101 can be encoded and then stored as instructions in a memory (not shown) of the application device 110, and the processing unit 111 executes the stored instructions to implement the deblurring network 101. In some examples, the deblurring network 101 can be encoded and then stored as instructions in the memory of the processing unit 111 (e.g., the weights of the deblurring network 101 can be stored in a corresponding weight memory of the processing unit 111, which can be embodied as follows). Figure 2 The neural network processor 200 shown is illustrated. In some examples, the deblurring network 101 may be implemented in the integrated circuit of the application device 110 (as software and / or hardware). Although Figure 1 An example is shown where the meta-training device 120 and the application device 110 are separate. It should be understood that this disclosure is not limited to this embodiment. In some examples, separate meta-training device 120 and application device 110 may not exist. That is, meta-training of the deblurred network 101 and application-time training of the deblurred network 101 can be performed on the same device (e.g., application device 110).

[0061] Application device 110 can be a user device, such as a client terminal, mobile terminal, tablet computer, laptop computer, augmented reality (AR) device, virtual reality (VR) device, or in-vehicle terminal, etc. Application device 110 can also be a server, cloud computing platform, etc., which users can access through their user devices. Figure 1In this application device 110, an I / O interface 112 is included for data interaction with external devices. For example, the application device 110 can provide uploaded data (e.g., image data, such as photos and / or videos captured by the application device 110) to the database 130 via the I / O interface 112. Although Figure 1 An example of direct interaction between a user and application device 110 is shown. It should be understood that this disclosure is not limited to this embodiment. In some examples, the user device may be separate from the application device 110, with the user interacting with the user device, and the user device instead exchanging data with the application device 110 through I / O interface 112.

[0062] In this example, application device 110 includes a data storage device 114, which may be system memory (e.g., random access memory (RAM), read-only memory (ROM), etc.) or mass storage device (e.g., solid-state drive and hard disk drive, etc.). The data storage device 114 can store data accessible to the processing unit 111. For example, the data storage device 114 may be separate from the processing unit 111, storing captured and / or repaired images on the application device 110.

[0063] In some examples, the application device 110 may optionally call data and code from an external data storage system 150 for processing, or may store data and instructions obtained through the corresponding processing in the data storage system 150.

[0064] It is important to note that Figure 1 This is merely a schematic diagram of an example system architecture 100 according to an embodiment of this disclosure. Figure 1 The relationships and interactions between the devices, components, and processing units shown are not intended to limit this disclosure.

[0065] Figure 2 This is a block diagram of an example hardware structure of an example neural network processor 200 according to an embodiment of the present disclosure. The neural network processor 200 may be disposed on an integrated circuit (also known as a computer chip). Figure 1 In the application device 110 shown, the processing unit 111 performs calculations and implements the deblurring network 101 (including training during the application of the deblurring network). Alternatively, the neural network processor 200 may be configured... Figure 1 Meta-training of the deblurring network 101 is performed in the meta-training device 120 shown. All algorithms of the layers in the neural network (e.g., the layers of the deblurring network 101 discussed further below) can be implemented in the neural network processor 200.

[0066] The neural network processor 200 can be any processor capable of performing the computations required in a neural network (e.g., computations involving numerous XOR operations). For example, the neural network processor 200 can be a neural processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The neural network processor 200 can be a coprocessor of an optional host central processing unit (CPU) 220. For example, the neural network processor 200 and the host CPU 220 can be mounted on the same package. The host CPU 220 can be responsible for performing the core functions of the application device 110 (e.g., execution of the operating system, management communication, etc.). The host CPU 220 can manage the operation of the neural network processor 200, for example, by assigning tasks to the neural network processor 200.

[0067] The neural network processor 200 includes an arithmetic circuit 203. The controller 204 of the neural network processor 200 controls the arithmetic circuit 203 to extract data (e.g., matrix data) from the input memory 201 and weight memory 202 of the neural network processor 200, and to perform data operations (e.g., addition and multiplication operations).

[0068] In some examples, the arithmetic circuit 203 internally includes multiple processing units (also called process engines, or PEs). In some examples, the arithmetic circuit 203 is a two-dimensional pulsating array. In other examples, the arithmetic circuit 203 can be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some examples, the arithmetic circuit 203 is a general-purpose matrix processor.

[0069] In one example operation, the arithmetic circuit 203 retrieves the weight data of the weight matrix B from the weight memory 202 and caches the weight data in each PE of the arithmetic circuit 203. The arithmetic circuit 203 retrieves the input data of the input matrix A from the input memory 201 and performs matrix operations based on the input data of matrix A and the weight data of matrix B. The resulting partial or final matrix result is stored in the accumulator 208 of the neural network processor 200.

[0070] In this example, the neural network processor 200 includes a vector computation unit 207. The vector computation unit 207 includes multiple computation processing units. If needed, the vector computation unit 207 further processes the output from the computation circuit 203 (which can be retrieved from the accumulator 208 by the vector computation unit 207), such as vector multiplication, vector addition, exponentiation, logarithmic operations, or magnitude comparisons. The vector computation unit 207 can be primarily used for operations in non-convolutional or fully connected layers of the neural network. For example, the vector computation unit 207 can perform processing such as pooling or normalization on the operations. The vector computation unit 207 can apply a nonlinear function, such as a vector of accumulated values, to the output of the computation circuit 203 to generate activation values. These activation values ​​can be used by the computation circuit 203 as activation inputs for the next layer of the neural network. In some examples, the vector computation unit 207 generates normalized values, combined values, or a combination of normalized and combined values.

[0071] In this example, the neural network processor 200 includes a memory access controller 205 (also known as direct memory access control, or DMAC). The memory access controller 205 is used to access external memory (e.g., data memory 114 of execution device 110) via a bus interface unit 210. The memory access controller 205 can access data from external memory and directly transfer data to one or more memories of the neural network processor 200. For example, the memory access controller 205 can directly transfer weight data to weight memory 202, and input data can be directly transferred to unified memory 206 and / or input memory 201. Unified memory 206 is used to store input and output data (e.g., processing vectors from vector computation unit 207).

[0072] Bus interface unit 210 is also used for interaction between memory access controller 205 and instruction fetch memory (also called instruction fetch cache) 209. Bus interface unit 210 is also used to enable instruction fetch memory 209 to fetch instructions from memory outside neural network processor 200 (e.g., data memory 114 of application device 110). Instruction fetch memory 209 is used to store instructions for use by controller 204.

[0073] Typically, the unified memory 206, input memory 201, weight memory 202, and instruction fetch memory 209 are all part of the neural network processor 200's memory (also known as on-chip memory). The data memory 114 is independent of the neural network processor 200's hardware architecture.

[0074] Figure 3This is a block diagram of an example architecture for the Deblurring Network 101. Details of the meta-training and application-time training of the Deblurring Network 101 will be discussed further below.

[0075] The deblurring network 101 is meta-trained by jointly training the auxiliary task and the primary deblurring task. The input to the deblurring network 101 is the blurred input image (denoted as I). b This can be represented as a two-dimensional matrix, which encodes multiple individual pixels of an input image with multiple channels (e.g., red, green, and blue (RGB) channels). The following section combines... Figure 3 The example below describes the architecture of the deblurring network 101. It should be understood that the architecture of the deblurring network 101 can be modified (e.g., having fewer or more neural network layers). In the following discussion, for simplicity, the neural network layers (or blocks) of the deblurring network 101 will be referred to as layers.

[0076] In this example, the deblurring network 101 is a single-scale network with multiple convolutional layers 306 and multiple deconvolutional layers 308, used to process the blurred input image (I b In this example, each convolutional layer 306 and each deconvolutional layer 308 is followed by a corresponding residual layer 310a. The first convolutional layer 306 of the deblurring network 101 receives the blurred input image (I b ), and perform a convolution operation on the blurred input image to generate a blurred input image (I). b The output feature map (feature representation) of the feature encoding is provided as the input feature map (i.e., feature representation) to the first residual layer 310a.

[0077] Each corresponding convolutional layer 306 receives an output feature map (i.e., feature representation), which is the result of the previous convolutional layer 306 processing the input feature map (i.e., feature representation); and performs convolution operations on the input feature map (i.e., feature representation) using different convolutional kernels. When convolutional layer 306 is the first convolutional layer 306 in the deblurring network 101, the output feature map (i.e., feature representation) of convolutional layer 306 is a representation of the blurred input image (I... bThe features of the input feature map (i.e., the feature representation) are encoded; or, when the convolutional layer is another convolutional layer 306 in the deblurring network 101, the output feature map (i.e., the feature representation) of the convolutional layer 306 encodes the features of the input feature map (i.e., the feature representation). Each deconvolutional layer 308 decodes the feature map generated by the corresponding residual layer 310a of the deblurring network 101 to generate an output feature map (i.e., the feature representation). The deblurring network 101 has two output layers 302 and 304, referred to herein as the main output layer 302 and the auxiliary output layer 304. The main output layer 302 performs a convolutional operation to generate a feature map (i.e., the feature representation) combined with the blurred input image, thereby generating an output image (i.e., the predicted clean image) for the main deblurring task. The auxiliary output layer 304 generates an output image (i.e., the reconstructed blurred image) for the auxiliary task. In this example, the auxiliary output layer 304 is preceded by an auxiliary convolutional layer 306b (followed by an auxiliary residual layer 310c). Short connections (indicated by dashed arrows) are used to provide direct paths between residual layers 310a (followed by corresponding convolutional layers 306) to provide feature maps (feature representations) to residual layer 310b (followed by corresponding deconvolutional layers 308), and to provide direct paths between the main output layer 302 and the auxiliary convolutional layers 306. Short connections (also known as skip connections) can be used in residual neural networks to facilitate faster learning of the weights of the deblurring network 101.

[0078] The deblurring network 101 has a shared subnet 312, which includes multiple shared layers (including convolutional layer 306, deconvolutional layer 308, and residual layers 310a and 310b). These shared layers are coupled to a main subnet 314 that performs the primary task and an auxiliary subnet 312 that performs an auxiliary task (where the primary task is the deblurring task, and the auxiliary task is defined as learning to support the deblurring task, as described below). This means that training the deblurring network 101 to perform the auxiliary task will also affect (and potentially improve) the performance of the primary deblurring task. Furthermore, the deblurring network 101 includes a main subnet 314 with layers specific to the primary deblurring task (i.e., the primary output layer 302) and an auxiliary subnet 316 with layers specific to the auxiliary task (i.e., auxiliary convolutional layer 306, auxiliary residual layer 310, and auxiliary output layer 304). The main subnet 314 and the auxiliary subnet 316 may also be referred to as the main branch and the auxiliary branch, respectively. It should be understood that each subnet 312, 314, and 316 is defined as a corresponding set of one or more layers of the entire deblurring network 101 (e.g., as shown in the original text). Figure 3As shown in the example, the main subnet 314 processes the feature maps (i.e., feature representations) generated by the shared subnet 312 to generate an output image (i.e., the predicted deblurred image) for the main deblurring task. The auxiliary subnet 316 processes the feature maps (i.e., feature representations) generated by the shared subnet 312 to generate an output image (i.e., the reconstructed blurred image) for the auxiliary task. It should be understood that the deblurring network 101 can be implemented with different subnets and layers than those described in this example. However, short connections from the main output layer 302 to the auxiliary convolutional layer 306b are useful, as discussed further below.

[0079] The main deblurring task is based on the blurred input image I b Output the predicted clean image (i.e., the deblurred image) (denoted as ). Predicted clean image With blurred input image I b They have the same size and resolution (i.e., the same pixel size). Figure 3 In the examples, the primary deblurring task utilizes residual learning to predict the blurred input image I. b The difference between the predicted and the baseline truth expected output image (also known as the residual). The predicted residual is added to the original blurred input image Ib to output the predicted deblurred image. In this example, the auxiliary task is defined as self-supervised reconstruction, which means that the auxiliary task is based on the blurred input image I. b Output the reconstructed blurred image (denoted as ). In other words, the auxiliary task performed by the deblurring network 101 is to learn weights to blur the input image I. b Map to a set of feature maps and reconstruct the blurred image using the feature maps. It should be understood that different auxiliary tasks (instead of the reconstruction task) can be defined; for example, any other auxiliary task that supports the learning of the main deblurring task can be defined, where the auxiliary task can be trained at application time without requiring benchmark truth data separate from the input image at application time (or where benchmark truth data can be easily obtained).

[0080] As described above, training the deblurring network 101 to perform the auxiliary task also benefits the performance of the primary deblurring task. Performing the auxiliary task during the training of the deblurring network 101 provides regularization, which guides the learning of the weights of the deblurring network when performing the primary deblurring task, which can improve the performance of the primary deblurring task. It should be noted that the weights of the deblurring network 101 learned during training include weights related to blur information about the blurred input image, which can be used to reconstruct the blurred input image and generate a deblurred output image (i.e., a clean output image). The output features of the main output layer 302 are copied to the auxiliary convolutional layer 306b via short connections (also known as feature propagation) so that training can backpropagate when the auxiliary task is applied, thereby updating the weights of the deblurring network 101, including the weights in the convolutional and deconvolutional kernels, which in turn benefits the primary task.

[0081] Figure 4 This is a flowchart of an example method 400 for deblurring an input image using a deblurring network 101. Method 400 provides an overview of the different training phases for the deblurring network 101 and may include, for example, training by... Figure 1 The steps performed by the meta-training device 120 and the application device 110 shown.

[0082] Method 400 may begin with optional step 402. In optional step 402, pre-training may be performed. By jointly training the deblurring network 101 to perform the main deblurring task and the auxiliary task, the deblurring network 101 may be pre-trained using labeled training data (e.g., from database 130). For example, pre-training may be performed by obtaining the deblurred output image (from the main sub-network 314) and the reconstructed blurred image (from the auxiliary sub-network 316), calculating the loss for each sub-network using a loss function, and performing backpropagation based on the sum of the losses. Then, in a subsequent step 404, the pre-trained weights may be used as initial weights at the start of meta-training.

[0083] If optional step 402 is not performed, the weights of the deblurred network 101 can be randomly initialized at the start of meta-training. For example, each weight can be set to a corresponding random value.

[0084] In step 404, meta-training of the deblurred network 101 is performed. In some examples, meta-training can be performed using labeled training data from database 130. Figure 1Meta-training is performed by the application device 110. In other examples, meta-training may be performed by the application device 110. If pre-training is performed in step 402, the labeled training data used for meta-training may differ from the labeled training data used for pre-training (e.g., sampled from a different training dataset, or from different samples of the same training dataset). Meta-training may be performed until a convergence criterion is met (e.g., the loss function is optimized for the weight values ​​of the deblurred network 101, or a predetermined number of training iterations have been completed). Further details of meta-training will be described below.

[0085] The meta-training weights of the deblurring network 101 are then stored. If the meta-training is performed in a meta-training device 120 separate from the application device 110, the meta-training device 120 can transfer the meta-training weights to the application device 110, and the meta-training weights can be stored in the local memory of the application device 110. If the meta-training is performed by the application device 110, the meta-training weights can be directly stored in the local memory of the application device 110. The deblurring network 101 with the meta-training weights can be referred to as the meta-trained deblurring network 101.

[0086] In step 406, application-time training of the deblurring network 101 is performed using a specific blurred input image. The specific blurred input image can be, for example, a real-world image captured by a digital camera (e.g., a digital photograph), such as a standalone digital camera or a digital camera in an electronic device, such as a smartphone, laptop, or tablet. Application-time training of the deblurring network 101 allows the meta-trained weights to be updated to application-time training weights, making the deblurring network 101 suitable for deblurring the specific blurred input image. After application-time training of the meta-trained deblurring network 101, the deblurring network 101 with application-time training weights receives the specific blurred input image, generates a predicted clean image (i.e., a deblurred image) based on the specific input image, and outputs the predicted clean image. The result of application-time training of the deblurring network 101 is an application-time trained deblurring network 101 that performs better than the meta-trained deblurring network 101 in deblurring specific input images. Further details of the application-time training of the meta-trained deblurring network 101 will be described below.

[0087] The predicted clean image generated by the application-time trained deblurring network 101 can be stored, for example, in the local memory of the application device 110. The generated predicted clean image can also be displayed to (e.g., on the display screen of the application device 110) or otherwise presented to the user. After generating the predicted clean image (for a specific blurred input image), the application-time trained deblurring network 101 can revert to the meta-trained deblurring network 101 (i.e., the application-time trained weights can be discarded). For example, the values ​​of the application-time trained weights can be replaced with previously stored meta-trained weights. Application-time training of the deblurring network 101 can be repeated from the meta-trained weights for deblurring each specific blurred input image.

[0088] Figure 5 This is a flowchart of an example method for performing step 404 to perform meta-training of the deblurred network 101. For example, Figure 5 The method in can be derived from Figure 1 The meta-training device 120 shown performs the training. In some examples, meta-training may be performed by the application device 110 instead of the meta-training device 120.

[0089] Figure 6 Example pseudocode 600 is shown, which can be used (e.g., as software instructions executed by a processor) for implementation. Figure 5 The method in the middle. Figure 5 and Figure 6 We will discuss this together.

[0090] Optionally, in step 502, pre-trained weights for the deblurring network 101 can be obtained. If this is performed... Figure 4 If step 402 is performed to pre-train the deblurring network 101, then step 502 can be executed.

[0091] In step 504, parameters for the meta-training of the deblurring network 101 are initialized. For example, if pre-trained weights are obtained in step 502, the deblurring network 101 can be initialized with the values ​​of the pre-trained weights. If pre-trained weights are not available, the weights of the deblurring network 101 can be randomly initialized (e.g., by setting the value of each weight to a corresponding random value). Other parameters for the meta-training of the deblurring network 101 that can be initialized in step 504 include the gradient step size used to update the weights of the deblurring network 101 using gradient descent.

[0092] Step 504 can be executed using the instructions represented in line 602 of pseudocode 600. In line 602, the weights of the deblurring network 101 are denoted as θ, where θ s Indicates the weight of the shared subnet 312 (e.g., as shown in the image). Figure 3As shown, the weights in the shared layer of the shared subnet 312 of the deblurring network 101, θ pri This represents the weights of layers used only for the primary deblurring task (e.g., ...). Figure 3 As shown, the weights in the layers of the main subnet 314 of the deblurring network 101, θ aux This represents the weights of layers used only for auxiliary tasks (e.g., ...). Figure 3 As shown, the weights in the layers of the auxiliary subnet 316 of the deblurring network 101. The gradient step sizes used for calculating the gradient descent of the auxiliary loss function and the main loss function are denoted as α and β, respectively. For example, θ s θ pri and θ aux The values ​​of α and β can be obtained from pre-training or initialized randomly. For example, the values ​​of α and β can be chosen based on the expected convergence rate of gradient descent of the auxiliary loss function and the main loss function.

[0093] Meta-training involves training the deblurring network 101 to perform an auxiliary task, and then training the deblurring network 101 to perform the main deblurring task. Meta-training may include multiple rounds of training, where each round involves training the deblurring network 101 to perform multiple iterations of the auxiliary task, followed by one training iteration of the deblurring network 101 to perform the main deblurring task. Multiple rounds of training can be repeated until conditions are met (e.g., the auxiliary and main loss functions each converge to their optimal values, a predetermined number of rounds of training have been performed, or all labeled training data in the training dataset has been processed). Line 604 of pseudocode 600 can represent the instruction to repeat training until the convergence condition is met.

[0094] Now we will describe a training round (including multiple training iterations as described above).

[0095] In step 506, training for the auxiliary task is performed. This can be done via steps 508 and 510.

[0096] In step 508, the labeled training data is sampled. For example, a batch of labeled training data can be sampled from a database 130 containing labeled training data. The sampling batch of labeled training data includes paired training images, where each blurred input image is paired with a corresponding baseline truth-clean (i.e., unblurred) image. Each pair of blurred input images and corresponding baseline truth-clean training images may be referred to herein as an input-output image pair. The number of input-output image pairs in the sampling batch can be denoted as N (where N is a positive integer). Then, each blurred input image (denoted as...) is... ) and the corresponding baseline truth clean output image (denoted as The input-output image pairs are paired, where n is the index of each input-output image pair (n is a positive integer not greater than N). Line 606 of pseudocode 600 can represent the instructions for obtaining labeled training data.

[0097] In step 510, an auxiliary task is performed, and temporary weights are calculated based on the calculated auxiliary loss. Step 510 is iterated for each value of index n in the sampling batch (i.e., iterated from n=1 to n=N).

[0098] For the nth blurred image, use the blurred training image. The input image is used to perform an auxiliary task to generate a reconstructed blurred image, denoted as . Based on the auxiliary loss function and fuzzy training images and reconstructing blurred images Calculate the auxiliary loss (denoted as L) aux ), where the training images are blurred Used as baseline truth. Auxiliary loss L aux This can be calculated, for example, to reconstruct a blurred image. and blurred training images The absolute distance between them. It is important to note that clean training images... The weights of the deblurring network 101 are not used to train the network to perform auxiliary tasks. Then, the weights of the deblurring network 101 are updated according to the auxiliary loss using gradient descent relative to the weights of the network, which can be expressed by the following formula:

[0099]

[0100] Where n is the index of the blurred image used in one iteration of training the deblurring network 101 to perform the auxiliary task. This represents the temporary weights of the deblurring network 101 calculated based on one iteration, ← indicates an update operation (e.g., replacing the previous values ​​of the weights with updated values), and α is the step size of the gradient descent. This is a gradient operation. It should be noted that for each training iteration of the deblurring network, an auxiliary task is performed (i.e., for each corresponding blurred training image). In each training iteration, the corresponding temporary weights for the deblurring network 101 are calculated. set.

[0101] Each of the N blurred input images in the sampled batch of labeled training data is forward-propagated through the deblurring network 101. A corresponding temporary weight set is computed by calculating the corresponding auxiliary loss and updating the weights of the deblurring network 101 via backpropagation of the corresponding auxiliary loss (e.g., through gradient descent). Thus, N temporary weight sets are obtained, after which the method proceeds to step 512. Lines 608-614 of the pseudocode can represent the instructions for executing step 510.

[0102] In step 512, the deblurring network 101 is trained for the main deblurring task. This can be performed in step 514.

[0103] In step 514, the main deblurring task is performed to predict the clean image, and the weights of the deblurring network 101 are updated based on the computed main loss. Step 514 can be performed once for each round of training. To compute the main loss, the weights are calculated for every n blurred training images in the sampling batch. Generate a predicted clean image, denoted as .

[0104] Through the main loss function and benchmark truth clean image and the predicted clean image Calculate the principal loss (denoted as L) pri A clean benchmark image of the nth labeled training data from a sampling batch. (with the nth blurred input image) Pairing was used as the baseline truth. The primary loss L pri This can be calculated, for example, as the generated clean image. and clean training images The absolute distance between them. Main loss L pri The calculation is also based on the temporary weights computed while training the deblurring network 101 to perform the auxiliary task. For each labeled training data (input-output image pair) in a sampling batch of labeled training data, calculate the principal loss L. pri The gradients of all computed main losses are summed. Then, the weights of the deblurred network 101 are updated using the summed gradients, which can be expressed by the following formula:

[0105]

[0106] Where β is the step size of gradient descent. Lines 616-618 of the pseudocode can represent the instructions for executing step 514.

[0107] It should be noted that training the deblurring network 101 based on the primary loss is based on temporary weights calculated according to the auxiliary loss. Training the deblurring network 101 to perform the primary deblurring task is designed so that updates based on the auxiliary loss can improve the performance of the deblurring network 101 when performing the primary deblurring task.

[0108] The method then checks the conditions to determine whether the meta-training should end. If the conditions are not met (e.g., the auxiliary and main loss functions of the deblurred network 101 have not both converged), the method returns to step 506 to perform another round of training. If the conditions are met, the method proceeds to step 516.

[0109] In step 516, after meta-training is complete, the weights learned during meta-training are stored; these weights may be referred to as meta-training weights. For example, if meta-training is performed in a meta-training device 120 separate from application device 110, then meta-training device 120 may transfer the meta-training weights to application device 110, and the meta-training weights may be stored by application device 110 in its local memory. If meta-training is performed by application device 110, then the meta-training weights may be directly stored in the local memory of application device 110.

[0110] The meta-training weights can be stored and used as the initial weights for the deblurring network 101 for application-time training. Application-time training uses specific application-time blurred images (e.g., real-world blurred digital images without corresponding benchmark truth images) to update the meta-training weights of the deblurring network 101 in a relatively small number of training iterations (e.g., less than ten iterations, or fewer than ten iterations). The updated meta-training weights are the application-time training weights. The deblurring network 101 with application-time training weights performs better on specific blurred input images after application-time training.

[0111] like Figure 4 As shown, the application-time training of the deblurring network 101 follows its meta-training. However, it should be understood that the application-time training of the deblurring network 101 does not necessarily immediately follow its meta-training. For example, the meta-training weights may be stored in local memory by the application device 110. When it is desired to deblur a particular blurred input image, such as in response to user input on the application device 110 (e.g., the user selects the option to deblur the captured image), the meta-training weights can be used to train the weights later in the application-time training. Application-time training can be considered as training on real-world data (e.g., a real-world blurred image without a benchmark clean image) during the application phase, unlike meta-training performed on a training dataset that includes labeled training data (i.e., training data consisting of input-output pairs, where each input-output pair includes a blurred input image and a corresponding benchmark clean image).

[0112] Figure 7 This is a flowchart of an example method for training when performing step 406 to perform the application of the deblurring network 101. For example, Figure 7 The method in can be derived from Figure 1 The application device 110 shown is used for execution.

[0113] In step 702, parameters for training the deblurring network 101 at application time are initialized. Initialization may include obtaining the deblurring network 101 with meta-training weights (e.g., by initializing the weights of the deblurring network 101 with stored values ​​of the meta-training weights). Other parameters for application-time training that may be initialized in step 702 include the gradient step size for adapting the weights via gradient descent.

[0114] In step 704, an application-time blurred input image is acquired. For example, the application-time blurred input image may be a real-world image captured by the digital camera of the application device 110. The application-time blurred input image may be acquired in real time during image capture, or it may be a previously captured image, for example, retrieved from the local memory of the application device 110.

[0115] During training of the deblurring network 101, its weights are updated by further training the network to perform auxiliary tasks (e.g., self-supervised assisted reconstruction). This is because the deblurring network 101 (see...) Figure 3 Features passed from the main subnet 314 to the auxiliary subnet 316 are included, and updating the weights of the deblurring network 101 based on the auxiliary task also benefits the main deblurring task.

[0116] In step 706, an auxiliary task is performed on the applied blurred input image to predict the reconstructed blurred image. In this example, the auxiliary task is to reconstruct the blurred input image, and the features are passed from the main deblurring task. However, as discussed above, the auxiliary task can be defined as any other auxiliary task that supports learning the main deblurring task (i.e., learning to perform the auxiliary task enables the deblurring network 101 to learn weights to generate feature maps related to performing the main task).

[0117] In step 708, the weights of the deblurring network 101 are updated based on the calculated auxiliary loss (these weights were initialized to meta-training weights in step 702). The updated weights of the deblurring network 101 are referred to as the application-time training weights. The calculation of the auxiliary loss can be the same as the calculation of the auxiliary loss in meta-training (e.g., as shown in line 612 of pseudocode 600). That is, the auxiliary loss can be calculated based on the auxiliary loss function (the same auxiliary loss function used in meta-training), the application-time blurred input image, and the reconstructed blurred image. The update of the weights of the deblurring network 101 can be performed over several iterations (e.g., less than ten or less than five iterations) (by repeating steps 706 and 708). Alternatively, steps 706 and 708 can be performed only once as one iteration. Typically, the number of iterations performed during application-time training should be equal to the number of iterations performed during the auxiliary task during meta-training for better performance.

[0118] In step 710, after application-time training is complete (e.g., after the defined number of iterations in steps 706 and 708 have been performed), the deblurring network 101 with application-time training weights is used to generate a predicted clean image based on the application-time blurred input image. For example, the predicted clean image may be stored locally on the application device 110 and / or may be output to a display device (e.g., the display screen of the application device 110) for user viewing.

[0119] In various examples, this disclosure describes methods and systems for image deblurring. A machine learning-based method is described that enables deblurring of dynamic scenes and images with uniform image blur.

[0120] The disclosed method for training the deblurring network involves meta-training and application-time (or test-time) training to perform an auxiliary task (also known as meta-assisted learning), which allows the weights of the deblurring network 101 to be adapted to each specific application-time input image in relatively few iterations (e.g., less than five application-time iterations). This rapid adaptation of the weights of the deblurring network 101 can make the disclosed deblurring method and system useful in practical applications (e.g., for deblurring realistic images captured by a digital camera of an electronic device).

[0121] Design deblurring networks (e.g., with a system architecture) such that training on auxiliary tasks (e.g., self-supervised auxiliary reconstruction tasks) is beneficial for the primary deblurring task.

[0122] In various evaluation tests, the examples disclosed herein have been found to outperform some other existing machine learning-based deblurring techniques without adaptation.

[0123] This disclosure describes examples in the context of image deblurring, including deblurring captured digital images. It should be understood that this disclosure is applicable to both still images (e.g., digital photographs) and video images (e.g., each image to be deblurred is a corresponding frame in a video).

[0124] Although described in the context of deblurring, it should be understood that this disclosure is applicable to other image restoration tasks, such as dehazing, denoising, or deraining. For example, the primary deblurring task could be a primary dehazing task, a primary denoising task, or a primary deraining task. It should be understood that this disclosure is not limited to image deblurring only. Any task that would benefit from application-time training and for which no benchmark truth data is available during application-time training can benefit from this disclosure.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this disclosure can be implemented in electrical hardware, or a combination of computer software and electrical hardware. Whether the function is executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but should not consider that such implementation is beyond the scope of this disclosure.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0127] It should be understood that the disclosed systems and methods can be implemented in other ways. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units; they may be located in one location or distributed across multiple network units. Some or all units can be selected according to actual needs to achieve the objectives of the embodiments. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0128] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to instruct a computer device (which may be a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a universal serial bus (USB) flash drive, a removable hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0129] The above description is merely a specific embodiment of this application and is not intended to limit the scope of protection of this disclosure. Any variations or substitutions that are readily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should fall within the scope of protection of this disclosure.

Claims

1. A method for image deblurring, characterized in that, include: Obtain a deblurring neural network with meta-training weights, wherein the meta-training weights were previously obtained by meta-training the deblurring neural network on a primary deblurring task and an auxiliary reconstruction task; Obtain a blurred input image when applying the application; Using the applied-time blurred input image, the deblurring neural network with the meta-training weights is trained in an applied-time manner to obtain the applied-time training weights of the deblurring neural network through the following steps: The auxiliary reconstruction task is performed on the blurred input image during the application to predict the reconstructed blurred image; as well as The meta-training weights of the deblurring neural network are updated based on the auxiliary loss calculated according to the auxiliary loss function, the blurred input image at the time of application, and the reconstructed blurred image. as well as After training is completed at the application time, the deblurred output image is generated based on the application-time blurred input image using the deblurred neural network with the application-time training weights.

2. The method according to claim 1, characterized in that, The application-time training includes multiple iterations of the application-time blurred input image, which is a single blurred image, wherein each iteration includes performing the auxiliary reconstruction task and updating the weights of the deblurring neural network.

3. The method according to claim 2, characterized in that, The application training includes a maximum of five iterations.

4. The method according to any one of claims 1 to 3, characterized in that, The auxiliary reconstruction task is performed using features passed from the main deblurring task.

5. The method according to any one of claims 1 to 3, characterized in that, The deblurring neural network includes: A shared subnet for processing a blurred input image during the application, wherein the shared subnet is coupled to a main subnet for performing the primary deblurring task and an auxiliary subnet for performing the auxiliary reconstruction task; The main subnet includes a main output neural network layer for performing the main deblurring task, and the main subnet processes the output from the shared subnet to generate the deblurred output image; The auxiliary subnet includes an auxiliary output neural network layer for performing the auxiliary reconstruction task, and the auxiliary subnet processes the output from the shared subnet to generate the reconstructed blurred image.

6. The method according to claim 5, characterized in that, The features output by the main output neural network layer are copied to the neural network layer of the auxiliary subnet.

7. The method according to claim 5, characterized in that, In the main subnet, the output from the main output neural network layer represents the residual between the deblurred output image and the applied blurred input image, and the residual is added to the applied blurred input image to generate the deblurred output image.

8. The method according to any one of claims 1 to 3, characterized in that, The meta-training of the deblurring neural network is performed based on a training dataset, which includes input-output pairs labeled with training data.

9. A training method for a deblurred neural network, characterized in that, include: Initialize the weights of the deblurring neural network; Perform one round of meta-training on the deblurring neural network to perform the main deblurring task and the auxiliary reconstruction task, and obtain the meta-training weights through the following steps: The training data is sampled in batches, wherein each batch of sampled data includes multiple blurred training images, and each blurred training image is paired with a corresponding clean training image in the batch of sampled data. For each given blurred training image in the sampling batch, the auxiliary reconstruction task is performed on the given blurred training image to predict the corresponding reconstructed blurred image, and a corresponding temporary weight set is calculated based on the auxiliary loss calculated according to the auxiliary loss function, the given blurred training image and the corresponding reconstructed blurred image; For each pair of given blurred training images and corresponding clean training images in the sampling batch, the main deblurring task is performed to predict the corresponding predicted clean image, and the corresponding main loss is calculated based on the main loss function, the corresponding clean training image, the corresponding predicted clean image, and the corresponding temporary weight set. as well as The weights of the deblurring neural network are updated by calculating the gradient summation of the corresponding main losses; as well as After meta-training is completed, the meta-training weights are stored, and the deblurring neural network with the meta-training weights can be further trained for deblurring applications when dealing with blurred input images.

10. The method according to claim 9, characterized in that, The auxiliary reconstruction task is performed using features passed from the main deblurring task.

11. The method according to claim 9 or 10, characterized in that, The deblurring neural network includes: A shared subnet for processing the blurred training images, wherein the shared subnet is coupled to a main subnet for performing the main deblurring task and an auxiliary subnet for performing the auxiliary reconstruction task; The main subnet includes a main output neural network layer for performing the main deblurring task, and the main subnet processes the output from the shared subnet to generate the deblurred output image; The auxiliary subnet includes an auxiliary output neural network layer for performing the auxiliary reconstruction task, and the auxiliary subnet processes the output from the shared subnet to generate the reconstructed blurred image.

12. The method according to claim 11, characterized in that, The features output by the main output neural network layer are copied to the neural network layer of the auxiliary subnet.

13. The method according to claim 11, characterized in that, In the main subnet, the output from the main output neural network layer represents the residual between the deblurred output image and the applied blurred input image, and the residual is added to the applied blurred input image to generate the deblurred output image.

14. An image deblurring device, characterized in that, include: Processor, for executing instructions to cause the device to: Obtain a deblurring neural network with meta-training weights, wherein the meta-training weights were previously obtained by meta-training the deblurring neural network on a primary deblurring task and an auxiliary reconstruction task; Obtain a blurred input image when applying the application; Using the applied-time blurred input image, the deblurring neural network with the meta-training weights is trained in an applied-time manner to obtain the applied-time training weights of the deblurring neural network through the following steps: The auxiliary reconstruction task is performed on the blurred input image during the application to predict the reconstructed blurred image; as well as The meta-training weights of the deblurring neural network are updated based on the auxiliary loss calculated according to the auxiliary loss function, the blurred input image at the time of application, and the reconstructed blurred image. as well as After training is completed at the application time, the deblurred output image is generated based on the application-time blurred input image using the deblurred neural network with the application-time training weights.

15. The device according to claim 14, characterized in that, The application-time training includes multiple iterations of the application-time blurred input image, which is a single blurred image, wherein each iteration includes performing the auxiliary reconstruction task and updating the weights of the deblurring neural network.

16. The device according to claim 15, characterized in that, The application training includes a maximum of five iterations.

17. The device according to any one of claims 14 to 16, characterized in that, The auxiliary reconstruction task is performed using features passed from the main deblurring task.

18. The device according to any one of claims 14 to 16, characterized in that, The deblurring neural network includes: A shared subnet for processing a blurred input image during the application, wherein the shared subnet is coupled to a main subnet for performing the primary deblurring task and an auxiliary subnet for performing the auxiliary reconstruction task; The main subnet includes a main output neural network layer for performing the main deblurring task, and the main subnet processes the output from the shared subnet to generate the deblurred output image; The auxiliary subnet includes an auxiliary output neural network layer for performing the auxiliary reconstruction task, and the auxiliary subnet processes the output from the shared subnet to generate the reconstructed blurred image.

19. The device according to claim 18, characterized in that, The features output by the main output neural network layer are copied to the neural network layer of the auxiliary subnet.

20. The device according to any one of claims 15 to 16, characterized in that, The meta-training of the deblurring neural network is performed based on a training dataset, which includes input-output pairs labeled with training data.

21. A computer program product comprising instructions, characterized in that, When the instructions are executed by a computer, the computer performs the method according to any one of claims 1 to 13.

22. A computer-readable medium including instructions, characterized in that, When the instructions are executed by a computer, the computer performs the method according to any one of claims 1 to 13.

Images