A deep-prior based self-supervised multi-scale blind image deblurring method

Abstract

The application discloses a kind of self-supervised multi-scale blind image deblurring methods based on depth prior, utilize network structure implicit modeling image prior, only blur image is as supervision signal training multi-scale deblurring network, estimate multi-scale clear image and blur kernel simultaneously.The specific method of the above method is: blind deblurring problem is decomposed into blur kernel estimation subproblem and image estimation subproblem, is solved alternately in the mode of network and mathematical model until convergence;Multi-input multi-output image generation network is used to generate multi-scale clear image;The exact solution of blur kernel regularization constraint model is solved, and the blur kernel is estimated under each scale.The method of the application does not need paired clear / blur image dataset;The adaptability of optimization model is combined with the learning ability of depth network, without considering the design of complex image prior;Through multi-scale joint deblurring, the estimation problem of large size blur kernel is effectively solved, and the time overhead caused by iteration from coarse to fine is avoided.

Description

Technical Field

[0001] This invention belongs to the field of image deblurring, and more specifically, it is a self-supervised multi-scale blind image deblurring method based on depth prior. Background Technology

[0002] During image acquisition, images are often blurred to some extent due to factors such as atmospheric turbulence, relative motion between the imaging device and the target, and inaccurate focusing of the imaging device. This is a common image degradation process. Image deblurring technology uses image processing methods to restore clear images. By modeling the image degradation process and solving the inverse process of the degradation model, a clear image can be restored from a blurred image. It has wide applications in many fields such as traffic monitoring, biomedicine, astronomical observation, and remote sensing.

[0003] For the degradation process of uniformly blurred images, image deblurring methods can be divided into non-blind and blind image deblurring methods, depending on whether the blur kernel is known. Blind image deblurring methods estimate both the original sharp image and the blur kernel simultaneously when the blur kernel is unknown. The number of equations is less than the number of unknowns, exhibiting severe ill-conditioning. Blind image deblurring is an ill-conditioned inverse problem, requiring the introduction of prior image information to constrain the space of feasible solutions. Existing blind image deblurring methods can be broadly categorized into two types: optimization model-based methods and deep learning-based methods. Natural image priors provide effective additional information for image restoration, constraining the space of feasible solutions to the image restoration problem. Optimization model-based blind image deblurring methods utilize traditional statistical priors to mathematically model the statistical characteristics of natural images; however, mathematical expressions struggle to represent complex natural image priors. Deep learning-based methods adaptively learn the features of sharp images through deep neural networks (DNNs), and can be divided into supervised and unsupervised learning methods.

[0004] Supervised learning-based blind image deblurring methods require training the network with paired blurred / sharp image datasets to learn the mapping from blurred to sharp images. Multi-scale deblurring deep networks effectively solve the large-scale blurring problem. Nah et al. pioneered the use of multi-scale convolutional neural networks, simulating a pyramid structure by cascading multiple sub-networks to learn the mapping relationship between blurred and sharp images end-to-end. Cho et al. improved the multi-scale deblurring strategy by using a multi-input multi-output U-Net network to model the estimation process of sharp images, performing multi-scale deblurring within a single U-Net, and fusing multi-scale information. Compared with the structure of cascading multiple sub-networks, this significantly reduces the number of parameters that need to be learned. However, a problem with supervised learning-based blind image deblurring methods is that when the features of the image to be restored differ significantly from those of the training dataset, the reconstruction quality of the network cannot be guaranteed.

[0005] Unsupervised learning-based blind image deblurring methods do not require training the network with a dataset. Self-supervised learning is a commonly used unsupervised learning method that uses only the blurred image as the supervision signal, without needing a blur kernel or ground truth of the sharp image, making it better applicable to real-world scenarios. In recent years, deep priors have utilized the deep network itself to implicitly model prior information of the image, without needing to explicitly express the prior using a mathematical model, and can represent more complex image structural information. Ulyanov et al. discovered that during the process of the network fitting a degraded image, a sharp image is generated first, and pioneered the proposal to use the deep network itself to capture natural image priors, proposing DIP-Net (Deep image prior net, DIP-Net) on the basis of U-Net for handling image restoration tasks. Ren et al. applied the idea of ​​deep priors to the image deblurring problem and proposed the SelfDeblur model, using DIP-Net to estimate the sharp image and a fully connected network FCN to estimate the blur kernel, alternately iteratively updating the parameters of the two networks while estimating the blur kernel and the sharp image. Bai et al. proposed the Fast-SelfDeblur model, building upon SelfDeblur. This model combines networks and mathematical models to decompose the joint optimization problem of sharp image and blur kernel into two sub-problems: image estimation and blur kernel estimation. It uses DIP-Net to estimate the sharp image and a regularized least squares model to estimate the blur kernel, solving the blind image deblurring problem through an alternating solution approach. Compared to SelfDeblur's method, Fast-SelfDeblur estimates the blur kernel by solving the exact solution of the regularized model, and it directly estimates the two-dimensional blur kernel without representing it as a one-dimensional vector, preserving the two-dimensional spatial correlation of the blur kernel and accelerating algorithm convergence.

[0006] This invention discloses a self-supervised multi-scale blind image deblurring method based on deep priors. The method trains a multi-scale deblurring network in a self-supervised learning manner and decomposes the joint modeling problem of blind deconvolution into a fuzzy kernel estimation subproblem and an image estimation subproblem. The network model and the mathematical model are solved alternately. The novelty and inventiveness of this invention are reflected in the following aspects: ① Based on self-supervised methods, only the blurred image itself is used as a supervisory signal to estimate the sharp image. Compared with supervised deblurring methods, it does not require training on paired sharp and blurred datasets, and has stronger generalization ability; ② It combines the learning ability of deep network methods with the adaptability of optimization model methods. It uses deep networks to generate sharp images and implicitly models image priors through the network structure, eliminating the need for complex manual priors. By solving the exact solution of the regularized least squares problem about the blur kernel, it directly estimates the blur kernel at each scale separately. The above two steps are solved alternately until convergence; ③ It uses a multi-input multi-output image generation network in a single U-Net for multi-scale deblurring, solving the problem of estimating large-size blur kernels. It uses multi-scale networks to estimate sharp images at multiple scales, avoiding the process of estimating images from coarse to fine in traditional optimization model methods, reducing time overhead. Furthermore, it jointly estimates sharp images and blur kernels, eliminating the need for non-blind image deblurring methods. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a self-supervised multi-scale blind image deblurring method based on depth priors to restore the original clear image.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following solutions:

[0009] A self-supervised multi-scale blind image deblurring method based on depth prior is characterized by comprising the following four steps:

[0010] Step 1. Initialize multi-scale random inputs and network parameters, and set hyperparameters;

[0011] This invention constructs a multi-scale image generation network comprising four scales; it initializes the network input and network parameters, with the input being a random matrix {z}. s}, s∈{0, 1, 2, 3}, z 3 Let z be a random matrix uniformly distributed on the interval (0, 1), and the rest are random matrices. s s = 0, 1, 2 is derived from z 3 Perform 2 (3-s) The network parameters are obtained by upsampling from nearest neighbors; the He method is used to initialize the network parameters to 0; the input is a random matrix {z}. s The initial estimates of sharp images at four scales are calculated through forward propagation of the multi-scale image generation network.

[0012] Step 2. Estimate the multi-scale fuzzy kernel;

[0013] Fixed network parameters θ k-1 Resulting clear image Omit the superscript 's', solve the exact solution of the following linear equation system, and estimate the fuzzy kernel 'h' separately at each scale. k :

[0014] Avec(h k = vec(Z)

[0015] In the formula, the coefficient matrix A is

[0016] A=γ[vec(F)vec(U) T +vec(K)vec(V) T ]+I

[0017] The expression for Z is

[0018]

[0019] Matrices U, V, F, and K are respectively

[0020]

[0021] Where vec(·) is the matrix vectorization operation. Indicates Fourier transform, This represents the inverse Fourier transform. This represents the complex conjugate of the Fourier transform;

[0022] Step 3. Estimate the sharp image at multiple scales;

[0023] Estimation of the current multi-scale fuzzy kernel From θ k-1 Update θ k Generate multi-scale sharp image estimation It includes the following 3 steps:

[0024] Step 3.1 Calculate the gradient of the loss function with respect to the network parameters:

[0025] The loss function of the network in this invention is:

[0026]

[0027] Calculate the gradient of the loss function L(θ) with respect to the parameter θ.

[0028] Step 3.2 Update the network parameters of the multi-scale image generation network:

[0029] Update network parameters θ using the Adam gradient descent algorithm k :

[0030]

[0031] In the formula, η is the learning rate, ∈ is a constant, and the momentum after bias correction is... and second momentum They are respectively

[0032]

[0033] Where ⊙ represents element-wise multiplication, 0≤β1<1, 0≤β2<1;

[0034] Step 3.3 Generate multi-scale sharp images:

[0035] Clear images at four scales after updating network parameters via network forward propagation.

[0036] Step 4. Determine whether the loss function has converged or reached the maximum number of iterations K, and output the estimates of the blur kernel and the clear image;

[0037] Through steps 2 and 3, the estimate of the multi-scale fuzzy kernel is obtained. And the estimation of multi-scale sharp images Updated to Complete one iteration of the objective function; if the loss function converges to below a certain threshold or reaches the maximum number of iterations K, stop the iteration and output the final blur kernel and sharp image estimate; otherwise, repeat steps 2 and 3.

[0038] Preferably, the initial learning rate η0 of the multi-scale image generation network is 1×10⁻⁶. -3 The learning rate decay factor is 0.5.

[0039] Preferably, the clear image regularization parameter λ x The values ​​are 0 and 1 in the Lai and Kohler datasets, respectively.

[0040] Preferably, the fuzzy kernel regularization parameter λ h The values ​​were 10 and 120 in the Lai and Kohler datasets, respectively.

[0041] Preferably, the maximum number of iterations K is 2000.

[0042] Preferably, the number of input data channels in the image generation network is fixed at 16.

[0043] This invention discloses a self-supervised multi-scale blind image deblurring method based on deep priors. It utilizes the network structure to implicitly model prior information of the image, training the multi-scale deblurring network only with blurred images as supervisory signals, while simultaneously estimating sharp images and blur kernels at multiple scales. The TV-regularized blind deblurring problem is decomposed into a blur kernel estimation subproblem and an image estimation subproblem, which are solved iteratively using both network and mathematical models until convergence. A multi-input multi-output image generation network is used to generate sharp images at multiple scales. The exact solution of the regularized least squares model for blur kernels is directly solved, and the blur kernel is estimated separately at each scale. This method does not require training the network with paired sharp / blurred image datasets; it combines the adaptability of optimization model methods with the learning ability of deep network methods, eliminating the need to consider complex image priors; and it effectively solves the estimation problem of large-size blur kernels through multi-scale joint deblurring, avoiding the time overhead of coarse-to-fine iteration, and jointly estimating sharp images and blur kernels eliminates the need for non-blind image deblurring methods. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an overall structural diagram of the self-supervised multi-scale blind image deblurring method based on depth prior provided in the embodiments of the present invention;

[0046] Figure 2 The flowchart shows the self-supervised multi-scale blind image deblurring method based on depth prior provided in the embodiments of the present invention.

[0047] Figure 3 The image generation network structure diagram is shown in the embodiment of the present invention for the self-supervised multi-scale blind image deblurring method based on depth prior.

[0048] Figure 4 The diagram shows the structure of each sub-module in the image generation network of the self-supervised multi-scale blind image deblurring method based on depth prior provided in the embodiments of the present invention.

[0049] Figure 5 This is a comparison chart of quantitative results of the self-supervised multi-scale blind image deblurring method based on depth prior provided in the embodiments of the present invention on large-size blur kernels on the Kohler dataset. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In the problem of image deblurring, the degradation process of a uniformly blurred image can be represented by the following convolution form:

[0052] y = h*x + n (1)

[0053] In the formula, y represents the blurred image, h represents the blur kernel, x represents the sharp image, n represents noise, and * represents a two-dimensional convolution operation. Blind image deblurring estimates the sharp image x from the blurred image y, given that the blur kernel h is unknown.

[0054] To solve for large-sized blur kernels, blind image deblurring methods based on optimization models estimate sharp images from coarse to fine using an image pyramid approach. In contrast, deep learning-based multi-scale deblurring networks avoid the time overhead of iterative coarse-to-fine deblurring by jointly deblurring multiple scales. Furthermore, traditional statistical priors are limited by their inability to model complex image priors mathematically, while deep priors implicitly capture image priors using network structures. This invention discloses a self-supervised multi-scale blind image deblurring method based on deep priors, which simultaneously estimates sharp images at multiple scales using a multi-input, multi-output multi-scale deblurring network.

[0055] This invention incorporates a multi-scale deblurring network into a TV regularized blind deblurring model, and expresses the blind deblurring problem as the following optimization problem:

[0056]

[0057] In the formula, s is the scaling factor, and y s For a blurred image at scale s, h s Let λ be the fuzzy kernel at scale s. x λ is the regularization parameter for a sharp image. h f({z) is the regularization parameter of the fuzzy kernel. s};θ) is a multi-scale image generation network, where θ is the network parameter, z s Let x be a random matrix input at scale s. sThe target image is the clear image output at scale s. The first term in the objective function represents the data fidelity term, ensuring that the reconstructed image conforms to the degradation model; the second term represents the regularization constraint term for the clear image, which is constrained by TV regularization to ensure the smoothness of the estimated image; the third term represents the regularization constraint term for the blur kernel, which is constrained by the Frobenius norm to achieve l2 norm regularization, so that it satisfies the convex optimization condition and can quickly converge to the global optimum. Then, the blur kernel is sparsified to ensure its sparsity.

[0058] Equation (2) is an optimization problem with two variables θ and h. This invention solves equation (2) alternately, then decomposes it into two sub-problems: fuzzy kernel estimation and image estimation. In the fuzzy kernel estimation sub-problem, the network parameter θ is fixed. k-1 Generate clear images Solving the fuzzy kernel In the image estimation subproblem, the fuzzy kernel is fixed. Using the degraded image as a supervision signal, the network parameters θ are updated using the gradient descent method. k Generate clear images This process involves iteratively updating the sharp image and the blur kernel alternately until the algorithm converges, thereby achieving a joint estimation of the sharp image and the blur kernel. Figure 1 The overall structure of the self-supervised multi-scale blind image deblurring method based on depth prior disclosed in this invention combines a network model and a mathematical model. It uses a single U-Net network with multiple inputs and multiple outputs, i.e., a multi-scale image generation network, to estimate the sharp image. The process of updating θ using the gradient of the loss function with respect to the network parameter θ is the process of multi-scale image updating. The closed-form solution of the blur kernel estimation subproblem at each scale is directly solved to estimate the multi-scale blur kernel.

[0059] This invention discloses a self-supervised multi-scale blind image deblurring method based on depth priors to restore the original clear image. See also Figure 2 The above method includes at least the following four steps.

[0060] Step 1. Initialize multi-scale random input and network parameters, and set hyperparameters.

[0061] This invention constructs a multi-input multi-output image generation network with four scales, simultaneously performing blind image deblurring at different scales. First, the network input and parameters are initialized; the multi-scale network input is a random matrix {z}. s}, where s represents the scale factor, s∈{0,1,2,3}, z 3 Let z be a random matrix uniformly distributed on the interval (0, 1), and the rest are random matrices. s s = 0, 1, 2 is derived from z 3 Perform 2 (3-s)It is obtained by upsampling the nearest neighbor. The input z... 0 The output x is the same size as the original image. 0 This results in the restored, clear image. The network parameters are initialized using the He method, with an initial parameter value of θ0. The input is a random matrix {z}. s}, after passing through the multi-scale image generation network f({z s};θ0) can yield initial estimates of sharp images at four scales.

[0062] The hyperparameters set in this invention include the learning rate η and the sharp image regularization parameter λ. x Fuzzy kernel regularization parameter λ h The maximum number of iterations K, the number of channels, etc. The initial learning rate η0 of the multi-scale image generation network is set to 1×10⁻⁶. -3 The learning rate decays to 0.5 times its original value every 500 iterations, the maximum number of iterations K is set to 2000, and the number of channels is set to 16.

[0063] Step 2. Estimate the multi-scale fuzzy kernel

[0064] Solving the fuzzy kernel estimation subproblem with fixed network parameters θ k-1 Resulting clear image Calculate the fuzzy kernel at different scales Omit the superscript 's' and solve the fuzzy kernel 'h' separately at each scale. k The optimization problem can then be expressed as:

[0065]

[0066] In the formula, y is the blurred image, h is the blur kernel, and x is the blurred image. k-1 For a previously generated clear image from the network, λ h Let h be the regularization parameter of the fuzzy kernel. Solve for the fuzzy kernel h in the frequency domain. k The exact solution is obtained to quickly compute the convolution. Since the inherent periodicity of Fourier operations causes oscillations, the partial derivatives of the image are used to model the data fidelity term. Furthermore, a fuzzy kernel centroid correction regularization term is introduced, and equation (3) can be rewritten as:

[0067]

[0068] In the formula, For gradient operators, Let be the partial derivatives of the image along the horizontal and vertical directions, and γ be the regularization parameter, (x0, y0). T For the fuzzy kernel center, The first term is the fuzzy kernel centroid. The third term is the fuzzy kernel centroid correction regularization term, which aims to constrain the kernel centroid to deviate from the center of the kernel during iterative solving, thus correcting the fuzzy kernel centroid. Assume the fuzzy kernel h... k If the size is m×n, then we have in, t m = (1, 2, ..., m) T , t n = (1, 2, ..., n) T Equation (4) is about h k The quadratic function has a closed-form solution. The fuzzy kernel h can be solved directly. k The exact solution, i.e., the objective function in equation (4) with respect to h k Set the derivative of to 0 and expand it into a system of linear equations:

[0069] Avec(h k )=vec(Z) (5)

[0070] In the formula, vec(·) is the matrix vectorization operation, and the expression for Z is:

[0071]

[0072] The coefficient matrix A is

[0073] A=γ[vec(F)vec(U) T +vec(K)vec(V) T ]+I (7)

[0074] Where I is an mn×mn identity matrix, and matrices U, V, F, and K are respectively

[0075]

[0076] in, Indicates Fourier transform, This represents the inverse Fourier transform. This represents the complex conjugate of the Fourier transform. By directly solving the linear equations of equation (5) at various scales, the fuzzy kernel closure solutions corresponding to images at different scales can be obtained.

[0077] Step 3. Estimate the multi-scale sharp image

[0078] Solving the image estimation subproblem, i.e., estimating the current multi-scale blur kernel. From θ k-1 Update θ k Generate multi-scale sharp image estimation The objective function is now:

[0079]

[0080] In the formula, the scale s∈{0,1,2,3}, f s ({z s};θ) represents the sharp image x output by the multi-scale image generation network at scale s. s , {z s} represents a multi-scale random matrix input, θ represents network parameters, and y represents the input. s Represents a blurred image at scale s. λ represents the fuzzy kernel at scale s. x The first term represents the regularization parameter for a sharp image. The second term represents the sharp image {x}. s The TV regular expression term.

[0081] This invention utilizes a self-supervised training method to establish a mapping from random input to sharp images. The input to the multi-scale image generation network is a random matrix at four scales, and the output is a sharp image at four scales. The mapping between input and output is {x}. s}=f({z s};θ), s∈{0,1,2,3}, the network uses the blurred image itself as a supervision signal for self-supervised training, updates the network parameters, and thus estimates the clear image.

[0082] The specific structure of the multi-scale image generation network used in this invention is as follows: Figure 3 As shown, this network has an encoder / decoder structure similar to U-Net. For ease of description, it is divided into four parts: input converter, multi-input encoder, multi-output decoder, and output converter. Multi-scale input {z s The image is processed sequentially through an input converter, a multiple-input encoder, a multiple-output decoder, and an output converter to obtain a multi-scale clear image {x}. s The input transformer consists of convolutional layers, and its function is to transform the input random matrix {z} into a single input matrix. s The features are transformed into semantic features and fed into a multi-input encoder, which uses additional convolutional layers at a coarser scale to ensure that the features obtained from the multi-scale random input have a consistent semantic level. The multi-input encoder contains multiple feature extraction and downsampling operations. The feature extraction (FE) module (such as...) Figure 4 As shown in a), it consists of multiple stacked convolutional layers, where the first convolutional layer performs spatial downsampling with a stride of 2. The multi-scale feature fusion (SFF) module (such as...) Figure 4 (As shown in b) fusion is performed by fusing residual blocks with coarser-scale features, where the residual connections serve to better learn the feature representations. The multi-output decoder includes multiple feature extraction and upsampling operations. The high-low order feature fusion (LFF) module (such as...) Figure 4(As shown in c) This method integrates spatial resolution and high-order semantic features, using cross-layer connections to link high-order features with upsampled low-order features. These cross-layer connections are achieved by cascading the outputs of corresponding layers channel-by-channel. The output converter transforms the features decoded by the multi-output decoder into a sharp image {x}. s Similar to the input transformer, more convolutional layers are used at coarser scales to ensure semantic consistency across different scales. Channel fusion normalization (CFN) modules (such as...) Figure 4 As shown in d), the output features are concatenated with the random input matrix to aid convergence, and the output is normalized using the Sigmoid nonlinear function. The multi-scale image generation network employed in this invention fuses multi-scale information from degraded images through a multi-input encoder and a multi-output decoder, solving the estimation problem of large-size blur kernels with low computational cost.

[0083] For the problem of color image estimation, this invention simultaneously calculates the loss of the RGB channels of the blurred image to update the parameters. The model outputs 3 channels, and the network directly generates color images without the need for a synthesis process, thereby enhancing the color fidelity of the image and reducing the color distortion of the deblurred image.

[0084] Gradient descent is a commonly used first-order optimization method and one of the simplest and most classic methods for solving unconstrained optimization problems. For the unconstrained optimization problem minθL(θ), where L(θ) has continuous first-order partial derivatives, choosing an appropriate initial value θ0, since the negative gradient direction is the direction that causes the function value to decrease the fastest, the value of θ is updated in the negative gradient direction at each step during iteration, thereby minimizing the objective function. With the development of deep learning, more efficient gradient descent methods based on momentum and adaptive gradients have been proposed, such as AdaGrad, RMSProp, and Adam. Among them, the Adam algorithm can adaptively adjust the learning rate, converges faster than traditional gradient descent methods, and is less prone to getting trapped in local optima.

[0085] Step 3.1 Calculate the gradient of the loss function with respect to the network parameters:

[0086] This invention uses a multi-input multi-output multi-scale network to generate clear images. The objective function in equation (9) is the network's loss function:

[0087]

[0088] The loss function in equation (10) is a function of the network parameters θ. The gradient of this loss function with respect to the network parameters θ can be expressed as:

[0089] Step 3.2 Update the network parameters of the multi-scale image generation network:

[0090] This invention uses the Adam gradient descent algorithm to update the network parameters θ, which incorporates momentum υ. k And the second momentum s in the RMSProp algorithm k Parameter updates are performed, adaptively adjusting the learning rate while utilizing momentum to accelerate convergence. First, v0 = s0 = 0 is initialized, given the hyperparameter 0 ≤ β1 < 1, and the momentum v in the k-th iteration is... k Represented as gradient g k-1 Exponentially weighted moving average:

[0091] v k =β1v k-1 +(1-β1)g k-1 (11)

[0092] Given hyperparameters 0 ≤ β² < 1, s k Represented as the gradient squared term g k-1 ⊙g k-1 Exponentially weighted moving average:

[0093] s k =β2s k-1 +(1-β2)g k-1 ⊙g k-1 (12)

[0094] In the formula, ⊙ denotes element-wise multiplication. Since the elements in v0 and s0 are all initialized to zero, the momentum υ is [value missing] in the k-th iteration. k Represented as:

[0095]

[0096] By summing the gradient weights of each previous iteration, we can obtain:

[0097]

[0098] When k is small, the sum of the gradient weights in each iteration will be relatively small. To eliminate this effect, for the k-th iteration, v... k Divide by Making the sum of the gradient weights from each previous iteration equal to 1 is called bias correction. In the Adam algorithm, for the variable υ... k and s k All deviation corrections are made:

[0099]

[0100] Use the bias-corrected variable in the k-th iteration and Update network parameters θ k ,

[0101]

[0102] In the formula, η is the learning rate, and each element of the independent variable in Adam has a different adaptive learning rate; ∈ is a constant used to avoid the case where the denominator is 0 in formula (17).

[0103] Step 3.3 Generate multi-scale sharp images:

[0104] Multi-scale image generation network f({z) with updated network parameters s};θ k Based on the currently estimated network parameters θ k In the k-th iteration, clear images at four scales are generated through forward propagation of the network.

[0105] Step 4. Determine whether the loss function has converged or reached the maximum number of iterations K, and output the estimates of the blur kernel and the clear image.

[0106] Steps 2 and 3 complete one iterative solution to the objective function, that is, firstly using the multi-scale clear images generated by the network in the previous iteration. Calculate the fuzzy kernel Then based on the fuzzy kernel Compared to the clear image generated by the network previously Calculate the loss function with respect to the network parameters θ k-1 The gradient is used to update the network parameters θ through backpropagation. k Based on the currently estimated network parameters θ k Generate clear images Estimation of sharp images Updated to If the loss function converges below a certain threshold or reaches the maximum number of iterations N, then the iteration stops, and the final fuzzy kernel estimate is output. With sharp image estimation Otherwise, repeat steps 2 and 3.

[0107] This invention validates the performance of the disclosed method on the Lai and Kohler datasets. The Lai dataset contains a set of simulated blurred images, consisting of 25 sharp images and 4 blur kernels of different sizes. A total of 100 blurred images are generated through convolution operations between the sharp images and the blur kernels. These images are categorized into five classes: Manmade, Natural, People / Face, Saturated, and Text, with 20 blurred images in each class. The Kohler dataset consists of non-uniformly blurred images captured by a camera, containing 4 sharp images and 12 blur kernels, for a total of 48 blurred images.

[0108] In experiments on the Lai and Kohler datasets, the method of this invention was compared with algorithms such as MIMO-UNet, SelfDeblur, and Fast-SelfDeblur. Results for each algorithm were provided by the authors. MIMO-UNet utilizes a multi-input multi-output U-Net network to model the estimation process of a sharp image, performing multi-scale deblurring within a single U-Net. This is a supervised blind image deblurring method. SelfDeblur, Fast-SelfDeblur, and this invention are all self-supervised blind image deblurring methods. To demonstrate the effectiveness of the multi-scale structure in solving large-scale blur kernels, this invention designed a single-scale version of the separation experiment. In the single-input single-output image generation network, only the random input and image output at the original scale were retained; the multi-scale version is the method of this invention.

[0109] This invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as quantitative evaluation metrics. PSNR is the ratio of the peak power of the signal to the mean square error of the signal and noise. SSIM calculates the similarity between the reconstructed image and the ground truth image from three aspects: the mean, standard deviation, and correlation coefficient of the entire image. Higher PSNR and SSIM values ​​indicate better image reconstruction quality. Table 1 shows the average PSNR and SSIM of each algorithm on the Lai simulated image dataset, and Table 2 lists the comparison of the average PSNR and SSIM of each algorithm on the Kohler dataset. It can be seen that the method of this invention achieves the highest average PSNR and SSIM values ​​on both the Lai simulated image dataset and the Kohler dataset, verifying the performance of the method disclosed in this invention. In the separation experiment, the single-scale version of the method of this invention, compared with Fast-SelfDeblur, verifies the effectiveness of introducing a centroid correction strategy in the fuzzy kernel estimation subproblem for fuzzy kernel estimation; compared with the method of this invention, it significantly verifies the effectiveness of multi-scale structures in blind image deblurring tasks.

[0110] Table 1 Comparison of mean PSNR / SSIM in the Lai dataset

[0111]

[0112] Table 2 Comparison of average PSNR / SSIM in the Kohler dataset.

[0113]

[0114] Figure 5The average PSNR and SSIM of the method of this invention, SelfDeblur, and Fast-SelfDeblur algorithms on the 8th to 12th large-size blur kernels in the Kohler dataset were compared. It can be seen that the method of this invention achieves the highest value on each blur kernel, proving that the multi-scale deblurring network introduced by the method of this invention can effectively solve the large-size blur problem.

[0115] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A self-supervised multi-scale blind image deblurring method based on depth prior, characterized in that, It includes the following 4 steps: Step 1. Initialize multi-scale random inputs and network parameters, and set hyperparameters; Construct a multi-scale image generation network with four scales; initialize the network input and network parameters, with the input being a random matrix {z}. s }, s∈{0, 1, 2, 3}, z 3 Let z be a random matrix uniformly distributed on the interval (0, 1), and the rest are random matrices. s s = 0, 1, 2 is derived from z 3 Perform 2 (3-s) The network parameters are obtained by upsampling from nearest neighbors; the He method is used to initialize the network parameters, with the initial network parameters being θ0; the input is a random matrix {z}. s The initial estimates of sharp images at four scales are calculated through forward propagation of the multi-scale image generation network. Step 2. Estimate the multi-scale fuzzy kernel; Fixed network parameters θ k-1 Resulting clear image Omit the superscript 's', solve the exact solution of the following linear equation system, and estimate the fuzzy kernel 'h' separately at each scale. k : With(h k )=vec(Z) In the formula, the coefficient matrix A is A=γ[vec(F)vec(U) T +thing(K)thing(V) T ]+I The expression for Z is Matrices U, V, F, and K are respectively Where vec(·) is the matrix vectorization operation. Indicates Fourier transform, This represents the inverse Fourier transform. This represents the complex conjugate of the Fourier transform; Step 3. Estimate the sharp image at multiple scales; Estimation of the current multi-scale fuzzy kernel From θ k-1 Update θ k Generate multi-scale sharp image estimation It includes the following 3 steps: Step 3.1 Calculate the gradient of the loss function with respect to the network parameters: The loss function of the network in this invention is: Calculate the gradient of the loss function L(θ) with respect to the parameter θ. Step 3.2 Update the network parameters of the multi-scale image generation network: Update network parameters θ using the Adam gradient descent algorithm k : In the formula, η is the learning rate, ∈ is a constant, and the momentum after bias correction is... and second momentum They are respectively Where ⊙ represents element-wise multiplication, 0≤β1<1, 0≤β2<1; Step 3.3 Generate multi-scale sharp images: Clear images at four scales after updating network parameters via network forward propagation. Step 4. Determine whether the loss function has converged or reached the maximum number of iterations K, and output the estimates of the blur kernel and the clear image; Through steps 2 and 3, the estimate of the multi-scale fuzzy kernel is obtained. And the estimation of multi-scale sharp images Updated to Complete one iteration of the objective function; if the loss function converges to below a certain threshold or reaches the maximum number of iterations K, stop the iteration and output the final blur kernel and sharp image estimate; otherwise, repeat steps 2 and 3.

2. The self-supervised multi-scale blind image deblurring method based on depth prior as described in claim 1, characterized in that, The initial learning rate η0 of the multi-scale image generation network during training is 1×10. -3 .

3. The self-supervised multi-scale blind image deblurring method based on depth prior as described in claim 1, characterized in that, The learning rate decay factor during training is 0.

5.

4. The self-supervised multi-scale blind image deblurring method based on depth prior as described in claim 1, characterized in that, The maximum number of iterations K during training is 2000.

5. The self-supervised multi-scale blind image deblurring method based on depth prior as described in claim 1, characterized in that, During training, the number of input data channels in the image generation network is fixed at 16.

Images