A self-supervised deep image blind deblurring method, system and storage medium

Abstract

The application discloses a kind of self-supervision depth image blind deblurring method, system and storage medium, the method includes obtaining actual blur image;Noise in compliance with uniform distribution and the size and actual blur image consistent is input to self-supervision depth blind deblurring model, obtain clear image and blur kernel;Wherein, the self-supervision depth blind deblurring model is after training optimization by integrating generalized Gamma discriminative image prior into the loss function of SelfDeblur.This application only utilizes blur image to realize end-to-end self-supervised learning blind deblurring, without using blur image and clear image pair training network, high effectiveness;By integrating generalized Gamma discriminative image prior into the loss function of SelfDeblur, the optimization process of self-supervision depth blind deblurring model can be effectively guided to converge to clear image, and then the blur kernel and clear image can be better estimated.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a self-supervised blind deblurring method, system, and storage medium for depth images. Background Technology

[0002] Images, as carriers of information, are closely linked to human life and have become an important means for people to acquire and transmit information in today's society. However, during the acquisition, processing, transmission, and recording of images, various factors, such as atmospheric turbulence, defocusing of the imaging device itself, camera shake, relative motion between the image and the target object during imaging, as well as unavoidable factors like lighting conditions and noise pollution, can lead to image quality degradation and blurring. To obtain the original clear image, image restoration is necessary. Estimating a clear image when the blur kernel is unknown is a challenging signal processing problem in image imaging.

[0003] The blurring process is typically assumed to be uniform and spatially invariant. It is modeled using convolution operations, i.e., y = k*x + n, where y is the blurred image to be processed, x is the original sharp image, k is the blur kernel representing the diffusion process of a pixel along a certain motion trajectory, and n is additive noise (usually assumed to be zero-mean, Gaussian white noise with known variance). Analysis of the image degradation model shows that blind image restoration estimates the original sharp image x and the blur kernel k from the known degraded image y.

[0004] Blind image deblurring is mathematically considered a severely ill-conditioned problem, characterized by the fact that even small changes in the data during the solution process can lead to significant variations in the final solution, resulting in an extremely large solution space, potentially even infinitely many solutions. To transform this ill-conditioned problem into a well-conditioned one, it is necessary to incorporate prior knowledge about the sharp image and the blur kernel into the regularization constraint of the entire process. This is achieved by utilizing the maximum a posteriori probability principle of blind image restoration, incorporating prior knowledge into the corresponding cost function. Then, the prior constraints in both the blur kernel estimation and sharp image restoration processes are investigated separately. Specifically, by uncovering prior knowledge about the image and blur kernel, appropriate regularization terms are constructed in the cost function to constrain the image restoration process, allowing for the optimal estimation of the original sharp image under certain criteria. There are generally two approaches to solving the blind image restoration problem. The first is to directly apply a deblurring method to simultaneously and alternately estimate the original sharp image and the blur kernel. The second is to first estimate the blur kernel of the blurred image, and then use a non-blind deblurring method to estimate the original sharp image given the blur kernel.

[0005] To solve the blind deblurring problem, it is usually transformed into a maximum a posteriori probability problem: ; First item This is called the data fidelity item, the second item. The first term represents the prior knowledge of the clear image, and the third term represents the prior knowledge of the blur kernel. Traditional methods generally introduce the L0 norm of the image gradient as prior knowledge for blind deblurring based on the maximum a posteriori (MAP) framework. Further, a series of L0+X works have emerged. For example, Pan et al. (J. Pan, D. Sun, H. Pfister, MHYang. Blind image deblurring using dark channel prior[C]. IEEE Conferenceon Computer Vision and Pattern Recognition, 2016: 1628-1636.) combined the sparsity of the dark channel of the image with the L0 norm of the image gradient to achieve blind deblurring of natural images. Experiments show that this method has high robustness in the deblurring problem of natural images and can be extended to non-uniform deblurring. However, Yan et al. found that when the input image contains few or no dark pixels, the dark channel prior may not be very effective in estimating the intermediate image. Therefore, Yan et al. (Yan Y, Ren W, Guo Y, et al. Image Deblurring via ExtremeChannels Prior[C] / / 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.) introduced a method combining bright and dark channels based on the L0 norm of image gradients. This method considers both bright and dark channel information, eliminating the need for complex processing techniques and edge selection steps, but with higher computational complexity. With the popularization of computer applications, deep learning methods have become popular due to their simplicity and speed. Li et al. (Li L, Pan J, Lai WS, et al. Learning a Discriminative Prior for Blind Image Deblurring[C] / / 2018 IEEE / CVF Conferenceon Computer Vision and Pattern Recognition. IEEE, 2018.) trained a binary classification convolutional neural network model on a dataset, used the learned image information as an image prior, and then combined it with the image gradient L0 sparse prior to achieve image deblurring. This method can be applied to deblurring problems in various scenarios.Kupyn (O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, J. Matas. DeblurGAN: blindmotion deblurring using conditional adversarial networks[C]. IEEE Conferenceon Computer Vision and Pattern Recognition, 2018:8183-8192.) et al. proposed the conditional generative adversarial network DeblurGAN, which treats the image deblurring task as a special image-to-image translation processing task and uses content loss and adversarial loss for end-to-end training and learning. Later, Kupyn et al. (O. Kupyn, T. Martyniuk, J. Wu, Z. Wang. DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better[J]. IEEE International Conference on Computer Vision, 2019: 8877-8886.) proposed DeblurGAN-v2 based on Deblur-GAN. By introducing a feature pyramid network and a shallow deep network, DeblurGAN-v2 can better complete the task of deblurring images by removing motion blur.

[0006] Both traditional methods and deep convolutional neural network methods have obvious drawbacks. On the one hand, traditional optimization methods rely excessively on manually designed priors, which are significantly insufficient in representing sharp images and blur kernels. Furthermore, the non-convexity of MAP (maximum a posterior) based models increases the difficulty of optimization. On the other hand, deep deblurring networks require a large amount of training data to establish an end-to-end network mapping from blurred image y to sharp image x, but they often struggle to handle more complex blur types and perform poorly in cross-domain tasks. Due to the severe pathological nature caused by large and complex blur kernels, existing deep learning-based methods still cannot outperform traditional optimization-based methods in blind deconvolution. In 2019, Ren et al. (Dongwei Ren, Qilong Wang, Qinghua Hu, Kai Zhang, and Wangmeng Zuo. Neural Blind Deconvolution Using Deep Priors[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2020: 3338-3347.) first proposed the self-supervised learning method SelfDeblur within the maximum a posteriori framework. It utilizes a deep convolutional neural network to generate intermediate sharp images and uses a simple fully connected network to estimate the blur kernel.

[0007] Although SelfDeblur reduces the requirements for training datasets due to the advantages of self-supervised learning, experimental results on synthetic datasets such as Levin (A. Levin, Y. Weiss, F. Durand, WT Freeman. Understanding blind deconvolution algorithms[J]. IEEE Trans. Pattern Analysis and Machine Intelligence, 2011, 33(12): 2354-2367.), Lai (WS Lai, JB Huang, Z. Hu, N. Ahuja, MH Yang. A comparative study for single image blind deblurring[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1701-1709.), and a large number of real blurred image datasets reveal the following problems with SelfDeblur: some reconstructed images still retain blur and exhibit abnormal artifacts, affecting visual perception; the blur kernel of blurred images in low-light scenes cannot be recovered. To further evaluate the deblurring effect of SelfDeblur, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) values ​​were tested again on 100 restored images from the Lai dataset. The average PSNR value was 20.163 dB, and the average SSIM value was 0.716. This indicates that SelfDeblur's overall deblurring effect on the Lai dataset is not ideal. SelfDeblur reveals that it only uses the Mean Squared Error (MSE) loss function as the optimization function for the entire model. While the classic MSE loss function does reduce the difference between the generated and original images to some extent, SelfDeblur clearly ignores the mining of prior knowledge for blind deblurring and the nature of deblurring itself.

[0008] The research on the above works reveals that the effectiveness of current estimation methods for blur kernels and sharp images still needs to be improved. Summary of the Invention

[0009] The purpose of this invention is to provide a self-supervised blind deblurring method, system, and storage medium for depth images, so as to improve the effectiveness of blur kernel and sharp image estimation.

[0010] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0011] In a first aspect, the present invention discloses a self-supervised blind deblurring method for depth images, comprising:

[0012] Obtain the actual blurred image;

[0013] Noise that follows a uniform distribution and has a size consistent with the actual blurred image is input into a self-supervised deep blind deblurring model to obtain a clear image and a blur kernel; wherein, the self-supervised deep blind deblurring model is obtained by training and optimizing the loss function of the SelfDeblur model after integrating the generalized Gamma discriminative image prior into it;

[0014] The method for obtaining the self-supervised deep blind deblurring model includes:

[0015] Initialize the number of iterations for the self-supervised deep blind deblurring model;

[0016] Noise that follows a uniform distribution and has a size consistent with the actual blurred image is input into the deep convolutional neural network and fully connected network of the self-supervised deep blind deblurring model to generate an initial sharp image and an initial blur kernel;

[0017] The loss function is calculated using an initial sharp image and an initial blur kernel.

[0018] Calculate the gradients of the parameters of the deep convolutional neural network and the fully connected network based on the loss function;

[0019] The ADAM algorithm is used to update the parameters of the deep convolutional neural network and the fully connected network based on the gradients of the calculated parameters of the deep convolutional neural network and the fully connected network.

[0020] Repeatedly iterate and update the parameters of the deep convolutional neural network and the fully connected network until the maximum number of iterations is reached, to obtain a self-supervised deep blind deblurring model, and output the blur kernel and the deblurred image.

[0021] Furthermore, noise with a uniform distribution and a size consistent with the actual blurred image is input into the deep convolutional neural network and fully connected network of the self-supervised deep blind deblurring model. The expression for generating the initial sharp image and the initial blur kernel is as follows:

[0022] ;

[0023] ;

[0024] in, Indicates a clear image. Indicates the fuzzy kernel. Indicates the number of iterations. , Let represent the uniformly distributed random noise inputs to the deep convolutional neural network and the fully connected network, respectively. , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively.

[0025] Furthermore, the expression for calculating the loss function using the initial sharp image and the initial blur kernel is as follows:

[0026] ;

[0027] in, , These represent the parameters of a deep convolutional neural network and the parameters of a fully connected network, respectively. , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively. This represents the discriminative image prior of generalized Gamma; Represents the regularization coefficient; y represents the actual blurred image; This indicates a convolution operation.

[0028] Furthermore, the formulas for calculating the gradients of the parameters in a deep convolutional neural network and the fully connected network are as follows:

[0029] ;

[0030] ;

[0031] In the formula, , These represent the gradients of a deep convolutional neural network and a fully connected network, respectively.

[0032] Furthermore, the formulas for calculating the updated parameters of the deep convolutional neural network and the fully connected network are as follows:

[0033] ;

[0034] ;

[0035] in, , The initial learning rate was set to 0.01, and the total number of iterations was set to... T =5000, when the number of iterations reaches...T When the learning rate is 2000, 3000, or 4000, it decays to half of its original value.

[0036] Furthermore, the expression for the generalized Gamma discriminative image prior is:

[0037] ;

[0038] in, Indicates a clear image. This represents the image gradient. It is the image domain One pixel in Describing the L2 norm, , is the scale parameter; , is a power parameter; , where is the shape parameter.

[0039] Furthermore, the expression for the integration of the generalized Gamma discriminative image prior and the loss function of the SelfDeblur model is as follows:

[0040] ;

[0041] in, , These represent deep convolutional neural networks and fully connected networks, respectively. , These represent random noise that follows a uniform distribution; , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively. y represents the generalized Gamma discriminative image prior; y represents the actual blurred image.

[0042] Secondly, a self-supervised depth image blind deblurring system, characterized in that it includes:

[0043] Memory, used to store instructions;

[0044] A processor for executing the storage instructions, causing the device to perform the method as described in any of the first aspects.

[0045] Thirdly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements any of the methods described in the first aspect.

[0046] The beneficial effects of this invention are as follows: By inputting noise of the same size as the actual blurred image into the self-supervised deep blind deblurring model, a clear image and a blur kernel can be directly obtained. End-to-end self-supervised learning blind deblurring is achieved using only the blurred image, without the need to train the network with blurred and clear images, resulting in high effectiveness. By incorporating the generalized Gamma discriminative image prior into the loss function of SelfDeblur, the optimization process of the self-supervised deep blind deblurring model can be effectively guided to converge toward the clear image, thereby enabling better estimation of the blur kernel and the clear image. Attached Figure Description

[0047] Figure 1 Here is a flowchart of the deblurring method of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the self-supervised deep blind deblurring model in the deblurring method of the present invention;

[0049] Figure 3 This is a comparison diagram of the deblurring method of the present invention with other methods. Detailed Implementation

[0050] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments. Example 1

[0051] This invention introduces a generalized Gamma discriminative image prior as a regularization term into the loss function of SelfDeblur, which can effectively guide the optimization process of the self-supervised deep blind deblurring model to converge toward the clear image, thereby enabling better estimation of the blur kernel and the clear image.

[0052] Research has found that traditional MAP-based image deblurring and de-saturation models can be represented as an optimization problem. ; First item This is called the data fidelity item, the second item. The third term represents prior knowledge about the image. Represents the prior knowledge of the fuzzy kernel. and The regularization parameter controls the weight of restoration error and prior constraints in the objective function. However, the prior information in traditional deblurring models based on image priors is generally manually designed, and the non-convexity of the MAP model increases the difficulty of optimizing the blur kernel, limiting its ability to capture blur kernel priors. Furthermore, it relies on various optimizations during deconvolution, resulting in limited processing capability on images with high levels of blur. Therefore, traditional deblurring models based on image priors require further optimization. Besides manual design, image priors can also be obtained through deep learning networks. The Deep Image Prior (DIP) proposed by Ulyanov et al. posits that the neural network itself inherently possesses implicit prior information, requiring neither a training set nor an undisturbed original image as a ground truth label. Only a single degraded image is needed; image restoration can be performed by capturing the image prior using a convolutional network.

[0053] Inspired by DIP, SelfDeblur reparameterizes the sharp image x and the blur kernel k using a neural network. Instead of directly optimizing x, it treats x as a neural network... The random output is used to optimize the network parameters. .

[0054] Specifically, define a clear image ,in It is a random vector that follows a uniform distribution, i.e. Similarly, define a fuzzy kernel. . It is an autoencoder-decoder network with skip connections, and the output layer uses the sigmoid activation function. It is a fully connected network with a SoftMax nonlinear output. Random vector. and As network input, a corresponding clear image is obtained. and fuzzy kernel .

[0055] That is, SelfDeblur utilizes and The depth priors of the sharp latent image x and the blurred kernel k are captured separately to establish a neural blind deconvolution model. Therefore, a neural blind deconvolution model is then used. and Replace x and k, and remove the regularization term. and The neural blind deconvolution optimization problem can be expressed as:

[0056] ;

[0057] in, , These represent deep convolutional neural networks and fully connected networks, respectively. , These represent random noise that follows a uniform distribution; , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively. For image regularization constraints, It is the regularization coefficient.

[0058] Since generalized Gamma discriminative image priors are helpful for estimating blur kernels and restoring sharp images, this invention attempts to incorporate generalized Gamma discriminative image priors into the loss function of SelfDeblur, thereby achieving an effective combination of self-supervised learning and traditional methods.

[0059] The expression for the generalized Gamma discriminative image prior is:

[0060] ;

[0061] in, Indicates a clear image. This represents the image gradient. It is the image domain One pixel in Describing the L2 norm, , is the scale parameter; , is a power parameter; , where is the shape parameter. In this invention Generally, 0.1 is used. Generally, it is taken as 1. Generally, 2.5 is used. .

[0062] The specific approach is as follows: Within the SelfDeblur framework, the generalized Gamma discriminative image prior is integrated into the SelfDeblur loss function, resulting in the training optimization formula for the self-supervised deep blind deblurring model of this invention, as shown below:

[0063] ;

[0064] in, This represents the discriminative image prior of the generalized Gamma.

[0065] In summary, the steps of the image blind deblurring method of this application can be summarized as follows: obtaining the actual blurred image; inputting noise that follows a uniform distribution and has the same size as the actual blurred image into a self-supervised deep blind deblurring model to obtain a clear image and a blur kernel; wherein, the self-supervised deep blind deblurring model is obtained by training and optimizing the loss function of SelfDeblur after integrating the generalized Gamma discriminative image prior into it.

[0066] The following is a detailed description of this application, such as... Figure 1 As shown, the specific steps of this application include:

[0067] 1) Input a blurred image y with a blur kernel k, and the clear image to be solved is x.

[0068] 2) The inputs to deep convolutional neural networks and fully connected networks are respectively , , are all random vectors uniformly distributed on the interval (0, 1), i.e. Its size is consistent with the blurred image; a deep convolutional neural network is used to generate a sharp image x, using This indicates that a fully connected network is used to generate the fuzzy kernel k, using... express.

[0069] 3) Solve for the fuzz kernel k and the clear image x by using the method of jointly solving the fuzz kernel and the clear image.

[0070] ① Initialize the number of iterations The maximum number of iterations is T .

[0071] ② Through Solving for the fuzzy kernel, by Solve for a clear image; where , Let represent random noise that follows a uniform distribution. , Let x represent the sharp image generated by the deep convolutional neural network and the corresponding blur kernel generated by the fully connected network, respectively. A convolution operation is performed between the sharp image x and the blur kernel k to obtain the synthesized blur image. .

[0072] ③ Calculate using the loss function , The difference between the synthesized blurred image and the actual blurred image y is determined by the following loss function:

[0073] ;

[0074] in, The parameters represent those of a deep convolutional neural network. Parameters representing a fully connected network (FCN) This represents the discriminative image prior of the generalized Gamma.

[0075] ④ Update network parameters using conventional gradient descent and backpropagation.

[0076] First, calculate the gradient of the loss function with respect to the network parameters:

[0077] , ;

[0078] In the formula, It is the gradient of a fully connected network. It is the gradient of a deep convolutional neural network;

[0079] Update using ADAM algorithm again , Network parameters:

[0080] , ;

[0081] 4) Determine if the iteration number t is greater than the maximum iteration number. If the maximum number of iterations is reached at this point, stop iterating and output the final blur kernel and sharp image estimate; otherwise, let... Then repeat steps ②, ③, and ④ in step 3).

[0082] The above steps can be summarized as follows: the method for obtaining the self-supervised deep blind deblurring model in this application includes: initializing the iteration number of the self-supervised deep blind deblurring model; inputting noise with a uniform distribution and a size consistent with the actual blurred image into the deep convolutional neural network and fully connected network of the self-supervised deep blind deblurring model to generate an initial sharp image and an initial blur kernel; calculating the loss function using the initial sharp image and the initial blur kernel; calculating the gradients of the parameters of the deep convolutional neural network and the fully connected network based on the loss function; using the ADAM algorithm to update the parameters of the deep convolutional neural network and the fully connected network based on the calculated gradients of the parameters of the deep convolutional neural network and the fully connected network; repeatedly iterating and updating the parameters of the deep convolutional neural network and the fully connected network until the iteration number is maximized to obtain the self-supervised deep blind deblurring model, and simultaneously outputting the blur kernel and the deblurred image.

[0083] This invention employs a deep convolutional neural network to model a clear image x, specifically using an asymmetric autoencoder with skipped connections as... .like Figure 2As shown, the deep convolutional neural network framework consists of a 5-layer encoder-decoder, with skip connections used in between to compensate for information loss during downsampling. Finally, a clear image is generated using the convolutional output layer. The convolutional output layer uses the sigmoid activation function to limit the generated vector's value range to 0-1. Table 1 shows the deep convolutional neural network configuration for the self-supervised deep blind deblurring model.

[0084] Table 1

[0085]

[0086] in, , , Representing encoders decoder and skip connection layer The number of convolutional kernels is n, the kernel size is k×k, and the padding size for the convolution operation is p×p. The kernel size in the output layer is set to 1×1. The activation function in the DIP framework is the LeakyReLU activation function with a slope of 0.2. Downsampling is implemented using stride=2, and upsampling is implemented using bilinear.

[0087] This invention employs a fully connected network to model the blur kernel k. On one hand, deep convolutional neural networks are designed to capture statistical information from natural images, but they have limitations in prior modeling of blur kernels. On the other hand, the blur kernel k typically contains far less information than the potential clean image x, and therefore can be generated well by simpler deep neural networks. Therefore, this invention uses a fully connected network as... The fully connected network employs one-dimensional noise with 200 dimensions. As input, and with a hidden layer of 1000 nodes and The output layer has 10 nodes. To ensure that the constraints of nonnegativity and equality are always satisfied, SoftMax nonlinearity is applied. The output layer of the network. Finally, the network's... The one-dimensional output is reshaped into a two-dimensional K×K fuzzy kernel.

[0088] The deep convolutional neural network and fully connected network of this invention are respectively represented by four-dimensional vectors. and one-dimensional vector As network input, and All are from a uniform distribution The parameters are obtained through random sampling. During training, the deep convolutional neural network and the fully connected network are trained in a joint optimization manner; that is, the parameters of the deep convolutional neural network and the fully connected network are updated simultaneously in the same optimization process. The optimization algorithm uses parameters that are respectively... , The ADAM optimization algorithm. Setting the total number of iterations. T =5000, initial learning rate set to 0.01, when the number of iterations reaches... T When the learning rate is 2000, 3000, or 4000, it decays to half of its original value.

[0089] The deblurring effect of the present invention will be further described below in conjunction with experimental conditions.

[0090] 1. Experimental conditions

[0091] The hardware environment of this invention is a server with an Intel Core i7-8700 CPU and an NVIDIA GeForce GTX 1080Ti GPU. The simulation software used is PyCharm (r2018), and the software environment is the PyTorch deep learning framework.

[0092] 2. Data Source

[0093] The blurred images used in the experiment were sourced from the blurred natural image database in the article "A Comparative Study for Single Image Blind Deblurring" published by WS Lai, JB Huang, Z. Hu, N. Ahuja, MH Yang, et al. The database consisted of 25 clean images and 4 large blur kernels. The blurred images were divided into five categories: artificial, natural, people, saturated, and text, with 20 blurred images in each category. For example... Figure 3 (a) shows a blurry image of “People”.

[0094] 3. Experiment Content

[0095] This invention is compared with two existing natural image deblurring methods, which are:

[0096] A blind deblurring method based on dark channel prior, disclosed by Pan et al. (Jinshan Pan, Deqing Sun, MingHsuan Yang, Hanspeter Pfister. Blind Image Deblurring Using DarkChannel Prior. In CVPR, 2016.), is denoted as Method 1;

[0097] Ren et al. published a blind deblurring method based on neural blind deconvolution priors, namely SelfDeblur (Dongwei Ren, Qilong Wang, Qinghua Hu, Kai Zhang, and Wangmeng Zuo. NeuralBlind Deconvolution Using Deep Priors. In CVPR, 2020.), denoted as Method 2.

[0098] 4. Simulation Result Analysis

[0099] In the simulation experiments, the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) were used to evaluate the experimental results. SSIM and PSNR were defined as follows:

[0100]

[0101] in: , and , Let x and y represent the mean and variance, respectively. Represents the covariance of x and y. and It is a constant. The structural similarity value is between 0 and 1, and the closer the value is to 1, the more similar the two images are.

[0102]

[0103] Where: x represents the original, clear image, This represents the recovered deblurred image. N This represents the number of pixels in the image. A higher peak signal-to-noise ratio indicates better deblurring performance.

[0104] (1) Quantitative assessment

[0105] To better evaluate the deblurring performance of these three methods, we conducted experiments on the same set of standard blurred images on the Lai dataset, comparing the SSIM and PSNR values ​​of Method 1, Method 2, and the method of this invention. The results are shown in Tables 2 and 3 below.

[0106] Table 2

[0107]

[0108] Table 3

[0109]

[0110] Based on the above data, it can be seen that the SSIM and PSNR values ​​of the deblurring results obtained by the method of the present invention are slightly higher than those of the deblurring results obtained by other methods, that is, the present invention has a better deblurring effect than the prior art.

[0111] (2) Qualitative assessment

[0112] like Figure 3 The diagram shows the deblurring effect of the method of the present invention compared to methods 1 and 2. In this diagram, (a) is the input blurred image; (b) is the effect of the blind deblurring method based on dark channel prior disclosed in method 1; (c) is the effect of the blind deblurring method of SelfDeblur disclosed in method 2; and (d) is the effect of the deblurring method obtained by the method of the present invention. Figure 3 It can be seen that the deblurring result (d) of the method of the present invention is clearer and the structure is smoother, and has a better deblurring effect than the prior art.

[0113] In summary, by incorporating the generalized Gamma discriminative image prior into the loss function of SelfDeblur, this invention can effectively guide the optimization process of the self-supervised deep blind deblurring model to converge toward the clear image, thereby enabling better estimation of the blur kernel and the clear image.

[0114] Example 2

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A self-supervised blind deblurring method for depth images, characterized in that, include: Obtain the actual blurred image; Noise that follows a uniform distribution and has a size consistent with the actual blurred image is input into a self-supervised deep blind deblurring model to obtain a clear image and a blur kernel; wherein, the self-supervised deep blind deblurring model is obtained by training and optimizing the loss function of the SelfDeblur model after integrating the generalized Gamma discriminative image prior into it; The method for obtaining the self-supervised deep blind deblurring model includes: Initialize the number of iterations for the self-supervised deep blind deblurring model; Noise that follows a uniform distribution and has a size consistent with the actual blurred image is input into the deep convolutional neural network and fully connected network of the self-supervised deep blind deblurring model to generate an initial sharp image and an initial blur kernel; The loss function is calculated using an initial sharp image and an initial blur kernel. Calculate the gradients of the parameters of the deep convolutional neural network and the fully connected network based on the loss function; The ADAM algorithm is used to update the parameters of the deep convolutional neural network and the fully connected network based on the gradients of the calculated parameters of the deep convolutional neural network and the fully connected network. Repeatedly iterate and update the parameters of the deep convolutional neural network and the fully connected network until the maximum number of iterations is reached, to obtain a self-supervised deep blind deblurring model, and output the blur kernel and the deblurred image.

2. The self-supervised depth image blind deblurring method according to claim 1, characterized in that, The expression for generating the initial sharp image and the initial blur kernel by inputting noise that follows a uniform distribution and has the same size as the actual blurred image into the deep convolutional neural network and fully connected network of the self-supervised deep blind deblurring model is as follows: ； ； in, Indicates a clear image. Indicates the fuzzy kernel. Indicates the number of iterations. , Let represent the uniformly distributed random noise inputs to the deep convolutional neural network and the fully connected network, respectively. , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively.

3. The self-supervised depth image blind deblurring method according to claim 1, characterized in that, The expression for calculating the loss function using the initial sharp image and the initial blur kernel is as follows: ； in, , These represent the parameters of a deep convolutional neural network and the parameters of a fully connected network, respectively. , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively. This represents the discriminative image prior of generalized Gamma; Represents the regularization coefficient; This represents the actual blurred image; This indicates a convolution operation.

4. The self-supervised depth image blind deblurring method according to claim 3, characterized in that, The formulas for calculating the gradients of parameters in a deep convolutional neural network and fully connected networks are as follows: ； ； In the formula, , These represent the gradients of a deep convolutional neural network and a fully connected network, respectively.

5. The self-supervised depth image blind deblurring method according to claim 4, characterized in that, The formulas for calculating the parameters of the deep convolutional neural network and the fully connected network are as follows: ； ； in, , The initial learning rate was set to 0.01, and the total number of iterations was set to... T =5000, when the number of iterations reaches... T When the learning rate is 2000, 3000, or 4000, it decays to half of its original value.

6. The self-supervised depth image blind deblurring method according to claim 1, characterized in that, The expression for the generalized Gamma discriminative image prior is: ； in, Indicates a clear image. This represents the image gradient. It is the image domain One pixel in Describing the L2 norm, , is the scale parameter; , is a power parameter; , where is the shape parameter.

7. The self-supervised depth image blind deblurring method according to claim 6, characterized in that, The expression for the integration of the generalized Gamma discriminative image prior and the loss function of the SelfDeblur model is as follows: ； in, , These represent deep convolutional neural networks and fully connected networks, respectively. , These represent random noise that follows a uniform distribution; , These represent the blur kernels of the sharp image generated by the deep convolutional neural network and the corresponding sharp image generated by the fully connected network, respectively. y represents the generalized Gamma discriminative image prior; y represents the actual blurred image. This represents the regularization coefficient.

8. A self-supervised blind deblurring system for depth images, characterized in that, include: Memory, used to store instructions; A processor for executing the storage instructions, causing the processor to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images