A self-supervised image blind deblurring method based on hierarchical bayesian representation

Abstract

The present application relates to the technical field of image processing, computer vision and image blind deblurring, and particularly relates to a self-supervised image blind deblurring method based on hierarchical Bayesian representation, comprising: obtaining a clear image and processing it, synthesizing a blurred image, and establishing a blurred image dataset; constructing a hierarchical Bayesian prior model and an objective function based on a motion degradation model combined with the blurred image dataset, and defining random variables; minimizing the objective function to solve the posterior distribution of the random variables in the hierarchical Bayesian prior model; designing a deep neural network framework, establishing a loss function combined with the posterior distribution of the random variables and training to obtain an optimal model; outputting an image restoration result of the blurred image dataset through the optimal model; and realizing sparse adaptive optimization based on the hierarchical Bayesian prior model, solving the problems of insufficient generalization and poor robustness of the blind deblurring algorithm based on VB, and being applicable to high-precision demand scenarios such as image enhancement.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing, computer vision and image blind deblurring, and in particular to a self-supervised image blind deblurring method based on hierarchical Bayesian representation. Background Art

[0002] Image deblurring has long been a key research topic in the fields of computer vision and image processing. During image acquisition, processing, transmission, and storage, many factors can negatively impact image quality, such as device vibration during use, relative displacement between the device and the target during imaging, and noise contamination. These factors can cause blurring, distortion, and additive noise in the image, significantly reducing image clarity. This phenomenon is known as image degradation. To address this issue, image deblurring technology has emerged, aiming to restore clarity and usability to degraded images.

[0003] To address image degradation, existing research is typically based on the maximum a posteriori (MAP) framework, which alternates between estimating a blur kernel and a potential sharp image to gradually approximate the true image. However, studies have shown that under certain conditions, this method tends to interpret the degraded image as generated by a delta function kernel, making effective image restoration difficult and inherently ill-conditioned. Variational Bayes (VB) methods, on the other hand, have demonstrated significant theoretical advantages in blind deblurring applications. By estimating the posterior distribution of the image, VB methods are theoretically more robust than traditional MAP methods, reducing the risk of convergence to local minima.

[0004] However, when using the variational Bayesian method to deblur images, a class of sparse image priors suitable for modeling natural images, such as Gaussian mixture models and super-Gaussian distributions, are introduced. These priors can capture the sparse characteristics of images and are converted into a form suitable for inference through variational representation, thereby simplifying the calculation. However, the non-natural sparse image priors used are usually based on empirical design rather than adaptive learning, which is not conducive to the generalization and robustness of the algorithm. In addition, the traditional VB method has a complex matrix inversion problem, which makes the solution difficult and affects the computational efficiency. In addition, despite the significant theoretical advantages of VB, in practical applications, VB-based blind deblurring algorithms face huge challenges in many scenarios. Its theoretical advantages have not been fully utilized, and the actual restoration accuracy is often lower than that of optimization methods based on the maximum a posteriori method framework. Summary of the Invention

[0005] In order to solve the technical problems that the existing technology for VB image processing applications cannot adaptively learn, is not conducive to generalization and robustness, is computationally complex and difficult to solve, and has lower restoration accuracy than the traditional maximum a posteriori method, the purpose of the present invention is to provide a self-supervised image blind deblurring method based on hierarchical Bayesian representation. The technical solution adopted is as follows:

[0006] Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset;

[0007] Based on the motion degradation model and blurred image dataset, a hierarchical Bayesian prior model and objective function are constructed, and random variables are defined;

[0008] Minimize the objective function and solve the posterior distribution of random variables in the hierarchical Bayesian prior model;

[0009] Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model;

[0010] The image restoration results of the blurred image dataset are output by the optimal model.

[0011] Preferably, a clear image is acquired and processed to synthesize a blurred image and establish a blurred image dataset, specifically:

[0012] Get a clear image, use the camera trajectory to generate a blur trajectory to get a blur kernel, convolve the clear image with the blur kernel to synthesize a blurred image, which is recorded as ,in, 、 Represent the width and height of the blurred image respectively, 3 represents the number of channels of the blurred image, and a blurred image dataset is established.

[0013] Preferably, a hierarchical Bayesian prior model and objective function are constructed based on the motion degradation model combined with the blurred image dataset, and random variables are defined, including:

[0014] A hierarchical Bayesian prior model is established based on the motion degradation model and the blurred image dataset. The model includes at least three layers, each of which defines the random variables in the hierarchical Bayesian prior model. In the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function. In the second and third layers, the prior distribution is defined as a gamma probability density function.

[0015] Construct the objective function through random variables.

[0016] Preferably, the random variables in the hierarchical Bayesian prior model are defined separately, and the corresponding calculation formula is:

[0017]

[0018]

[0019]

[0020]

[0021] in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

[0022] Preferably, the objective function is constructed by random variables, and the corresponding calculation formula is:

[0023]

[0024]

[0025] in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

[0026] Preferably, minimizing the objective function and solving the posterior distribution of the random variable in the hierarchical Bayesian prior model includes:

[0027] Based on the first layer of latent clear image The posterior probability Obey multivariate Gaussian distribution , the corresponding calculation formula is:

[0028]

[0029]

[0030] The random variables in the second and third layers all obey the gamma distribution, and the corresponding calculation formulas are:

[0031]

[0032]

[0033]

[0034]

[0035] in, represents the posterior distribution of the potential sharp image; 、 represents the intermediate parameter representation in the first layer; express expectations; Indicates a blurred image; Indicates a clear image; 、 All represent random variables; 、 、 All represent hyperparameters; represents the blur kernel; 、 They represent the shape parameters of the corresponding random variables respectively; 、 They represent the rate parameters of the corresponding random variables respectively; represents a random variable The posterior distribution of represents a random variable The posterior distribution of 、 Respectively represent the gradient operation of the clear image in the horizontal and vertical directions; represents a diagonal matrix.

[0036] Preferably, a deep neural network framework is designed, a loss function is established in combination with the posterior distribution of random variables, and the deep neural network framework is trained to obtain an optimal model, including:

[0037] Design deep neural network frameworks, including encoder-decoder structure networks and fully connected networks;

[0038] Fixed noise is input to the encoder-decoder structure network and the fully connected network for processing, and the potential clear image and blur kernel are output respectively;

[0039] The loss function is constructed by combining the latent clear image and the blur kernel, and is optimized based on the posterior distribution of random variables. The deep neural network framework is trained to obtain the optimal model.

[0040] Preferably, the loss function is constructed by combining the potential clear image and the blur kernel, and the posterior distribution of the random variable is optimized to train the deep neural network framework to obtain the optimal model, specifically:

[0041] The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

[0042] Preferably, the optimal model is obtained, and the corresponding calculation formula is:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] in, represents KL divergence; represents the posterior probability; Indicates a clear image; Represents the first pixels; 、 All represent random variables; represents the blur kernel; Indicates a blurred image; 、 、 、 All represent hyperparameters; Represents the prior distribution of blurred images; Represents the variables in the loss function; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; express expectations; Indicates the number of samples; Indicates the noise level; represents a random scalar sampled from a standard Gaussian distribution; represents Gaussian distribution; represents the identity matrix; Represents pixel-by-pixel multiplication.

[0050] The present invention has the following beneficial effects:

[0051] By introducing deep image prior, a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed; that is, the blur kernel is regarded as a deterministic parameter, and the hierarchical Bayesian prior model is used to model the clear image, and the posterior distribution of the random variables involved in the clear image and image prior is automatically estimated by referring to the mean field approximation; then the obtained posterior and the constructed objective function are integrated into the deep neural network framework, and the objective function is optimized using a self-supervised deep learning method to utilize the image expression ability of the deep neural network, while avoiding the matrix inversion in the traditional optimization process, and converting the theoretical advantages of the VB method into actual performance improvement and accuracy gain; that is, it aims to achieve sparse adaptive optimization based on the hierarchical Bayesian prior model, and solves the problems of insufficient generalization and poor robustness of the VB-based blind deblurring algorithm, especially in complex blur kernel scenes and real image tests, giving full play to the theoretical advantages of the VB algorithm, providing a new solution to the image blind deblurring problem, and can be applied to high-precision demand scenarios such as image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a probabilistic graphical model of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a deep neural network framework for a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention;

[0056] Figure 4A schematic diagram comparing a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention with other methods. DETAILED DESCRIPTION

[0057] To further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effects of a self-supervised image blind deblurring method based on hierarchical Bayesian representation proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable form.

[0058] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0059] The following describes in detail a specific solution of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the present invention in conjunction with the accompanying drawings.

[0060] See also Figure 1 , which shows a flowchart of the steps of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present invention, the method comprising:

[0061] Step S1: Obtain a clear image and process it to synthesize a blurred image and establish a blurred image dataset;

[0062] Step S2: constructing a hierarchical Bayesian prior model and objective function based on the motion degradation model and the blurred image dataset, and defining random variables;

[0063] Step S3: Minimize the objective function and solve the posterior distribution of the random variable in the hierarchical Bayesian prior model;

[0064] Step S4: Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model;

[0065] Step S5: Output the image restoration result of the blurred image dataset through the optimal model.

[0066] To better illustrate, the variational Bayes (VB) method has shown significant theoretical advantages in the application of blind deblurring, but the non-natural coefficient image prior used by the VB method is usually based on empirical design rather than adaptive learning, which limits the flexibility of the algorithm and is not conducive to the generalization and robustness of the algorithm; and the traditional VB method has a complex matrix inversion problem, which not only increases the computational complexity, but also leads to numerical instability, making it difficult to solve, and in practical applications, the restoration accuracy is usually lower than that of the traditional maximum posterior method, and thus the VB method is improved, and a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed, that is, an adaptive image prior representation is constructed to adapt to the complexity of the blurred image.

[0067] Further, in step S1, specifically:

[0068] A clear image is obtained, a blur kernel is obtained by generating a blur trajectory using a camera trajectory, a clear image is convolved with the blur kernel, and a blurred image is synthesized, denoted as , wherein , respectively represent the width and height of the blurred image, 3 represents the channel number of the blurred image, and a blurred image dataset is established.

[0069] It can be explained that, in order to realize the comparison and verification of the image blind deblurring effect, different scene categories are selected in the embodiment to train and evaluate a dataset covering a variety of blur kernels; specifically, first, a plurality of clear images are collected from the Internet as real samples, then a 6D camera trajectory is used to form a blur trajectory under the assumption that the scene depth is constant, the six degrees of freedom of the camera in three-dimensional space, i.e. three translations and three rotations, are captured, so that each instantaneous image is slightly different, a dynamic blur effect is synthesized, and a continuous blur trajectory is generated, and adjusted to construct a spatially varying blur kernel set, a plurality of blur kernels are obtained, wherein the blur kernel is used to describe the image blurring process, and defines how each pixel affects adjacent pixels to cause the entire image to become blurred; the clear image is convolved with the blur kernel, and 1% Gaussian noise is added to simulate camera noise to synthesize a plurality of non-uniform blurred images, and a blurred image dataset is established.

[0070] Preferably, in the embodiment, the blurred image dataset is the Lai dataset, 25 clear images are selected, 4 blur kernels are determined, and the clear images are divided into five categories of Manmade (man-made), Natural (natural landscape), People (person), Saturated (saturated color) and Text (text) according to the attributes, so that the subsequent analysis can be more clear and reliable.

[0071] Please refer to Figure 2, which shows a schematic diagram of a probabilistic graphical model of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present invention;

[0072] Furthermore, step S2 includes:

[0073] Step S21: A hierarchical Bayesian prior model is established based on the motion degradation model combined with the blurred image data set, which includes at least three layers, respectively defining the random variables in the hierarchical Bayesian prior model; wherein, in the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function; in the second and third layers, they are both defined as gamma probability density functions.

[0074] For better explanation, in the construction of traditional VB methods, two-layer modeling is usually adopted. In order to improve the versatility and adaptability of modeling, this application adopts at least three layers of Bayesian modeling for image priors to establish a hierarchical Bayesian prior model. The multi-level modeling method can more finely capture the complex features and patterns in the image, improve the accuracy and robustness of the model, and by adding at least one layer, better separate and process the features at different levels, which has stronger generalization ability than two layers; that is, mathematical modeling is performed for complex motion blurred images under spatial invariant conditions, and corresponding optimization strategies are proposed on this basis. The first layer promotes the sparsity of the image and captures the basic features of the image such as texture and edges; the second layer refines the image details, enhances the clarity and contrast of the image, and reduces noise interference; the third layer uses contextual information to perform advanced semantic analysis to achieve a deep understanding of the image, that is, focuses on globalization; based on the hierarchical Bayesian prior model, the expression ability is improved and the adaptability in different scenarios is enhanced.

[0075] Furthermore, in step S21, random variables in the hierarchical Bayesian prior model are defined respectively, and the corresponding calculation formula is:

[0076]

[0077]

[0078]

[0079]

[0080] in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

[0081] It is explained that by defining random variables in the hierarchical Bayesian prior model, the clarity and detail restoration ability of the image can be gradually improved, that is, the distribution characteristics of the random variables are clarified to help improve the overall quality of the image; among them, 、 Both represent random variables, and their distributions are respectively given by and Decision, that is, given In the case of The probability distribution and The probability distribution of .

[0082] Specifically, in the first layer of the hierarchical Bayesian prior model, the latent clear image The prior distribution of is defined as a Gaussian probability density function to effectively process the detail features in the blurred image dataset and ensure accurate analysis of the potential clear image, where Represents a potential clear image The inverse variance of , which can adaptively promote the potential clear image The sparsity of retains the significant edges in the image, which is beneficial to the estimation of the blur kernel; in the second layer, It is defined as the gamma probability density function to adapt to different data distributions, where the hyperparameter As a shape parameter, it determines the shape of the distribution; random variable As a rate parameter, it controls the concentration of the distribution. A higher rate parameter will make the distribution more concentrated, and vice versa. In the third layer, the second layer random variables The rate parameter is modeled as a random variable ,because and There is conjugation between them, so in order to facilitate the derivation and calculation of the posterior distribution, the random variable It is also defined as the gamma probability density function, where the hyperparameter As a shape parameter, as the rate parameter.

[0083] Step S22: Constructing the objective function through random variables.

[0084] Furthermore, in step S22, the corresponding calculation formula is:

[0085]

[0086]

[0087] in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

[0088] It can be explained that blurred images usually refer to images caused by shaking during shooting, defects in the optical system or noise reduction operations in post-processing. They usually lack details, have unclear edges, and appear blurred as a whole; clear images refer to images with rich details, sharp edges, bright colors and no obvious distortion, which can accurately reflect the true appearance of the photographed object and improve the visual experience; and the motion degradation model established based on blurred images and clear images is a mathematical model that describes image blur caused by object or camera movement.

[0089] To illustrate, the blur kernel here is obtained when the blur trajectory is generated by the camera trajectory in step S1; the construction of the objective function is essentially a hyperparameter 、 、 The regularized maximum likelihood estimate of , where use The norm limits the range of model parameters, making all parameters evenly distributed, preventing individual parameters from being too large, constraining the solution space of fuzzy kernel estimation, effectively preventing overfitting, and improving the generalization ability of the model; preferably, in this embodiment, the hyperparameter .

[0090] It can be understood that in step S3, the objective function is minimized and the posterior distribution of random variables in the hierarchical Bayesian prior model is solved; that is, the posterior distribution of random variables in the hierarchical Bayesian prior model is estimated and variational parameter optimization is given, that is, the objective function is minimized, the mean field approximation is used, and it is assumed that the posterior distributions of each random variable are independent of each other, and the posterior mean of the potential variables in each layer is solved.

[0091] Further, in step S3, comprising:

[0092] based on the first layer latent sharp image posterior probability subject to a multivariate Gaussian distribution , the corresponding calculation formula is:

[0093]

[0094]

[0095] The second layer and the third layer random variable are subject to a gamma distribution, and the corresponding calculation formula is:

[0096]

[0097]

[0098]

[0099]

[0100] wherein, the posterior distribution of the latent sharp image is represented by p (x | y) ; , the intermediate parameter representation in the first layer is represented by z1; the expectation is represented by E; the blurred image is represented by y; the sharp image is represented by x; , all represent random variables; , , all represent hyperparameters; the blur kernel is represented by K; , respectively represent the shape parameters of the corresponding random variables; , respectively represent the rate parameters of the corresponding random variables; the posterior distribution of the random variable is represented by p (z1 | y) ; the posterior distribution of the random variable is represented by p (z2 | y) ; , respectively represent the gradient operation in the horizontal direction and the vertical direction of the sharp image; the diagonal matrix is represented by D;

[0101] It is explained that, , the mean of the random variable , is represented by E; wherein, acting on the variational parameter optimization; specifically, due to the prior distribution of sparse images is essentially determined by three deterministic hyperparameters, and the deterministic parameters are obtained by deriving the representation formula to obtain the corresponding representations of the three parameters, and the corresponding calculation formula is:

[0102]

[0103] wherein, denotes the gamma function; and then the derivative of is:

[0104]

[0105]

[0106]

[0107]

[0108] wherein, denotes the digamma function.

[0109] Please refer to Figure 3 , which shows a deep neural network framework of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present application; Understandably, under the framework of traditional variational Bayesian inference, it is difficult to calculate the mean and variance of the multivariate Gaussian distribution , therefore, in order to solve this technical defect, the method of self-supervised deep learning is used for calculation, aiming at mining the potential advantages of VB method, and converting the theoretical advantages of the method into actual performance improvement and precision gain; That is, inspired by the attractive optimization network method of deep image prior, the goal is to estimate the mean and variance of the latent clear image , in order to solve the two sub-problems of generating latent clear image and blur kernel, and then build two neural sub-networks to establish a deep neural network framework to realize the above tasks.

[0110] Further, in step S4, it includes:

[0111] ​Step S41: design a deep neural network framework, including an encoder-decoder structure network and a fully connected network; that is, construct a generative network for blind image deblurring, use the encoder-decoder structure network as an image generator, and take any image sample in the blurred image data set as a guide to restore a clear image from noise that is equal to the input image sample and the blurred image, that is, fixed noise; use the fully connected network as a blur kernel generator to restore the blur kernel from the input fixed noise; in the subsequent iterative process, use the backpropagation algorithm to iteratively update the parameters of the deep neural network to minimize the value of the loss function, so that the optimization proceeds in the right direction.

[0112] Step S42: input fixed noise to the encoder-decoder structure network and the fully connected network for processing, and output the potential clear image and blur kernel respectively.

[0113] Specifically, the encoder includes a convolution layer, a BN layer (Batch Normalization, i.e., a normalization layer) and an activation function, and inputs a fixed noise, denoted as The input tensor is sent to the encoder, and the 3-channel tensor is converted to 128-channel through the convolution layer, where the convolution kernel size of the convolution layer is ,下采样的策略采用 Down sampling is performed; while the decoder's up sampling strategy uses bilinear interpolation, , and in the last layer of the encoder-decoder structure network, the number of channels is changed from 128 channels to 6 channels, and the mean of the potential clear image is output and standard deviation ; Preferably, in the encoder-decoder structure network, each group of encoders and decoders adopts a jump connection, and the number of channels is set to 16; the input fixed noise is recorded as This is fed into the blur kernel generator, a fully connected network, which, after passing through two linear layers and a normalized exponential function, outputs the estimated blur kernel. To illustrate, the image generator, an encoder-decoder network, generates the potential sharp image; the blur kernel generator, a fully connected network, generates the corresponding blur kernel.

[0114] Step S43: Construct a loss function by combining the potential clear image and the blur kernel, optimize it in combination with the posterior distribution of the random variable, train the deep neural network framework, and obtain the optimal model.

[0115] 进一步地,步骤S43中,具体为:

[0116] The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

[0117] It can be explained that KL divergence (Kullback–Leibler divergence, also known as relative entropy) is an asymmetric measure used to measure the difference between the posterior distribution and the prior distribution. In this embodiment, the posterior distribution of the random variable is ,损失函数中的变量后验分布为 ,具体地,通过采用 approximate , and optimizes this approximation by minimizing the KL divergence. In this way, the traditional method of using complex variance solution is transformed into a deep neural network training process to simplify the solution process and improve computational efficiency. Monte Carlo sampling is a random sampling method that can simulate various possible scenarios and obtain accurate probability distribution and statistical results. It is suitable for problems in high-dimensional space and can also handle situations with complex boundary conditions or nonlinear relationships. As the number of samples increases, the distribution becomes more accurate.

[0118] Furthermore, the optimal model is obtained, and the corresponding calculation formula is:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] in, 表示KL散度; 表示后验概率; 表示清晰图像; 表示去模糊后的清晰图像中的第 个像素; 、 均表示随机变量; 表示模糊核; 表示模糊图像; 、 、 、 均表示超参数; 表示模糊图像的先验分布; 表示损失函数中的变量; 、 Respectively represent the gradient operation in the vertical and horizontal directions; 表示对角矩阵; 表示期望; 表示采样数; 表示噪声等级; represents a random scalar sampled from a standard Gaussian distribution; 表示高斯分布; 表示单位矩阵; 表示逐像素的乘法。

[0126] Understandably, the core of the deep neural network framework is the deep image prior module, which captures the intrinsic structure and statistical information of the image, constrains the blur kernel and the potential clear image at the same time, guides the image restoration process, and avoids unreasonable restoration results; then, through iterative training, the optimal model is obtained to generate robust image restoration results, which exhibits superior performance and significant generalization ability.

[0127] Specifically, in this embodiment, the deep neural network is trained using the Adam Weight Decay Regularization (ADAM) optimizer. In the experimental parameter setting, the learning rate of the image generator, i.e., the encoder-decoder structure network, is set to 0.001, the learning rate of the blur kernel generator, i.e., the fully connected network, is set to 0.0001, and the number of optimization rounds is set to 5000 rounds. The hyperparameters 的值域区间限定在0到0.1, 的值域区间限定为0到1, 的值域区间限定为0到 The graphics processor used is RTX4090 (Ray Tracing Technology), the video memory is 24GB, the operating system is Ubuntu 20.04, the deep learning framework is Pytorch 2.0.0, and training is performed by optimizing the loss function.

[0128] It can be explained that in step S5, the image restoration result of the blurred image data set is output by the optimal model; that is, after the iterative training is completed, the optimal model outputs a clear image and the corresponding blur kernel, and obtains the corresponding deep neural network weight parameters, and then sets the relevant parameters to generate a clear image result, that is, the image restoration result.

[0129] See also Figure 4To better illustrate the effectiveness of the image blind deblurring method proposed in the present application, the present application is compared with the prior art, including method 1 (Bai Y., Cheung G., Liu X., Gao W. Graph-Based Blind Image Deblurring from a Single Photograph[J]. IEEE Transactions on Image Processing, 2019, 28(3): 1404-1418), method 2 (Wen F., Ying R., Liu Y., Liu P., Truong T. K. A Simple Local Minimal Intensity Prior and an Improved Algorithm for Blind Image Deblurring[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(8): 2923-2937), method 3 (Chen L., Fang F., Zhang J., Liu J., Zhang G. OID: Outlier Identifying and Discarding in Blind Image Deblurring[C] / / Computer Vision – ECCV 2020. Cham: Springer, 2020: 604-621), method 4 (Ren D, Zhang K, Wang Q, Hu Q, Zuo W. Neural Blind Deconvolution Using Deep Priors[C] / / 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 3338-3347), and method 5 (Shao W Z, Lin Y Z, Liu Y Y, et al. Gradient-based Discriminative Modeling for Blind Image Deblurring[J].Neurocomputing, 2020, 413: 305-327); In the experiment, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were used as the main evaluation indicators, and the specific results are shown in Table 1 and Table 2 respectively. Figure 4 In the comparison diagram, visual comparison shows that the method provided by the present application can better remove blur.

[0130] Table 1 Performance evaluation of various methods grouped by category on the Lai dataset (PSNR)

[0131]

[0132] Table 2 Performance evaluation of various methods grouped by category on the Lai dataset (SSIM)

[0133]

[0134] It can be shown that according to the quantitative evaluation of the five attribute categories in the Lai dataset, combined with two objective evaluation indicators, it is reflected that the image blind deblurring method provided by the present application is superior to the above-mentioned compared methods, and achieves better image blind deblurring effect; it shows that the present invention can perform blur removal well in complex motion blur scenes, and provides an efficient and reliable solution for the image blind deblurring task.

[0135] Understandably, by introducing deep image prior, a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed; that is, the blur kernel is regarded as a deterministic parameter, and the hierarchical Bayesian prior model is used to model the clear image, and the posterior distribution of the random variables involved in the clear image and image prior is automatically estimated by referring to the mean field approximation; then the obtained posterior and the constructed objective function are integrated into the deep neural network framework, and the objective function is optimized using a self-supervised deep learning method to utilize the image expression ability of the deep neural network, while avoiding the matrix inversion in the traditional optimization process, and converting the theoretical advantages of the VB method into actual performance improvement and accuracy gain; that is, it aims to achieve sparse adaptive optimization based on the hierarchical Bayesian prior model, and solves the problems of insufficient generalization and poor robustness of the VB-based blind deblurring algorithm, especially in complex blur kernel scenes and real image tests, and gives full play to the theoretical advantages of the VB algorithm, providing a new solution to the image blind deblurring problem, which can be applied to high-precision demand scenarios such as image enhancement.

[0136] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A self-supervised image blind deblurring method based on hierarchical Bayesian representation, characterized in that: The method comprises: Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset; Based on the motion degradation model and the blurred image dataset, a hierarchical Bayesian prior model and objective function are constructed, and random variables are defined, including: A hierarchical Bayesian prior model is established based on the motion degradation model and the blurred image dataset. The model includes at least three layers, each of which defines the random variables in the hierarchical Bayesian prior model. In the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function. In the second and third layers, the prior distribution is defined as a gamma probability density function. Construct the objective function through random variables; Minimize the objective function and solve the posterior distribution of random variables in the hierarchical Bayesian prior model, including: Based on the first layer of latent clear image The posterior probability Obey multivariate Gaussian distribution , the corresponding calculation formula is: ； ； The random variables in the second and third layers all obey the gamma distribution, and the corresponding calculation formulas are: ； ； ； ； in, represents the posterior distribution of the potential sharp image; 、 represents the intermediate parameter representation in the first layer; express expectations; Indicates a blurred image; Indicates a clear image; 、 All represent random variables; 、 、 All represent hyperparameters; represents the blur kernel; 、 They represent the shape parameters of the corresponding random variables respectively; 、 They represent the rate parameters of the corresponding random variables respectively; represents a random variable The posterior distribution of represents a random variable The posterior distribution of 、 Respectively represent the gradient operation of the clear image in the horizontal and vertical directions; represents a diagonal matrix; Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model; The image restoration results of the blurred image dataset are output by the optimal model.

2. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 1, characterized in that: Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset. Specifically: Get a clear image, use the camera trajectory to generate a blur trajectory to get a blur kernel, convolve the clear image with the blur kernel to synthesize a blurred image, which is recorded as ,in, 、 Represent the width and height of the blurred image respectively, 3 represents the number of channels of the blurred image, and a blurred image dataset is established.

3. The self-supervised image blind deblurring method based on hierarchical Bayesian representation according to claim 1, characterized in that: Define the random variables in the hierarchical Bayesian prior model respectively, and the corresponding calculation formula is: ； ； ； ； in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

4. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 1, characterized in that: The objective function is constructed by random variables, and the corresponding calculation formula is: ； ； in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

5. The self-supervised image blind deblurring method based on hierarchical Bayesian representation according to claim 2, characterized in that: Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model, including: Design deep neural network frameworks, including encoder-decoder structure networks and fully connected networks; Fixed noise is input to the encoder-decoder structure network and the fully connected network for processing, and the potential clear image and blur kernel are output respectively; The loss function is constructed by combining the latent clear image and the blur kernel, and is optimized based on the posterior distribution of random variables. The deep neural network framework is trained to obtain the optimal model.

6. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 5, characterized in that: The loss function is constructed by combining the potential clear image and the blur kernel, and optimized by combining the posterior distribution of the random variable. The deep neural network framework is trained to obtain the optimal model, specifically: The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

7. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 6, characterized in that: The optimal model is obtained, and the corresponding calculation formula is: ； ； ； ； in, represents KL divergence; represents the posterior probability; Indicates a clear image; Represents the first pixels; 、 All represent random variables; represents the blur kernel; Indicates a blurred image; 、 、 、 All represent hyperparameters; Represents the prior distribution of blurred images; Represents the variables in the loss function; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; express expectations; Indicates the number of samples; Indicates the noise level; represents a random scalar sampled from a standard Gaussian distribution; represents Gaussian distribution; represents the identity matrix; Represents pixel-by-pixel multiplication.

Images