Self-supervised image blind deblurring method based on layered Bayesian representation

Abstract

The invention relates to the technical field of image processing, computer vision and image blind deblurring, in particular to a self-supervised image blind deblurring method based on hierarchical Bayesian representation, which comprises the following steps: acquiring and processing a clear image, synthesizing a blurred image, and establishing a blurred image data set; constructing a hierarchical Bayesian prior model and a target function based on the motion degradation model in combination with the blurred image data set, and defining a random variable; the objective function is minimized, and posterior distribution of random variables in the layered Bayesian prior model is solved; a deep neural network framework is designed, and a loss function is established and trained in combination with posterior distribution of random variables to obtain an optimal model; outputting an image recovery result of the blurred image data set through the optimal model; according to the method, sparse adaptation optimization is realized on the basis of the hierarchical Bayesian prior model, the problems of insufficient generalization and poor robustness of a VB-based blind deblurring algorithm are solved, and the method can be applied to high-precision demand scenes such as image enhancement.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing, computer vision and image blind deblurring, and in particular to a self-supervised image blind deblurring method based on hierarchical Bayesian representation. Background Art

[0002] Image deblurring has long been a key research topic in the fields of computer vision and image processing. During image acquisition, processing, transmission, and storage, many factors can negatively impact image quality, such as device vibration during use, relative displacement between the device and the target during imaging, and noise contamination. These factors can cause blurring, distortion, and additive noise in the image, significantly reducing image clarity. This phenomenon is known as image degradation. To address this issue, image deblurring technology has emerged, aiming to restore clarity and usability to degraded images.

[0003] To address image degradation, existing research is typically based on the maximum a posteriori (MAP) framework, which alternates between estimating a blur kernel and a potential sharp image to gradually approximate the true image. However, studies have shown that under certain conditions, this method tends to interpret the degraded image as generated by a delta function kernel, making effective image restoration difficult and inherently ill-conditioned. Variational Bayes (VB) methods, on the other hand, have demonstrated significant theoretical advantages in blind deblurring applications. By estimating the posterior distribution of the image, VB methods are theoretically more robust than traditional MAP methods, reducing the risk of convergence to local minima.

[0004] However, when using the variational Bayesian method to deblur images, a class of sparse image priors suitable for modeling natural images, such as Gaussian mixture models and super-Gaussian distributions, are introduced. These priors can capture the sparse characteristics of images and are converted into a form suitable for inference through variational representation, thereby simplifying the calculation. However, the non-natural sparse image priors used are usually based on empirical design rather than adaptive learning, which is not conducive to the generalization and robustness of the algorithm. In addition, the traditional VB method has a complex matrix inversion problem, which makes the solution difficult and affects the computational efficiency. In addition, despite the significant theoretical advantages of VB, in practical applications, VB-based blind deblurring algorithms face huge challenges in many scenarios. Its theoretical advantages have not been fully utilized, and the actual restoration accuracy is often lower than that of optimization methods based on the maximum a posteriori method framework. Summary of the Invention

[0005] In order to solve the technical problems that the existing technology for VB image processing applications cannot adaptively learn, is not conducive to generalization and robustness, is computationally complex and difficult to solve, and has lower restoration accuracy than the traditional maximum a posteriori method, the purpose of the present invention is to provide a self-supervised image blind deblurring method based on hierarchical Bayesian representation. The technical solution adopted is as follows:

[0006] Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset;

[0007] Based on the motion degradation model and blurred image dataset, a hierarchical Bayesian prior model and objective function are constructed, and random variables are defined;

[0008] Minimize the objective function and solve the posterior distribution of random variables in the hierarchical Bayesian prior model;

[0009] Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model;

[0010] The image restoration results of the blurred image dataset are output by the optimal model.

[0011] Preferably, a clear image is acquired and processed to synthesize a blurred image and establish a blurred image dataset, specifically:

[0012] Get a clear image, use the camera trajectory to generate a blur trajectory to get a blur kernel, convolve the clear image with the blur kernel to synthesize a blurred image, which is recorded as ,in, 、 Represent the width and height of the blurred image respectively, 3 represents the number of channels of the blurred image, and a blurred image dataset is established.

[0013] Preferably, a hierarchical Bayesian prior model and objective function are constructed based on the motion degradation model combined with the blurred image dataset, and random variables are defined, including:

[0014] A hierarchical Bayesian prior model is established based on the motion degradation model and the blurred image dataset. The model includes at least three layers, each of which defines the random variables in the hierarchical Bayesian prior model. In the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function. In the second and third layers, the prior distribution is defined as a gamma probability density function.

[0015] Construct the objective function through random variables.

[0016] Preferably, the random variables in the hierarchical Bayesian prior model are defined separately, and the corresponding calculation formula is:

[0017]

[0018]

[0019]

[0020]

[0021] in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

[0022] Preferably, the objective function is constructed by random variables, and the corresponding calculation formula is:

[0023]

[0024]

[0025] in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

[0026] Preferably, minimizing the objective function and solving the posterior distribution of the random variable in the hierarchical Bayesian prior model includes:

[0027] Based on the first layer of latent clear image The posterior probability Obey multivariate Gaussian distribution , the corresponding calculation formula is:

[0028]

[0029]

[0030] The random variables in the second and third layers all obey the gamma distribution, and the corresponding calculation formulas are:

[0031]

[0032]

[0033]

[0034]

[0035] in, represents the posterior distribution of the potential sharp image; 、 represents the intermediate parameter representation in the first layer; express expectations; Indicates a blurred image; Indicates a clear image; 、 All represent random variables; 、 、 All represent hyperparameters; represents the blur kernel; 、 They represent the shape parameters of the corresponding random variables respectively; 、 They represent the rate parameters of the corresponding random variables respectively; represents a random variable The posterior distribution of represents a random variable The posterior distribution of 、 Respectively represent the gradient operation of the clear image in the horizontal and vertical directions; represents a diagonal matrix.

[0036] Preferably, a deep neural network framework is designed, a loss function is established in combination with the posterior distribution of random variables, and the deep neural network framework is trained to obtain an optimal model, including:

[0037] Design deep neural network frameworks, including encoder-decoder structure networks and fully connected networks;

[0038] Fixed noise is input to the encoder-decoder structure network and the fully connected network for processing, and the potential clear image and blur kernel are output respectively;

[0039] The loss function is constructed by combining the latent clear image and the blur kernel, and is optimized based on the posterior distribution of random variables. The deep neural network framework is trained to obtain the optimal model.

[0040] Preferably, the loss function is constructed by combining the potential clear image and the blur kernel, and the posterior distribution of the random variable is optimized to train the deep neural network framework to obtain the optimal model, specifically:

[0041] The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

[0042] Preferably, the optimal model is obtained, and the corresponding calculation formula is:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] in, represents KL divergence; represents the posterior probability; Indicates a clear image; Represents the first pixels; 、 All represent random variables; represents the blur kernel; Indicates a blurred image; 、 、 、 All represent hyperparameters; Represents the prior distribution of blurred images; Represents the variables in the loss function; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; express expectations; Indicates the number of samples; Indicates the noise level; represents a random scalar sampled from a standard Gaussian distribution; represents Gaussian distribution; represents the identity matrix; Represents pixel-by-pixel multiplication.

[0050] The present invention has the following beneficial effects:

[0051] By introducing deep image prior, a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed; that is, the blur kernel is regarded as a deterministic parameter, and the hierarchical Bayesian prior model is used to model the clear image, and the posterior distribution of the random variables involved in the clear image and image prior is automatically estimated by referring to the mean field approximation; then the obtained posterior and the constructed objective function are integrated into the deep neural network framework, and the objective function is optimized using a self-supervised deep learning method to utilize the image expression ability of the deep neural network, while avoiding the matrix inversion in the traditional optimization process, and converting the theoretical advantages of the VB method into actual performance improvement and accuracy gain; that is, it aims to achieve sparse adaptive optimization based on the hierarchical Bayesian prior model, and solves the problems of insufficient generalization and poor robustness of the VB-based blind deblurring algorithm, especially in complex blur kernel scenes and real image tests, giving full play to the theoretical advantages of the VB algorithm, providing a new solution to the image blind deblurring problem, and can be applied to high-precision demand scenarios such as image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention; Figure 2 A schematic diagram of a probabilistic graphical model of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention; Figure 3 A schematic diagram of a deep neural network framework for a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention; Figure 4 A schematic diagram comparing a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by one embodiment of the present invention with other methods. DETAILED DESCRIPTION

[0054] To further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effects of a self-supervised image blind deblurring method based on hierarchical Bayesian representation proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable form.

[0055] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0056] The following describes in detail a specific solution of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the present invention in conjunction with the accompanying drawings.

[0057] See also Figure 1 , which shows a flowchart of the steps of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present invention, the method comprising:

[0058] Step S1: Obtain a clear image and process it to synthesize a blurred image and establish a blurred image dataset;

[0059] Step S2: constructing a hierarchical Bayesian prior model and objective function based on the motion degradation model and the blurred image dataset, and defining random variables;

[0060] Step S3: Minimize the objective function and solve the posterior distribution of the random variable in the hierarchical Bayesian prior model;

[0061] Step S4: Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model;

[0062] Step S5: Output the image restoration result of the blurred image dataset through the optimal model.

[0063] To better illustrate, the Variational Bayes (VB) method has shown significant theoretical advantages in the application of blind deblurring. However, the non-natural coefficient image priors it commonly uses are based on empirical design rather than adaptive learning, which limits the flexibility of the algorithm and is not conducive to the generalization and robustness of the algorithm. In addition, the traditional VB method has a complex matrix inversion problem, which not only increases the computational complexity, but also leads to numerical instability, making the solution difficult. In practical applications, the restoration accuracy is usually lower than that of the traditional maximum a posteriori method. Therefore, the VB method is improved, and a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed in this application, that is, an adaptive image prior representation is constructed to adapt to the complexity of the blurred image.

[0064] Furthermore, in step S1, specifically:

[0065] Get a clear image, use the camera trajectory to generate a blur trajectory to get a blur kernel, convolve the clear image with the blur kernel to synthesize a blurred image, which is recorded as ,in, 、 Represent the width and height of the blurred image respectively, 3 represents the number of channels of the blurred image, and a blurred image dataset is established.

[0066] It can be explained that, in order to achieve the comparison and verification of the blind deblurring effect of the image, this embodiment selects different scene categories and a data set covering multiple blur kernels for training and evaluation; specifically, first, a number of clear images are collected from the Internet as real samples, and then a 6D camera trajectory is used to form a blur trajectory under the assumption that the scene depth is constant. By capturing the six degrees of freedom of the camera in three-dimensional space, namely three translations and three rotations, the image at each moment is slightly different, and a dynamic blur effect is synthesized to generate a continuous blur trajectory, and then adjusted to construct a spatially varying blur kernel set to obtain multiple blur kernels, where the blur kernel is used to describe the image blurring process and define how each pixel affects adjacent pixels, causing the entire image to become blurred; the clear image is convolved with the blur kernel, and 1% Gaussian noise is added to simulate camera noise to synthesize multiple non-uniform blurred images, and a blurred image dataset is established.

[0067] Preferably, in this embodiment, the blurred image dataset is the Lai dataset, 25 clear images are selected, the blur kernel is determined to be 4, and the clear images are divided into five categories according to their attributes: Manmade, Natural, People, Saturated, and Text, so that subsequent analysis can be clearer and more reliable.

[0068] See also Figure 2, which shows a schematic diagram of a probabilistic graphical model of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present invention;

[0069] Furthermore, step S2 includes:

[0070] Step S21: A hierarchical Bayesian prior model is established based on the motion degradation model combined with the blurred image data set, which includes at least three layers, respectively defining the random variables in the hierarchical Bayesian prior model; wherein, in the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function; in the second and third layers, they are both defined as gamma probability density functions.

[0071] For better explanation, in the construction of traditional VB methods, two-layer modeling is usually adopted. In order to improve the versatility and adaptability of modeling, this application adopts at least three layers of Bayesian modeling for image priors to establish a hierarchical Bayesian prior model. The multi-level modeling method can more finely capture the complex features and patterns in the image, improve the accuracy and robustness of the model, and by adding at least one layer, better separate and process the features at different levels, which has stronger generalization ability than two layers; that is, mathematical modeling is performed for complex motion blurred images under spatial invariant conditions, and corresponding optimization strategies are proposed on this basis. The first layer promotes the sparsity of the image and captures the basic features of the image such as texture and edges; the second layer refines the image details, enhances the clarity and contrast of the image, and reduces noise interference; the third layer uses contextual information to perform advanced semantic analysis to achieve a deep understanding of the image, that is, focuses on globalization; based on the hierarchical Bayesian prior model, the expression ability is improved and the adaptability in different scenarios is enhanced.

[0072] Furthermore, in step S21, random variables in the hierarchical Bayesian prior model are defined respectively, and the corresponding calculation formula is:

[0073]

[0074]

[0075]

[0076]

[0077] in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

[0078] It is explained that by defining random variables in the hierarchical Bayesian prior model, the clarity and detail restoration ability of the image can be gradually improved, that is, the distribution characteristics of the random variables are clarified to help improve the overall quality of the image; among them, 、 Both represent random variables, and their distributions are respectively given by and Decision, that is, given In the case of The probability distribution and The probability distribution of .

[0079] Specifically, in the first layer of the hierarchical Bayesian prior model, the latent clear image The prior distribution of is defined as a Gaussian probability density function to effectively process the detail features in the blurred image dataset and ensure accurate analysis of the potential clear image, where Represents a potential clear image The inverse variance of , which can adaptively promote the potential clear image The sparsity of retains the significant edges in the image, which is beneficial to the estimation of the blur kernel; in the second layer, It is defined as the gamma probability density function to adapt to different data distributions, where the hyperparameter As a shape parameter, it determines the shape of the distribution; random variable As a rate parameter, it controls the concentration of the distribution. A higher rate parameter will make the distribution more concentrated, and vice versa. In the third layer, the second layer random variables The rate parameter is modeled as a random variable ,because and There is conjugation between them, so in order to facilitate the derivation and calculation of the posterior distribution, the random variable It is also defined as the gamma probability density function, where the hyperparameter As a shape parameter, as the rate parameter.

[0080] Step S22: Constructing the objective function through random variables.

[0081] Furthermore, in step S22, the corresponding calculation formula is:

[0082]

[0083]

[0084] in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

[0085] It can be explained that blurred images usually refer to images caused by shaking during shooting, defects in the optical system or noise reduction operations in post-processing. They usually lack details, have unclear edges, and appear blurred as a whole; clear images refer to images with rich details, sharp edges, bright colors and no obvious distortion, which can accurately reflect the true appearance of the photographed object and improve the visual experience; and the motion degradation model established based on blurred images and clear images is a mathematical model that describes image blur caused by object or camera movement.

[0086] To illustrate, the blur kernel here is obtained when the blur trajectory is generated by the camera trajectory in step S1; the construction of the objective function is essentially a hyperparameter 、 、 The regularized maximum likelihood estimate of , where use The norm limits the range of model parameters, making all parameters evenly distributed, preventing individual parameters from being too large, constraining the solution space of fuzzy kernel estimation, effectively preventing overfitting, and improving the generalization ability of the model; preferably, in this embodiment, the hyperparameter .

[0087] It can be understood that in step S3, the objective function is minimized and the posterior distribution of random variables in the hierarchical Bayesian prior model is solved; that is, the posterior distribution of random variables in the hierarchical Bayesian prior model is estimated and variational parameter optimization is given, that is, the objective function is minimized, the mean field approximation is used, and it is assumed that the posterior distributions of each random variable are independent of each other, and the posterior mean of the potential variables in each layer is solved.

[0088] Furthermore, step S3 includes:

[0089] Based on the first layer of latent clear image The posterior probability Obey multivariate Gaussian distribution , the corresponding calculation formula is:

[0090]

[0091]

[0092] The random variables in the second and third layers all obey the gamma distribution, and the corresponding calculation formulas are:

[0093]

[0094]

[0095]

[0096]

[0097] in, represents the posterior distribution of the potential sharp image; 、 represents the intermediate parameter representation in the first layer; express expectations; Indicates a blurred image; Indicates a clear image; 、 All represent random variables; 、 、 All represent hyperparameters; represents the blur kernel; 、 They represent the shape parameters of the corresponding random variables respectively; 、 They represent the rate parameters of the corresponding random variables respectively; represents a random variable The posterior distribution of represents a random variable The posterior distribution of 、 Respectively represent the gradient operation of the clear image in the horizontal and vertical directions; represents a diagonal matrix.

[0098] Make an explanation, 、 Represent random variables 、 The mean of ; among them, Acts on variational parameter optimization; specifically, due to the prior distribution of sparse images Essentially, it is The three deterministic hyperparameters are determined and the deterministic parameters are obtained by deriving the characterization formula , to obtain the corresponding representation of the three parameters, the corresponding calculation formula is:

[0099]

[0100] in, represents the gamma function; and then we get The derivative of is:

[0101]

[0102]

[0103]

[0104]

[0105] in, represents the digamma function.

[0106] See also Figure 3 , which shows a schematic diagram of a deep neural network framework of a self-supervised image blind deblurring method based on hierarchical Bayesian representation provided by the first embodiment of the present invention; It can be understood that under the framework of traditional variational Bayesian inference, for multivariate Gaussian distribution The mean and variance of are difficult to calculate, so in order to solve this technical defect, the self-supervised deep learning method is used for calculation, aiming to explore the potential advantages of the VB method and transform its theoretical advantages into actual performance improvement and accuracy gain; that is, inspired by the optimization network method with attractive deep image prior, the goal is to estimate the potential clear image The mean and variance of are used to solve the two sub-problems of generating potential clear images and blur kernels, and then two neural sub-networks are constructed to establish a deep neural network framework to achieve the above tasks.

[0107] Furthermore, step S4 includes:

[0108] Step S41: design a deep neural network framework, including an encoder-decoder structure network and a fully connected network; that is, construct a generative network for blind image deblurring, use the encoder-decoder structure network as an image generator, and take any image sample in the blurred image data set as a guide to restore a clear image from noise that is equal to the input image sample and the blurred image, that is, fixed noise; use the fully connected network as a blur kernel generator to restore the blur kernel from the input fixed noise; in the subsequent iterative process, use the backpropagation algorithm to iteratively update the parameters of the deep neural network to minimize the value of the loss function, so that the optimization proceeds in the right direction.

[0109] Step S42: input fixed noise to the encoder-decoder structure network and the fully connected network for processing, and output the potential clear image and blur kernel respectively.

[0110] Specifically, the encoder includes a convolution layer, a BN layer (Batch Normalization, i.e., a normalization layer) and an activation function, and inputs a fixed noise, denoted as The input tensor is sent to the encoder, and the 3-channel tensor is converted to 128-channel through the convolution layer, where the convolution kernel size of the convolution layer is , the downsampling strategy adopts Down sampling is performed; while the decoder's up sampling strategy uses bilinear interpolation, , and in the last layer of the encoder-decoder structure network, the number of channels is changed from 128 channels to 6 channels, and the mean of the potential clear image is output and standard deviation ; Preferably, in the encoder-decoder structure network, each group of encoders and decoders adopts a jump connection, and the number of channels is set to 16; the input fixed noise is recorded as This is fed into the blur kernel generator, a fully connected network, which, after passing through two linear layers and a normalized exponential function, outputs the estimated blur kernel. To illustrate, the image generator, an encoder-decoder network, generates the potential sharp image; the blur kernel generator, a fully connected network, generates the corresponding blur kernel.

[0111] Step S43: Construct a loss function by combining the potential clear image and the blur kernel, optimize it in combination with the posterior distribution of the random variable, train the deep neural network framework, and obtain the optimal model.

[0112] Furthermore, in step S43, specifically:

[0113] The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

[0114] It can be explained that KL divergence (Kullback–Leibler divergence, also known as relative entropy) is an asymmetric measure used to measure the difference between the posterior distribution and the prior distribution. In this embodiment, the posterior distribution of the random variable is , the posterior distribution of the variables in the loss function is Specifically, by adopting approximate , and optimizes this approximation by minimizing the KL divergence. In this way, the traditional method of using complex variance solution is transformed into a deep neural network training process to simplify the solution process and improve computational efficiency. Monte Carlo sampling is a random sampling method that can simulate various possible scenarios and obtain accurate probability distribution and statistical results. It is suitable for problems in high-dimensional space and can also handle situations with complex boundary conditions or nonlinear relationships. As the number of samples increases, the distribution becomes more accurate.

[0115] Furthermore, the optimal model is obtained, and the corresponding calculation formula is:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122] in, represents KL divergence; represents the posterior probability; Indicates a clear image; Represents the first pixels; 、 All represent random variables; represents the blur kernel; Indicates a blurred image; 、 、 、 All represent hyperparameters; Represents the prior distribution of blurred images; Represents the variables in the loss function; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; express expectations; Indicates the number of samples; Indicates the noise level; represents a random scalar sampled from a standard Gaussian distribution; represents Gaussian distribution; represents the identity matrix; Represents pixel-by-pixel multiplication.

[0123] Understandably, the core of the deep neural network framework is the deep image prior module, which captures the intrinsic structure and statistical information of the image, constrains the blur kernel and the potential clear image at the same time, guides the image restoration process, and avoids unreasonable restoration results; then, through iterative training, the optimal model is obtained to generate robust image restoration results, which exhibits superior performance and significant generalization ability.

[0124] Specifically, in this embodiment, the deep neural network is trained using the Adam Weight Decay Regularization (ADAM) optimizer. In the experimental parameter setting, the learning rate of the image generator, i.e., the encoder-decoder structure network, is set to 0.001, the learning rate of the blur kernel generator, i.e., the fully connected network, is set to 0.0001, and the number of optimization rounds is set to 5000 rounds. The hyperparameters The range of the value is limited to 0 to 0.1. The value range of is limited to 0 to 1. The range of values ​​is limited to 0 to The graphics processor used is RTX4090 (Ray Tracing Technology), the video memory is 24GB, the operating system is Ubuntu 20.04, the deep learning framework is Pytorch 2.0.0, and training is performed by optimizing the loss function.

[0125] It can be explained that in step S5, the image restoration result of the blurred image data set is output by the optimal model; that is, after the iterative training is completed, the optimal model outputs a clear image and the corresponding blur kernel, and obtains the corresponding deep neural network weight parameters, and then sets the relevant parameters to generate a clear image result, that is, the image restoration result.

[0126] See also Figure 4, for better explanation and to verify the effectiveness of the image blind area blurring method proposed in this application, this application is compared with the existing technology, including method 1 (Bai Y., Cheung G., Liu X., Gao W. Graph-Based Blind Image Deblurring from a Single Photograph[J]. IEEE Transactions on Image Processing, 2019, 28(3): 1404-1418), method 2 (Wen F., Ying R., Liu Y.,Liu P., Truong TK A Simple Local Minimal Intensity Prior and an Improved Algorithm for Blind Image Deblurring[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(8): 2923 - 2937), and method 3 (Chen L., Fang F., Zhang J., Liu J., Zhang G. OID: Outlier Identifying and Discarding in Blind Image Deblurring[C] / / Computer Vision – ECCV 2020. Cham: Springer, 2020:604 - 621), Method 4 (Ren D, Zhang K, Wang Q, Hu Q, Zuo W. Neural BlindDeconvolution Using Deep Priors[C] / / 2020 IEEE / CVF Conference on ComputerVision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 3338-3347), Method 5 (Shao WZ, Lin YZ, Liu YY, et al. Gradient-based DiscriminativeModeling for Blind Image Deblurring[J].Neurocomputing, 2020, 413: 305-327); In the experiment, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were used as the main evaluation indicators, and the specific results are shown in Table 1 and Table 2 respectively. Figure 4 In the comparison diagram, visual comparison shows that the method provided by the present application can better remove blur.

[0127] Table 1 Performance evaluation of various methods grouped by category on the Lai dataset (PSNR)

[0128] Table 2 Performance evaluation of various methods grouped by category on the Lai dataset (SSIM)

[0129] It can be shown that according to the quantitative evaluation of the five attribute categories in the Lai dataset, combined with two objective evaluation indicators, it is reflected that the image blind deblurring method provided by the present application is superior to the above-mentioned compared methods, and achieves better image blind deblurring effect; it shows that the present invention can perform blur removal well in complex motion blur scenes, and provides an efficient and reliable solution for the image blind deblurring task.

[0130] Understandably, by introducing deep image prior, a self-supervised image blind deblurring method based on hierarchical Bayesian representation is proposed; that is, the blur kernel is regarded as a deterministic parameter, and the hierarchical Bayesian prior model is used to model the clear image, and the posterior distribution of the random variables involved in the clear image and image prior is automatically estimated by referring to the mean field approximation; then the obtained posterior and the constructed objective function are integrated into the deep neural network framework, and the objective function is optimized using a self-supervised deep learning method to utilize the image expression ability of the deep neural network, while avoiding the matrix inversion in the traditional optimization process, and converting the theoretical advantages of the VB method into actual performance improvement and accuracy gain; that is, it aims to achieve sparse adaptive optimization based on the hierarchical Bayesian prior model, and solves the problems of insufficient generalization and poor robustness of the VB-based blind deblurring algorithm, especially in complex blur kernel scenes and real image tests, and gives full play to the theoretical advantages of the VB algorithm, providing a new solution to the image blind deblurring problem, which can be applied to high-precision demand scenarios such as image enhancement.

[0131] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A self-supervised image blind deblurring method based on hierarchical Bayesian representation, characterized in that: The method comprises: Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset; Based on the motion degradation model and blurred image dataset, a hierarchical Bayesian prior model and objective function are constructed, and random variables are defined; Minimize the objective function and solve the posterior distribution of random variables in the hierarchical Bayesian prior model; Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model; The image restoration results of the blurred image dataset are output by the optimal model.

2. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 1, characterized in that: Obtain and process clear images, synthesize blurred images, and establish a blurred image dataset. Specifically: Get a clear image, use the camera trajectory to generate a blur trajectory to get a blur kernel, convolve the clear image with the blur kernel to synthesize a blurred image, which is recorded as ,in, 、 Represent the width and height of the blurred image respectively, 3 represents the number of channels of the blurred image, and a blurred image dataset is established.

3. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 1, characterized in that: Based on the motion degradation model and the blurred image dataset, a hierarchical Bayesian prior model and objective function are constructed, and random variables are defined, including: A hierarchical Bayesian prior model is established based on the motion degradation model and the blurred image dataset. The model includes at least three layers, each of which defines the random variables in the hierarchical Bayesian prior model. In the first layer, the potential clear image is defined as a random variable, and the prior distribution of the potential clear image is determined, which is defined as a Gaussian probability density function. In the second and third layers, the prior distribution is defined as a gamma probability density function. Construct the objective function through random variables.

4. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 3, characterized in that: Define the random variables in the hierarchical Bayesian prior model respectively, and the corresponding calculation formula is: ； ； ； ； in, A hierarchical representation of the latent sharp image; Indicates a potentially clear image; 、 、 All represent hyperparameters; 、 All represent random variables; Represents the prior distribution of potential clear images; represents Gaussian distribution; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; represents the prior distribution of the second-level random variables; The number of pixels representing the potential sharp image; represents the gamma distribution; represents the prior distribution of the third-level random variables.

5. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 3, characterized in that: The objective function is constructed by random variables, and the corresponding calculation formula is: ； ； in, represents the objective function; 、 、 、 All represent hyperparameters; Indicates a blurred image; Indicates a clear image; represents the blur kernel; represents Gaussian distribution; Represents the motion degradation model.

6. The self-supervised image blind deblurring method based on hierarchical Bayesian representation according to claim 3, characterized in that: Minimize the objective function and solve the posterior distribution of random variables in the hierarchical Bayesian prior model, including: Based on the first layer of latent clear image The posterior probability Obey multivariate Gaussian distribution , the corresponding calculation formula is: ； ； The random variables in the second and third layers all obey the gamma distribution, and the corresponding calculation formulas are: ； ； ； ； in, represents the posterior distribution of the potential sharp image; 、 represents the intermediate parameter representation in the first layer; express expectations; Indicates a blurred image; Indicates a clear image; 、 All represent random variables; 、 、 All represent hyperparameters; represents the blur kernel; 、 They represent the shape parameters of the corresponding random variables respectively; 、 They represent the rate parameters of the corresponding random variables respectively; represents a random variable The posterior distribution of represents a random variable The posterior distribution of 、 Respectively represent the gradient operation of the clear image in the horizontal and vertical directions; represents a diagonal matrix.

7. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 2, characterized in that: Design a deep neural network framework, establish a loss function based on the posterior distribution of random variables, train the deep neural network framework, and obtain the optimal model, including: Design deep neural network frameworks, including encoder-decoder structure networks and fully connected networks; Fixed noise is input to the encoder-decoder structure network and the fully connected network for processing, and the potential clear image and blur kernel are output respectively; The loss function is constructed by combining the latent clear image and the blur kernel, and is optimized based on the posterior distribution of random variables. The deep neural network framework is trained to obtain the optimal model.

8. The self-supervised image blind deblurring method based on hierarchical Bayesian representation according to claim 7, characterized in that: The loss function is constructed by combining the potential clear image and the blur kernel, and optimized by combining the posterior distribution of the random variable. The deep neural network framework is trained to obtain the optimal model, specifically: The loss function is constructed by combining the potential clear image and the blur kernel. The posterior distribution of the random variables is used to approximate the posterior distribution of the variables in the loss function. The optimization is performed by minimizing the KL divergence to maximize some variables in the loss function. Combined with the posterior distribution of the random variables, the variables in the loss function are calculated through Monte Carlo sampling to obtain the optimal model.

9. The self-supervised blind image deblurring method based on hierarchical Bayesian representation according to claim 8, characterized in that: The optimal model is obtained, and the corresponding calculation formula is: ； ； ； ； ； ； in, represents KL divergence; represents the posterior probability; Indicates a clear image; Represents the first pixels; 、 All represent random variables; represents the blur kernel; Indicates a blurred image; 、 、 、 All represent hyperparameters; Represents the prior distribution of blurred images; Represents the variables in the loss function; 、 Respectively represent the gradient operation in the vertical and horizontal directions; represents a diagonal matrix; express expectations; Indicates the number of samples; Indicates the noise level; represents a random scalar sampled from a standard Gaussian distribution; represents Gaussian distribution; represents the identity matrix; Represents pixel-by-pixel multiplication.

Images