Non-blind deblurring method, device, storage medium and computer program

Abstract

The invention discloses a non-blind deblurring method and device, a storage medium and a computer program. The method comprises the steps of obtaining a blurred image and a blurring kernel corresponding to the blurred image; performing data enhancement on the fuzzy kernel to obtain a disturbance fuzzy kernel; inputting the blurred image and the disturbance blurring kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image; and integrating the deblurred images corresponding to different disturbance blurring cores of the blurred image to obtain a predicted clear image. According to the method, the deblurred images corresponding to the different disturbance blurring kernels of the blurred image are integrated to obtain the predicted clear image, so that the defect of the clear image determined based on the single blurring kernel is compensated, the output clear image is closer to a true value, and the influence caused by the error of the blurring kernel is reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of non-blind image deconvolution, and in particular to a non-blind deblurring method, device, storage medium and computer program. Background Art

[0002] With the continuous development of social informatization, the speed of digital data generation has increased rapidly. This explosive growth of data has greatly promoted the rapid development of artificial intelligence, especially deep learning. Thanks to the huge amount of data, efficient algorithms and advanced hardware support, deep learning has achieved remarkable results. However, deep learning technology that relies on big data still faces many challenges. In the field of image processing, non-blind deconvolution technology refers to the process of restoring a clear image from a blurred image and its corresponding blur kernel. The specific role of this technology in scientific imaging and image restoration includes: how to effectively suppress noise increase while restoring image details. Traditional non-blind deblurring methods, such as regularization techniques based on image priors, usually solve the problem through artificially defined image priors. Although these methods are theoretically feasible, their iterative optimization process is usually time-consuming, and each image needs to optimize the regularization parameters separately, which makes it difficult to achieve the ideal standard in restoration effect.

[0003] The optimization and research of non-blind deblurring methods with blur kernel errors are gradually moving towards automation and intelligence. At present, the use of machine learning and deep learning technologies has become the mainstream trend. These technologies can automatically adjust parameters by learning from a large number of blur and clear image pairs, significantly improving the processing speed and image restoration quality. In particular, deep learning technology has been widely used in the field of image deblurring due to its powerful capabilities in feature extraction and pattern recognition, and has become a hot spot in the research of non-blind deblurring algorithms. Non-blind deblurring methods based on deep learning usually adopt supervised learning strategies and require a large amount of training data with known blur kernels and corresponding clear images. However. In some scientific imaging applications, such as microscopy imaging and remote sensing imaging, it may be very difficult to obtain a large number of clear training samples. In addition, in many real-world application scenarios, due to environmental and equipment limitations, accurate estimation of blur kernels is often a challenge. Therefore, it is particularly important to study how to perform effective non-blind deblurring in the presence of blur kernel estimation errors. This includes developing deblurring algorithms that can tolerate blur kernel estimation errors to reduce dependence on accurate blur kernels. Such research can not only promote the further development of image deblurring technology, but also be extended to more practical applications, improving the practicality and flexibility of image processing technology. Summary of the invention

[0004] Based on the above technical problems, the present disclosure provides a non-blind deblurring method, device, storage medium and computer program.

[0005] In a first aspect, the present disclosure provides a non-blind deblurring method, comprising:

[0006] Acquire a blurred image and a blur kernel corresponding to the blurred image;

[0007] Performing data enhancement on the blur kernel to obtain a disturbed blur kernel;

[0008] Inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image;

[0009] The deblurred images corresponding to different disturbance blur kernels of the blurred image are integrated to obtain a predicted clear image.

[0010] In some embodiments, the step of performing data enhancement on the blur kernel to obtain a disturbed blur kernel includes:

[0011] Random noise is added to the blur kernel to obtain a disturbed blur kernel.

[0012] In some embodiments, the step of acquiring a blurred image and a blur kernel corresponding to the blurred image comprises:

[0013] Get the blurred image;

[0014] Extracting straight line features from the blurred image using integral transform;

[0015] A blur kernel corresponding to the blurred image is determined based on the straight line features.

[0016] In some embodiments, the step of integrating the deblurred images corresponding to different perturbation blur kernels of the blurred image to obtain a predicted clear image comprises:

[0017] The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

[0018] In some embodiments, before the step of inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain the deblurred image, the step further includes:

[0019] Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image;

[0020] An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

[0021] In some embodiments, the step of constructing an initial non-blind deblurring model based on a machine learning algorithm, taking the blurred image and the blur kernel as inputs of the initial non-blind deblurring model, taking the clear image as output of the initial non-blind deblurring model, and optimizing and training the initial non-blind deblurring model using corresponding sample data to obtain a non-blind deblurring prediction model includes:

[0022] Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image;

[0023] Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model;

[0024] A non-blind deblurring prediction model is determined based on the adjusted parameters.

[0025] In some embodiments, the step of determining the non-blind deblurring prediction model based on the adjusted parameters comprises:

[0026] Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval;

[0027] Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold;

[0028] When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

[0029] In a second aspect, the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the above aspect when executing the computer program.

[0030] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the above aspects when executed by a processor.

[0031] In a fourth aspect, the present disclosure provides a computer program product, including a computer program, which implements the steps of the method described in the above aspects when executed by a processor.

[0032] The present invention provides a non-blind deblurring method, a computer device, a storage medium and a computer program product, which integrate deblurred images corresponding to different perturbation blur kernels of the blurred image to obtain a predicted clear image, so as to compensate for the defects of the clear image determined based on a single blur kernel, make the output clear image closer to the true value, reduce the influence of the error of the blur kernel itself, and thus achieve the output of the clear image does not depend on the accuracy of the blur kernel estimation, and ensure that an accurate clear image can be output when there is an error in the blur kernel estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0034] Figure 1 A flowchart of a non-blind deblurring method provided by an embodiment of the present disclosure;

[0035] Figure 2 A flowchart of a non-blind deblurring method provided by an embodiment of the present disclosure;

[0036] Figure 3 A flowchart of a non-blind deblurring method provided in an embodiment of the present disclosure;

[0037] Figure 4 A network structure based on optimized expansion of a non-blind defuzzification model provided in an embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram showing a comparison of deblurred images output by the models with and without disturbance training provided in an embodiment of the present disclosure;

[0039] Figure 6 A schematic diagram of a deblurred image output based on a non-blind deblurring prediction model provided in an embodiment of the present disclosure;

[0040] Figure 7 A schematic diagram of a deblurred image output based on a non-blind deblurring prediction model provided in an embodiment of the present disclosure.

[0041] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] Figure 1 A flowchart of a non-blind deblurring method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, a non-blind deblurring method comprises:

[0047] Step 110: Obtain a blurred image and a blur kernel corresponding to the blurred image.

[0048] The blur kernel, also called the convolution kernel or blur operator, is a mathematical matrix used to describe the interaction between pixels during the image blurring process. A blurred image can be obtained by convolving a clear image with the blur kernel, or a clear image can be obtained by deconvolving a blurred image with the blur kernel.

[0049] In one embodiment, blur may be intentionally created when taking a photo with a camera or a mobile phone, or a blurred image may be acquired from an existing image library.

[0050] In one embodiment, a blurred image is first acquired, and then the blurred image is estimated based on a blur kernel to obtain a blur kernel corresponding to the blurred image.

[0051] In one embodiment, each blurred image corresponds to a plurality of blur kernels, and a set of data is determined based on each blurred image and a corresponding blur kernel.

[0052] Step 120, performing data enhancement on the blur kernel to obtain a disturbed blur kernel.

[0053] Data augmentation of blur kernels refers to increasing the diversity and complexity of a dataset by introducing small changes.

[0054] In this embodiment, data enhancement is performed on the blur kernel, including adding noise to the blur kernel to obtain a disturbed blur kernel.

[0055] In one embodiment, the blur kernel may be smoothed or sharpened to obtain a disturbed blur kernel, which is not listed herein.

[0056] Step 130: input the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model to perform deblurring prediction to obtain a deblurred image.

[0057] In this embodiment, based on the constructed non-blind deblurring prediction model, the blurred image and the perturbation blur kernel are input into the model to obtain a deblurred image. Wherein, based on the same blurred image, according to multiple perturbation blur kernels, inputting into the non-blind deblurring prediction model will obtain multiple deblurred images.

[0058] Among them, the influence caused by the blur kernel error may cause obvious distortion or artifacts in the deconvolved image. Moreover, these artifacts caused by the blur kernel error have a certain degree of independence in the restored image. Therefore, depending on the different perturbation blur kernels, the deblurred images obtained in the non-blind deblurring prediction model are different and independent of each other.

[0059] In one embodiment, a non-blind defuzzification prediction model is constructed based on a machine learning algorithm. Furthermore, the model uses a quadratic splitting method to expand a data optimization model for optimizing non-blind defuzzification.

[0060] Step 140 , integrating the deblurred images corresponding to different disturbance blur kernels of the blurred image to obtain a predicted clear image.

[0061] The presence of blur kernel errors will cause obvious artifacts in the deblurred image. When the perturbed blur kernel after random perturbation is input into the non-blind deblurring prediction model for deblurring, the perturbed blur kernel error

[0062] In this embodiment, the deblurred images corresponding to different perturbation blur kernels of the blurred image are integrated, and the integration method includes averaging the pixel values ​​of each deblurred image, weighted averaging, image fusion, etc., and a predicted clear image is obtained based on the integration of each deblurred image. Due to the difference in the perturbation blur kernels, each deblurred image is different and independent of each other. The integration of each deblurred image to obtain a predicted clear image can effectively reduce the artifacts caused by the estimated blur kernel and improve the accuracy of the clear image.

[0063] In this embodiment, the deblurred images corresponding to different perturbation blur kernels of the blurred image are integrated to obtain a predicted clear image, so as to compensate for the defects of the clear image determined based on a single blur kernel, make the output clear image closer to the true value, reduce the influence of the error of the blur kernel itself, and thus realize that the output of the clear image does not depend on the accuracy of the blur kernel estimation, ensuring that an accurate clear image can be output when there is an error in the blur kernel estimation

[0064] Embodiment 2

[0065] Based on the above embodiment, the step of performing data enhancement on the blur kernel to obtain a disturbed blur kernel includes:

[0066] Random noise is added to the blur kernel to obtain a disturbed blur kernel.

[0067] In this embodiment, multiple disturbed blur kernels are obtained by adding random noise to the blur kernel, wherein the added random noise includes Gaussian noise or salt and pepper noise to simulate uncertainty or sensor noise in the blur process.

[0068] Specifically, the perturbation blur kernel is expressed as: Γ(K)=K+n.

[0069] Where K is the original blur kernel and n is the noise added to the blur kernel.

[0070] Embodiment 3

[0071] Based on the above embodiment, the step of obtaining a blurred image and a blur kernel corresponding to the blurred image includes:

[0072] Get the blurred image;

[0073] Extracting straight line features from the blurred image using integral transform;

[0074] A blur kernel corresponding to the blurred image is determined based on the straight line features.

[0075] In this embodiment, a blurred image is first acquired, and a blur kernel is estimated on the blurred image to determine a blur kernel corresponding to the blurred image.

[0076] Specifically, the steps include:

[0077] (1) Radon transform is used to extract straight line features from the blurred image, and then the direction and shape of the blur kernel are inferred. The core of this method lies in the application of Radon transform, which captures straight line features by calculating the integral projection of the image at different angles.

[0078] (2) First, perform Radon transform on the blurred image Y and calculate the projection R(θ, t) at each angle θ, which is expressed as:

[0079] R(θ,t)=∫Y(x,y)δ(xcosθ+ysinθ-t)dx dyR(θ,t)

[0080] Here, δ is the Dirac function used to integrate over the straight line xcosθ+ysinθ=t.

[0081] (3) By analyzing the results of Radon transform, observing the intensity distribution of the projection at a specific angle, it is helpful to infer the main direction of the blur kernel. Based on this information, a blur kernel K can be constructed, which is usually assumed to be a linear structure with a certain length in a specific direction. The blur kernel can be expressed as:

[0082] K(s)=λexp(-s 2 / σ 2 )

[0083] Here, s is the coordinate along the blur direction, and the parameters λ and σ are adjusted according to the Radon transform results to adapt to the strength and size of the blur kernel.

[0084] Embodiment 4

[0085] On the basis of the above embodiment, the step of integrating the deblurred images corresponding to different perturbation blur kernels of the blurred image to obtain a predicted clear image comprises:

[0086] The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

[0087] In one embodiment, the deblurred images corresponding to different perturbation blur kernels of the blurred image are integrated, and the integration method includes averaging the pixel values ​​of the deblurred images, weighted averaging, image fusion, etc., and a predicted clear image is obtained based on the integration of the deblurred images. When selecting the integration method, it is necessary to consider the similarities and differences between the deblurred images, as well as the distribution of noise and artifacts.

[0088] In this embodiment, the deblurred images corresponding to different disturbance blur kernels based on the same blurred image are integrated to obtain a predicted clear image; wherein the integration method is to calculate the pixel average value of each deblurred image.

[0089] Specifically, the integrated expression is:

[0090]

[0091] Where M is the non-blind deblurring prediction model, Y is the blurred image, K is the blur kernel corresponding to the blurred image, Γ(K) is the perturbation blur kernel, Represents the square of the Frobenius norm of the matrix.

[0092] In this embodiment, by integrating the deblurred images, the potential of the test data can be further explored to ensure that a better deblurring effect can be obtained. At the same time, the deblurring performance can be optimized by adjusting the disturbance method and the integration algorithm.

[0093] Embodiment 5

[0094] On the basis of the above embodiment, before the step of inputting the blurred image and the perturbation blur kernel into the non-blind deblurring prediction model for deblurring prediction to obtain the deblurred image, the step further includes:

[0095] Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image;

[0096] An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

[0097] In one embodiment, sample data consisting of a sample blurred image, a sample blur kernel corresponding to the sample blurred image, and a sample clear image is obtained. There may be multiple blur kernels corresponding to each blurred image, and each blur kernel, a corresponding blurred image, and a clear image constitute a set of sample data.

[0098] In this embodiment, the sample blur kernel corresponding to the sample blurred image is a perturbation blur kernel generated by adding random noise to the estimated blur kernel, wherein the number of perturbation blur kernels corresponding to the same blurred image is multiple, so that the potential deviation in the blur kernel can be simulated. Inputting each sample data into the model can enhance the model's adaptability to different blur types, and also introduces the diversity of the training process, which helps to improve the robustness of the model when facing complex situations in the real world.

[0099] In one embodiment, the initial non-blind deblurring model constructed is to expand the mathematical optimization model for optimizing non-blind deblurring using a semi-quadratic splitting method. The specific non-blind deconvolution model optimization model is as follows:

[0100]

[0101] Among them, φ(·) is called the regularization term, which is used to pre-apply a priori information to solve the ill-posedness of the result, * represents the convolution operation, and λ∈R determines the strength of the prior. K represents the blur kernel, Represents the square of the Frobenius norm of the matrix;

[0102] In order to achieve this optimization, an iterative framework based on semi-quadratic splitting is designed, which forms the overall framework of the network by sequentially connecting the optimization module and the noise reduction module in series. The specific iterative scheme is as follows: the overall framework of the network is formed by sequentially connecting the optimization module and the noise reduction module in series and alternating in series:

[0103]

[0104]

[0105] Among them, λ t is a hyperparameter, T is the total number of iterations, A p Represents a set of auxiliary variables.

[0106] The optimization module solves the above problem through an iterative optimization algorithm and outputs the optimized image X (t), the denoising module uses an end-to-end encoder-decoder network to solve the problem. The encoder part consists of three encoding units, each of which contains two convolutional layers. These convolutional layers use convolutional kernels of size 3x3, with a step size of 1 and a padding of 1 to effectively extract features without changing the spatial dimension of the feature map. The encoder part decoder consists of three decoding units, each of which contains two convolutional layers. These convolutional layers also use convolutional kernels of size 3x3, with a step size of 1 and a padding of 1. An upsampling unit is used for spatial upsampling. Between each encoding unit, the residual channel attention unit connects the input and output channels of each encoding unit, i.e., 64, 128, and 64 channels, respectively. Finally, the encoder-decoder outputs the denoised image solution

[0107] In this embodiment, the initial non-blind deblurring model is optimized and trained by sample data consisting of a sample blurred image, a sample blur kernel corresponding to the sample blurred image, and a sample clear image, so as to obtain a non-blind deblurring prediction model, which can provide a more accurate deblurring effect and significantly enhance the robustness of the model.

[0108] Embodiment 6

[0109] On the basis of the above embodiment, the step of constructing an initial non-blind deblurring model based on a machine learning algorithm, taking the blurred image and the blur kernel as inputs of the initial non-blind deblurring model, taking the clear image as output of the initial non-blind deblurring model, and optimizing and training the initial non-blind deblurring model using corresponding sample data to obtain a non-blind deblurring prediction model includes:

[0110] Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image;

[0111] Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model;

[0112] A non-blind deblurring prediction model is determined based on the adjusted parameters.

[0113] In this embodiment, in the process of optimizing the training of the model, that is, in the supervised learning stage of the network, the mean square error between the output of the network and the corresponding clear image is defined as the loss function, and the network parameters are gradually optimized through the back propagation algorithm. Specifically, the back propagation algorithm is used to minimize the loss function to optimize the parameters of the current model.

[0114] Specifically, the expression of the supervised loss is as follows:

[0115]

[0116] Among them, M represents the non-blind deconvolution model, the input of the non-blind deconvolution model includes the blurred image Y and the blur kernel K, X represents the clear image, Γ represents the data enhancement method, Represents the square of the Frobenius norm of the matrix.

[0117] In this embodiment, the model is optimized through the loss function, and the parameters of the current model are optimized based on the minimization of the loss function, thereby improving the prediction performance of the model.

[0118] Embodiment 7

[0119] Based on the above embodiment, the step of determining the non-blind defuzzification prediction model based on the adjusted parameters includes:

[0120] Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval;

[0121] Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold;

[0122] When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

[0123] In this embodiment, in the process of optimizing the model, the optimal model is determined by calculating the PSNR index (i.e., peak signal-to-noise ratio). Peak signal-to-noise ratio (PSNR) is an engineering term widely used in many fields, especially in the fields of image compression, video encoding, audio processing, etc., and it is often used as an important indicator to measure signal quality.

[0124] Specifically, during the training iteration process, the PSNR index (peak signal-to-noise ratio) of the model on the validation set is calculated regularly, and the PSNR value is checked to see if it decreases. The calculation formula of PSNR is as follows:

[0125]

[0126]

[0127] Where MAX represents the maximum value of the pixel, and m and n represent the length and width of the image, respectively. If PSNR continues to rise or is in a stable state, continue training. If PSNR drops significantly, the training should be terminated at this time, and the model version with the highest PSNR index in history should be selected as the final model.

[0128] Embodiment 8

[0129] Based on the above embodiments, this embodiment provides an application example. Figures 2 to 7 shown.

[0130] like Figures 2 to 3 As shown, the supervised learning non-blind deblurring method for blur kernel errors includes:

[0131] S1: Collect data. The collected data include blurred images, estimated blur kernels with errors, and corresponding clear images. The data are divided into training data set, validation data set, and test data set.

[0132] First, for each blurred image in the data set, the corresponding blur kernel is estimated using the traditional blur kernel estimation algorithm, including:

[0133] (1) Radon transform is used to extract straight line features from the blurred image, and then the direction and shape of the blur kernel are inferred. The core of this method lies in the application of Radon transform, which captures straight line features by calculating the integral projection of the image at different angles.

[0134] (2) First, perform Radon transform on the blurred image Y and calculate the projection R(θ, t) at each angle θ, which is expressed as:

[0135] R(θ,t)=∫Y(x,y)δ(xcosθ+ysinθ-t)dx dyR(θ,t)

[0136] Here, δ is the Dirac function used to integrate over the straight line xcosθ+ysinγ=t.

[0137] (3) By analyzing the results of Radon transform, observing the intensity distribution of the projection at a specific angle, it is helpful to infer the main direction of the blur kernel. Based on this information, a blur kernel K can be constructed, which is usually assumed to be a linear structure with a certain length in a specific direction. The blur kernel can be expressed as:

[0138] K(s)=λexp(-s 2 / σ 2 )

[0139] Here, s is the coordinate along the blur direction, and the parameters λ and σ are adjusted according to the Radon transform results to adapt to the strength and size of the blur kernel.

[0140] Each high-quality sharp image, the corresponding blurred image generated by applying the estimated blur kernel, and the estimated blur kernel itself, where the blur kernel may contain a certain amount of error. Each image pair (blurred image and sharp image) will be packaged together with its corresponding blur kernel. The entire dataset will be randomly split into three disjoint parts: training set, validation set, and test set. Ensure that the model does not overfit during the learning process and can effectively evaluate performance on unseen data.

[0141] S2. Construct a non-blind deblurring model. The model uses a semi-quadratic splitting method to expand the mathematical optimization model for optimizing non-blind deblurring. The specific non-blind deconvolution model optimization model is as follows:

[0142]

[0143] Among them, φ(·) is called the regularization term, which is used to pre-apply a priori information to solve the ill-posedness of the result, * represents the convolution operation, and λ∈R determines the strength of the prior. K represents the blur kernel, Represents the square of the Frobenius norm of the matrix;

[0144] In order to achieve this optimization, an iterative framework based on semi-quadratic splitting is designed, which forms the overall framework of the network by sequentially connecting the optimization module and the noise reduction module in series. The specific iterative scheme is as follows: the overall framework of the network is formed by sequentially connecting the optimization module and the noise reduction module in series and alternating in series:

[0145]

[0146]

[0147] Among them, λ t is a hyperparameter, T is the total number of iterations, A p Represents a set of auxiliary variables. Figure 4 shown.

[0148] The optimization module solves the above problem through an iterative optimization algorithm and outputs the optimized image X (t), the denoising module uses an end-to-end encoder-decoder network to solve the problem. The encoder part consists of three encoding units, each of which contains two convolutional layers. These convolutional layers use convolutional kernels of size 3x3, with a step size of 1 and a padding of 1 to effectively extract features without changing the spatial dimension of the feature map. The encoder part decoder consists of three decoding units, each of which contains two convolutional layers. These convolutional layers also use convolutional kernels of size 3x3, with a step size of 1 and a padding of 1. An upsampling unit is used for spatial upsampling. Between each encoding unit, the residual channel attention unit connects the input and output channels of each encoding unit, i.e., 64, 128, and 64 channels, respectively. Finally, the encoder-decoder outputs the denoised image solution like Figure 4 shown.

[0149] S3. Construct the training data of blur kernel perturbation for supervised training.

[0150] The effects caused by blur kernel errors may cause noticeable distortion or artifacts in the deconvolved image. Observations show that these artifacts caused by kernel errors have a certain degree of independence in the restored image. Based on this finding, data augmentation of the estimated blur kernel, including but not limited to adding random noise to the blur kernel, is an effective method to simulate the potential bias in the blur kernel. Such processing not only enhances the model's adaptability to different blur types, but also introduces diversity in the training process, which helps to improve the model's robustness when facing complex real-world situations. Specifically, applying Γ on the enhanced blur kernel can be expressed as: Γ( K) = K + n, where K is the original blur kernel and n is the noise added to the blur kernel.

[0151] In the supervised learning stage of the network, the mean square error between the network output and the corresponding clear image is defined as the loss function, and the network parameters are gradually optimized through the back propagation algorithm. The back propagation algorithm is used to minimize the loss function to optimize the parameters of the current model. The expression of the supervised loss is as follows:

[0152]

[0153] Among them, M represents the non-blind deconvolution model, the input of the non-blind deconvolution model includes the blurred image Y and the blur kernel K, X represents the clear image, Γ represents the data enhancement method, Represents the square of the Frobenius norm of the matrix.

[0154] S4. Judge the optimized model on the validation set.

[0155] During the training iteration process, the PSNR index of the model on the validation set is calculated regularly, and the PSNR value is checked to see if it decreases. The calculation formula of PSNR is as follows:

[0156]

[0157]

[0158] Where MAX represents the maximum value of the pixel, and m and n represent the length and width of the image, respectively. If the PSNR continues to rise or is stable, continue training. If the PSNR drops significantly, the training should be terminated and the model version with the highest PSNR index in history should be selected as the final model.

[0159] S5. Integrate the outputs of different blur kernel perturbations on the test machine to obtain a clear image after deblurring.

[0160] By performing data augmentation and adding random noise to the estimated blur kernel corresponding to each blurred image in the test set to generate N perturbed blur kernels, and then using these blur kernels to perform non-blind deblurring on the blurred image, multiple deblurred result images can be obtained. These deblurred result images are then integrated to obtain the final predicted deblurred image, expressed as follows:

[0161]

[0162] Where Γ is the same as S3: Γ( K) = K + n, where K is the original blur kernel, n is the noise added to the blur kernel, M is the non-blind deblurring prediction model, and Y is the blurred image. Represents the square of the Frobenius norm of the matrix. This strategy of enhancing the blur kernel and integrating the deblurring results can further tap the potential of the test data, make the deblurring model more robust, and ultimately achieve better deblurring results. At the same time, the deblurring performance of this strategy can be optimized by adjusting the enhancement, perturbation method and integration algorithm.

[0163] Figure 5 Schematic diagram of the comparison of the deblurred images output by the models with and without perturbation training. Figure 5 It can be seen that the noise and artifacts of the strategy image of the predicted image generated by blur kernel perturbation are effectively suppressed.

[0164] Figure 6 as well as Figure 7 The following is a visual effect diagram of the blurred image input based on the non-blind deblurring model and the corresponding clear image output. It can be seen that the noise and artifacts of the output clear image are effectively suppressed, and the details of the image can be seen more clearly.

[0165] This embodiment uses a perturbed blur kernel after random perturbation to input into a non-blind deblurring prediction model to obtain multiple deblurred images, and each deblurred image has a certain degree of independence. By integrating each deblurred image, it is possible to compensate for the defects of individual results, effectively reduce the artifacts caused by the error in the blur kernel, and then reduce the influence of the kernel error, so that the output result is closer to the true value. In addition, the model is trained based on different perturbation blur kernels, which significantly enhances the robustness and can effectively adapt to the uncertainty of the blur kernel, especially for blurred images from complex real scenes, and can provide a more accurate deblurring effect. Compared with the traditional deblurring method in the prior art that relies on an accurate blur kernel, this embodiment significantly reduces the reliance on accurate blur kernel estimation, and there is no need to estimate the blur kernel through repeated iterations. The entire deblurring process becomes more efficient and direct.

[0166] Embodiment 9

[0167] Based on the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the following steps:

[0168] Acquire a blurred image and a blur kernel corresponding to the blurred image;

[0169] Performing data enhancement on the blur kernel to obtain a disturbed blur kernel;

[0170] Inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image;

[0171] The deblurred images corresponding to different disturbance blur kernels of the blurred image are integrated to obtain a predicted clear image.

[0172] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0173] Random noise is added to the blur kernel to obtain a disturbed blur kernel.

[0174] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0175] Get the blurred image;

[0176] Extracting straight line features from the blurred image using integral transform;

[0177] A blur kernel corresponding to the blurred image is determined based on the straight line features.

[0178] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0179] The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

[0180] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0181] Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image;

[0182] An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

[0183] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0184] Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image;

[0185] Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model;

[0186] A non-blind deblurring prediction model is determined based on the adjusted parameters.

[0187] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0188] Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval;

[0189] Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold;

[0190] When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

[0191] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0192] Acquire a blurred image and a blur kernel corresponding to the blurred image;

[0193] Performing data enhancement on the blur kernel to obtain a disturbed blur kernel;

[0194] Inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image;

[0195] The deblurred images corresponding to different disturbance blur kernels of the blurred image are integrated to obtain a predicted clear image.

[0196] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0197] Random noise is added to the blur kernel to obtain a disturbed blur kernel.

[0198] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0199] Get the blurred image;

[0200] Extracting straight line features from the blurred image using integral transform;

[0201] A blur kernel corresponding to the blurred image is determined based on the straight line features.

[0202] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0203] The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

[0204] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0205] Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image;

[0206] An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

[0207] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0208] Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image;

[0209] Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model;

[0210] A non-blind deblurring prediction model is determined based on the adjusted parameters.

[0211] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0212] Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval;

[0213] Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold;

[0214] When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

[0215] In some implementations of this embodiment, a computer program product is provided, including a computer program / instruction, and when the computer program is executed by a processor, the following steps are implemented:

[0216] Acquire a blurred image and a blur kernel corresponding to the blurred image;

[0217] Performing data enhancement on the blur kernel to obtain a disturbed blur kernel;

[0218] Inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image;

[0219] The deblurred images corresponding to different disturbance blur kernels of the blurred image are integrated to obtain a predicted clear image.

[0220] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0221] Random noise is added to the blur kernel to obtain a disturbed blur kernel.

[0222] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0223] Get the blurred image;

[0224] Extracting straight line features from the blurred image using integral transform;

[0225] A blur kernel corresponding to the blurred image is determined based on the straight line features.

[0226] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0227] The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

[0228] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0229] Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image;

[0230] An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

[0231] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0232] Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image;

[0233] Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model;

[0234] A non-blind deblurring prediction model is determined based on the adjusted parameters.

[0235] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0236] Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval;

[0237] Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold;

[0238] When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

[0239] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0240] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0241] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0242] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0243] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0244] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0245] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0246] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0247] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A non-blind deblurring method, characterized in that: include: Acquire a blurred image and a blur kernel corresponding to the blurred image; Performing data enhancement on the blur kernel to obtain a disturbed blur kernel; Inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image; The deblurred images corresponding to different disturbance blur kernels of the blurred image are integrated to obtain a predicted clear image.

2. The non-blind deblurring method according to claim 1, characterized in that: The step of performing data enhancement on the blur kernel to obtain a disturbed blur kernel comprises: Random noise is added to the blur kernel to obtain a disturbed blur kernel.

3. The non-blind deblurring method according to claim 1, characterized in that: The step of obtaining a blurred image and a blur kernel corresponding to the blurred image comprises: Get the blurred image; Extracting straight line features from the blurred image using integral transform; A blur kernel corresponding to the blurred image is determined based on the straight line features.

4. The non-blind deblurring method according to claim 1, characterized in that: The step of integrating the deblurred images corresponding to different disturbance blur kernels of the blurred images to obtain a predicted clear image comprises: The pixel averages of the deblurred images corresponding to different disturbance blur kernels of the blurred image are calculated to obtain a predicted clear image.

5. The non-blind deblurring method according to claim 1, characterized in that: Before the step of inputting the blurred image and the perturbation blur kernel into a non-blind deblurring prediction model for deblurring prediction to obtain a deblurred image, the step further includes: Acquire sample data consisting of a sample blurred image, a sample blurred kernel corresponding to the sample blurred image, and a sample clear image; An initial non-blind deblurring model is constructed based on a machine learning algorithm, the blurred image and the blur kernel are used as inputs of the initial non-blind deblurring model, the clear image is used as output of the initial non-blind deblurring model, and the initial non-blind deblurring model is optimized and trained using corresponding sample data to obtain a non-blind deblurring prediction model.

6. The non-blind deblurring method according to claim 5, characterized in that: The steps of constructing an initial non-blind deblurring model based on a machine learning algorithm, taking the blurred image and the blur kernel as inputs of the initial non-blind deblurring model, taking the clear image as output of the initial non-blind deblurring model, and optimizing and training the initial non-blind deblurring model using corresponding sample data to obtain a non-blind deblurring prediction model include: Constructing a loss function of the initial non-blind deblurring model according to a relative error between a prediction result of a deblurred image output by the initial non-blind deblurring model and the sample clear image; Iterating the initial non-blind deblurring model based on minimization of the loss function, and adjusting parameters of the initial non-blind deblurring model; A non-blind deblurring prediction model is determined based on the adjusted parameters.

7. The non-blind deblurring method according to claim 6, characterized in that: The step of determining the non-blind defuzzification prediction model based on the adjusted parameters comprises: Acquire a peak signal-to-noise ratio between a prediction result of a deblurred image output by a candidate non-blind deblurring model determined based on current parameters and the sample clear image within a preset time interval; Determining whether a difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of a previous time interval is less than a preset threshold; When the difference between the peak signal-to-noise ratio and the peak signal-to-noise ratio of the previous time interval is less than a preset threshold, the candidate non-blind deblurring model when the peak signal-to-noise ratio is maximum is selected as the non-blind deblurring prediction model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images