Zero sample image deblurring method and system based on geometric prior and conditional diffusion

Abstract

The present application relates to the technical field of computer vision, and provides a zero-shot image deblurring method and system based on geometric prior and conditional diffusion, which restores the to-be-processed image to obtain an initial intermediate image, generates a geometric prior map based on the initial intermediate image, and finally inputs the to-be-processed image, the initial intermediate image and the geometric prior map into a conditional diffusion model to obtain a deblurred image corresponding to the to-be-processed image output by the conditional diffusion model. The method uses the initial intermediate image obtained by preliminarily restoring the to-be-processed image to explicitly construct a geometric prior map as a geometric guide signal, which can effectively avoid the misleading problem caused by directly obtaining the geometric prior from the to-be-processed image and break the closed loop of "biased prior - wrong recovery". Moreover, the method does not need to retrain the model or data adaptation for specific blur types, can have zero-shot generalization ability in unknown scenes, and has strong universality and practicality.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a zero-sample image deblurring method and system based on geometric priors and conditional diffusion. Background Technology

[0002] Images, as a key carrier of visual information, play a vital role in the information age and have become one of the main ways for people to communicate and exchange information in their daily lives. However, in practical applications, due to equipment limitations, environmental factors, and shooting conditions, digital images often suffer from various quality degradation problems during the acquisition process. These include motion blur caused by equipment shake or object movement, noise interference caused by sensor noise or harsh environments, and natural weather factors such as rain and snow. These problems not only affect the visual quality and aesthetics of the images but also reduce the accurate transmission of information.

[0003] Image restoration technology aims to recover high-quality, clear images from degraded images, which is crucial for improving visual experience and enhancing the performance of downstream computer vision tasks such as object detection, face recognition, and medical image analysis. Existing image restoration methods mainly face three core challenges:

[0004] (1) Prior misleading structural bias: Some methods introduce geometric priors to regularize the recovery of the solution space. However, due to the severe loss of high-frequency information, the obtained prior estimates are often biased. Therefore, incorporating geometric priors into the regularization term may lead to the recovered image falling into a suboptimal solution.

[0005] (2) Average image effect of deterministic models: The current mainstream end-to-end network model models deblurring as a function, and its training objective is usually to minimize L1 / L2 pixel loss. According to the expected risk minimization theory, the optimal solution under L2 loss tends to the conditional expectation, which makes the model output tend to the image average of all possible potential solutions, lose random high-frequency texture, and significantly reduce the visual perception quality of the image.

[0006] (3) Existing methods usually require customized training or fine-tuning on training sets for specific fuzzy types, which limits their generalization ability and makes it difficult to achieve "ready to use and deploy" in real complex scenarios. Summary of the Invention

[0007] This invention provides a zero-sample image deblurring method and system based on geometric priors and conditional diffusion, in order to overcome the deficiencies in the prior art.

[0008] This invention provides a zero-shot image deblurring method based on geometric priors and conditional diffusion, comprising:

[0009] Obtain the image to be processed;

[0010] The image to be processed is restored to obtain an initial intermediate image, and a geometric prior map is generated based on the initial intermediate image;

[0011] The image to be processed, the initial intermediate image, and the geometric prior image are input into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model.

[0012] According to the present invention, a zero-sample image deblurring method based on geometric prior and conditional diffusion is provided, wherein generating a geometric prior map based on the initial intermediate image includes:

[0013] Using the initial intermediate image as the current image, monocular depth estimation is performed on the image to be processed and the current image respectively to determine the first depth map of the image to be processed and the second depth map of the current image;

[0014] Calculate the overall difference between the first depth map of the image to be processed and the second depth map of the current image, and determine an adaptive threshold based on the overall difference;

[0015] If the overall difference is greater than the adaptive threshold, then based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, the current image is guided to be updated to obtain the updated image;

[0016] The updated image is used as the current image, and the above process is iteratively repeated until the overall difference is less than or equal to the adaptive threshold or a preset number of iterations is reached. The second depth map of the current image is then used as the geometric prior map.

[0017] According to the present invention, a zero-sample image deblurring method based on geometric prior and conditional diffusion is provided, wherein the current image is guided to be updated based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image to obtain an updated image, including:

[0018] Based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, a depth estimation confidence map is calculated.

[0019] Based on the depth estimation confidence map, target regions in the current image with confidence levels below a set threshold are identified. Within these target regions, the current image is guided to be updated based on the second depth map of the current image, resulting in the updated image.

[0020] According to the present invention, a zero-shot image deblurring method based on geometric prior and conditional diffusion is provided, wherein the method guides the updating of the current image based on a second depth map of the current image to obtain the updated image, specifically including:

[0021] The second depth map of the current image and the current image are input into the guiding filter to obtain the output result of the guiding filter;

[0022] Based on the output, the current image, and the depth estimation confidence map, the updated image is determined.

[0023] According to the present invention, a zero-sample image deblurring method based on geometric prior and conditional diffusion is provided, wherein determining an adaptive threshold based on the overall difference includes:

[0024] Calculate the mean and standard deviation of the overall differences;

[0025] The adaptive threshold is determined based on the mean and the standard deviation.

[0026] According to the present invention, a zero-shot image deblurring method based on geometric prior and conditional diffusion is provided, wherein calculating a depth estimation confidence map based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image includes:

[0027] Based on the second depth map of the current image and the second depth map of the previous image, calculate the first depth consistency between the current image and the previous image;

[0028] Based on the image to be processed and the second depth map of the current image, calculate the second depth consistency between the current image and the image to be processed;

[0029] The depth estimation confidence map is calculated based on the first depth consistency and the second depth consistency.

[0030] According to the present invention, a zero-shot image deblurring method based on geometric prior and conditional diffusion is provided, wherein the conditional diffusion model includes a control network, a pre-trained model encoder, an encoding and decoding module, and a denoising diffusion implicit model.

[0031] The step of inputting the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model includes:

[0032] The geometric prior map is input into the control network to obtain the geometric control features output by the control network;

[0033] The initial intermediate image is input into the pre-trained model encoder to obtain the perceptual content features output by the pre-trained model encoder;

[0034] The image to be processed, the geometric control features, and the perceptual content features are input into the encoding and decoding module to obtain the noise estimate output by the encoding and decoding module;

[0035] The noise estimate is input into the denoising diffusion implicit model to obtain the deblurred image output by the denoising diffusion implicit model.

[0036] This invention also provides a zero-shot image deblurring system based on geometric priors and conditional diffusion, comprising:

[0037] The image acquisition module is used to acquire the image to be processed.

[0038] The prior image generation module is used to recover the image to be processed to obtain an initial intermediate image, and generate a geometric prior image based on the initial intermediate image;

[0039] The image restoration module is used to input the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the zero-sample image deblurring method based on geometric prior and conditional diffusion as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the zero-sample image deblurring method based on geometric prior and conditional diffusion as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the zero-sample image deblurring method based on geometric prior and conditional diffusion as described above.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention provides a zero-shot image deblurring method and system based on geometric prior and conditional diffusion. First, the image to be processed is acquired. Then, the image is restored to obtain an initial intermediate image, and a geometric prior map is generated based on this initial intermediate image. Finally, the image to be processed, the initial intermediate image, and the geometric prior map are input into a conditional diffusion model to obtain the deblurred image corresponding to the image to be processed, output by the conditional diffusion model. This method utilizes the initial intermediate image obtained from the preliminary restoration of the image to be processed to explicitly construct a geometric prior map as a geometric guiding signal. This effectively avoids the misleading problem caused by directly obtaining geometric priors from the image to be processed, breaking the closed loop of "biased prior—erroneous restoration." Moreover, this method does not require retraining the model or data adaptation for specific blur types, and it has zero-shot generalization ability in unknown scenarios, exhibiting strong versatility and practicality. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0046] Figure 1 This is one of the flowcharts of the zero-shot image deblurring method based on geometric prior and conditional diffusion provided by the present invention;

[0047] Figure 2 This is the second flowchart of the zero-sample image deblurring method based on geometric prior and conditional diffusion provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the zero-sample image deblurring system based on geometric prior and conditional diffusion provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Existing methods mainly face three core challenges:

[0052] (1) Prior misleading structural bias

[0053] Some methods introduce geometric priors to regularize and recover the solution space, for example:

[0054] ;

[0055] Where I is the recovered image variable, To minimize the function of variables, This represents the value of I when A reaches its minimum value. It is the input image to be processed. This is the final restored image. For data fidelity items, For prior regularization terms, For coefficients, Geometric priors. In blind deblurring scenarios, As a geometric prior, it is usually estimated from the fuzzy image, i.e. ,in This is a geometric prior estimation method. Due to the severe loss of high-frequency information, the estimated geometric prior is often biased. Including it in the regularization term may actually cause the reconstructed image to fall into a suboptimal solution.

[0056] (2) Average image effect of deterministic models

[0057] Current mainstream end-to-end network models model defuzzification as a function. , These are real, clear images, and their training objective is typically to minimize L1 / L2 pixel loss. Based on the expected risk minimization theory, the optimal solution under L2 loss tends towards conditional expectation. This causes the model output to tend to the image average of all possible potential solutions, resulting in the loss of random high-frequency textures and a significant decrease in the visual perception quality of the image.

[0058] While some works have improved perceptual performance by introducing conditional generative models (such as GANs or Diffusion), this is still not ideal when the guiding signal simultaneously contains... (Providing low-frequency / semantic anchors to constrain the diffusion model from changing the main content) and its derived geometric priors. When guided by structure, the model receives contradictory and heterogeneous combinations of conditions. This leads to "decision conflicts" or "phantomization" in the generation process, potentially resulting in a significant deviation from the realistic scene (lighting, texture, identity).

[0059] ;

[0060] in, This represents the conditional probability distribution of the samples generated by the model, that is, the output distribution of the model under given conditions. Indicates will As a given condition Indicates will As a given condition The true posterior distribution, i.e., the true sharp image in the distribution. As a conditional distribution under given conditions. Inconsistency between the two means that "the model will generate results that deviate from the true posterior when the conditions are inconsistent or there is a biased prior".

[0061] (3) Existing methods usually require customized training or fine-tuning on training sets for specific fuzzy types, which limits their generalization ability and makes it difficult to achieve "ready to use and deploy" in real complex scenarios.

[0062] Based on this, this embodiment of the invention provides a zero-sample image deblurring method based on geometric priors and conditional diffusion.

[0063] Figure 1 This is a flowchart illustrating a zero-shot image deblurring method based on geometric priors and conditional diffusion provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0064] S1, Obtain the image to be processed;

[0065] S2, the image to be processed is restored to obtain an initial intermediate image, and a geometric prior map is generated based on the initial intermediate image;

[0066] S3, input the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model.

[0067] Specifically, the zero-shot image deblurring method based on geometric prior and conditional diffusion provided in this embodiment of the invention is executed by a zero-shot image deblurring system based on geometric prior and conditional diffusion. This system can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0068] First, execute step S1 to acquire the image to be processed. The image to be processed refers to a low-quality blurry image with degradation defects that needs to be restored to a clear image. The image to be processed may contain motion blur caused by device shaking or object movement, noise interference caused by sensor noise or harsh environment, rain or snow occlusion caused by natural weather factors, etc., which are not specifically limited here.

[0069] Then, step S2 is executed, using a pre-trained deep image restoration network to restore the image to be processed, obtaining an initial intermediate image. Here, the deep image restoration network can be a Nonlinear Activation Free Network (NAFNet). Through the deep image restoration network, the image to be processed can be mapped to an approximate point on the distribution of sharp images, obtaining an initial intermediate image, thereby enhancing the discernibility of image structure and alleviating the inadequacy of subsequent geometric estimation.

[0070] The mapping process implemented by depth image restoration networks can be defined as follows:

[0071] ;

[0072] in, Let H be the image to be processed, H be the height of the image to be processed, W be the width of the image to be processed, and the number of channels of the image to be processed is 3. This is the initial intermediate image. For the network parameters of the depth image recovery network, To initially restore the network parameters, For true and clear images, It is an L1 norm. The expectation is for a truly clear image.

[0073] By pre-training the initial reconstruction network, a deep image reconstruction network can be obtained. The initial intermediate image output by the deep image reconstruction network exhibits significantly better structural fidelity in high-frequency regions than the original image to be processed. Therefore, the structural enhancement properties of the initial intermediate image can be utilized to generate a geometric prior map with high confidence. Completely replaces traditional methods for processing images Direct estimation of geometric priors The introduced sources of deviation.

[0074] Here, the initial intermediate image can be input into the depth estimation network, which then outputs a geometric prior map. There are:

[0075] ;

[0076] in, This represents a depth estimation network.

[0077] In this embodiment of the invention, the depth estimation network can be the open-source DepthAnythingV2. The resulting geometric prior map is a refined depth map.

[0078] Finally, step S3 is executed, where the image to be processed, the initial intermediate image, and the geometric prior image are input into the conditional diffusion model. The conditional diffusion model uses the initial intermediate image and the geometric prior image to determine the conditional signal, and then uses the conditional signal to process the image to be processed, thereby achieving the synthesis of a high-fidelity deblurred image.

[0079] In this context, the initial intermediate image in the conditional signal can be used for content anchoring, and the geometric prior image can be used for structural guidance. The conditional diffusion model can use the open-source Stable Diffusion v2 (SD2) model as its core framework, utilizing the conditional signal to implement the conditional diffusion sampling process, and obtain and output the deblurred image corresponding to the image to be processed.

[0080] The zero-shot image deblurring method based on geometric prior and conditional diffusion provided in this invention first acquires the image to be processed; then, the image to be processed is restored to obtain an initial intermediate image, and a geometric prior map is generated based on the initial intermediate image; finally, the image to be processed, the initial intermediate image, and the geometric prior map are input into a conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model. This method utilizes the initial intermediate image obtained from the preliminary restoration of the image to be processed to explicitly construct a geometric prior map as a geometric guiding signal, effectively avoiding the misleading problem caused by directly obtaining geometric priors from the image to be processed, breaking the closed loop of "biased prior—erroneous restoration". Moreover, this method does not require retraining the model or data adaptation for specific blur types, and has zero-shot generalization ability in unknown scenarios, exhibiting strong versatility and practicality.

[0081] Based on the above embodiments, generating a geometric prior image based on the initial intermediate image includes:

[0082] Using the initial intermediate image as the current image, monocular depth estimation is performed on the image to be processed and the current image respectively to determine the first depth map of the image to be processed and the second depth map of the current image;

[0083] Calculate the overall difference between the first depth map of the image to be processed and the second depth map of the current image, and determine an adaptive threshold based on the overall difference;

[0084] If the overall difference is greater than the adaptive threshold, then based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, the current image is guided to be updated to obtain the updated image;

[0085] The updated image is used as the current image, and the above process is iteratively repeated until the overall difference is less than or equal to the adaptive threshold or a preset number of iterations is reached. The second depth map of the current image is then used as the geometric prior map.

[0086] Specifically, the process of generating the geometric prior map is an iterative process. The input of the first iteration includes the image to be processed and the initial intermediate image. The input of each subsequent iteration includes the image to be processed and the output of the previous iteration. The output of the previous iteration is obtained by updating the intermediate image input of the previous iteration. Therefore, in each iteration, either the initial intermediate image or the output of the previous iteration can be used as the current image.

[0087] Taking the first iteration as an example, the initial intermediate image can be used as the current image. The image to be processed and the current image are then input into the depth estimation network. The depth estimation network performs monocular depth estimation on the image to be processed and the current image respectively, thus determining the first depth map of the image to be processed. and the second depth map of the current image Where k is the iteration round, Let k be the current image in the k-th round. When the current image is the initial intermediate image, k=1.

[0088] To ensure prior confidence, a prior bias assessment index is introduced:

[0089] ;

[0090] in, This is a priori bias assessment index used to represent the overall difference between the first depth map and the second depth map. It is an L2 norm.

[0091] Subsequently, the mean and standard deviation can be calculated using the overall differences, and the adaptive threshold can be determined using the following formula based on the mean and standard deviation:

[0092] ;

[0093] in, and These represent the mean and standard deviation of the overall difference, respectively. This is the threshold coefficient, which can be set as needed, for example, it can be set to 2.

[0094] Subsequently, the overall difference is compared with the adaptive threshold. If the overall difference is greater than the adaptive threshold, the current image can be guided to be updated using the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image to obtain the updated image. That is, by comparing the second depth map of the current image and the second depth map of the previous image of the current image with the image to be processed, the target area in the current image that needs to be updated is determined and updated.

[0095] Continue using the updated image as the current image and iterate through the above process until the overall difference is less than or equal to the adaptive threshold or the preset number of iterations is reached. Then, use the second depth map of the current image as the geometric prior map.

[0096] In this embodiment of the invention, during the process of determining the geometric prior image, the intermediate image is updated through overall difference feedback to form a structural closed loop with high convergence and low drift risk, thereby eliminating structural misleading.

[0097] Based on the above embodiments, the step of guiding the update of the current image based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image to obtain the updated image includes:

[0098] Based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, a depth estimation confidence map is calculated.

[0099] Based on the depth estimation confidence map, target regions in the current image with confidence levels below a set threshold are identified. Within these target regions, the current image is guided to be updated based on the second depth map of the current image, resulting in the updated image.

[0100] Specifically, when performing guided updates on the current image, the depth estimation confidence map can be calculated using the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, using the following formula:

[0101] ;

[0102] in, Let x be the confidence level at pixel x in the depth estimation confidence map for the k-th round. This refers to the value at pixel x in the second depth map of the current image. This refers to the value at pixel x in the second depth map of the previous image of the current image. Let x be the value of pixel x in the image to be processed. All of these are smoothing parameters, all greater than 0, used to control the rate of exponential decay.

[0103] This refers to the first depth consistency between the current image and the previous image, i.e., structural consistency. Large changes indicate instability or unreliability.

[0104] The second depth consistency, or content consistency, between the current image and the image to be processed is used to determine whether the iteration deviates from the image to be processed in terms of trend.

[0105] Subsequently, the depth estimation confidence map can be used to identify target regions in the current image with confidence levels below a set threshold.

[0106] ;

[0107] in, To set the threshold, x represents the pixel position. Let x represent the confidence level at pixel x in the depth estimation confidence map for the k-th round. Let be the target region for the kth round.

[0108] Furthermore, in the target area Selective reflux is triggered, that is, by utilizing right Perform a local guided update to obtain the updated image. This is the output of the current iteration and the intermediate image used as the input for the next iteration.

[0109] In this embodiment of the invention, when the geometric prior is locally unreliable, a pixel-level confidence-driven selective reflow mechanism is provided to avoid structural illusions caused by unfavorable priors. This effectively solves the core problems in traditional blind deblurring methods, such as prior bias accumulation, perceptual degradation, and content drift where the generated content is inconsistent with the original image information. Moreover, by performing structural consistency reconstruction and content consistency reconstruction through heterogeneous conditional combinations, the clarity of the deblurred image can be improved.

[0110] Based on the above embodiments, the step of guiding the update of the current image based on the second depth map of the current image to obtain the updated image specifically includes:

[0111] The second depth map of the current image and the current image are input into the guiding filter to obtain the output result of the guiding filter;

[0112] Based on the output, the current image, and the depth estimation confidence map, the updated image is determined.

[0113] Specifically, when performing guided updates on the current image, the second depth map of the current image and the current image itself can be input into the guided filter first. The current image is filtered using the second depth map of the current image through a guided filter to obtain the output result.

[0114] Subsequently, a depth estimation confidence map is used to perform a weighted summation of the output and the current image to determine the updated image. That is:

[0115] ;

[0116] in, Let x be the value of pixel x in the image after the k-th update.

[0117] In this embodiment of the invention, guided updates are achieved through a guided filter, enabling accurate and rapid image updates.

[0118] Based on the above embodiments, the conditional diffusion model includes a control network, a pre-trained model encoder, an encoding / decoding module, and a denoising diffusion implicit model;

[0119] The step of inputting the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model includes:

[0120] The geometric prior map is input into the control network to obtain the geometric control features output by the control network;

[0121] The initial intermediate image is input into the pre-trained model encoder to obtain the perceptual content features output by the pre-trained model encoder;

[0122] The image to be processed, the geometric control features, and the perceptual content features are input into the encoding and decoding module to obtain the noise estimate output by the encoding and decoding module;

[0123] The noise estimate is input into the denoising diffusion implicit model to obtain the deblurred image output by the denoising diffusion implicit model.

[0124] Specifically, in this embodiment of the invention, the conditional diffusion model may include a ControlNet, a pre-trained model encoder, an encoder-decoder module, and a denoising diffusion implicit model. The pre-trained model encoder may be the encoder of a Contrastive Language-Image Pre-training (CLIP) model. The encoder-decoder module may include the encoder and decoder from U-Net.

[0125] By inputting the geometric prior map into the control network, the control network can extract and output geometric control features, i.e.: , For geometric control features, For network control.

[0126] The initial intermediate image is input into the pre-trained model encoder, which can extract and output perceptual content features, i.e.: , To perceive content features, This is the encoder for the pre-trained model.

[0127] The image to be processed, geometric control features, and perceived content features are input into the encoding / decoding module, which then performs noise prediction to obtain a noise estimate. .in, These are the network parameters for the encoding / decoding module.

[0128] Finally, the noise estimate is input into the Denoising Diffusion Implicit Model (DDIM), which performs a diffusion process and a backsampling process to obtain the deblurred image.

[0129] The diffusion process follows a standard forward Gaussian noise addition pattern, which can be expressed as:

[0130] ;

[0131] in, For a clear image, The latent variables of the noisy image at time step t are... The result after adding t-step noise. Indicates that in a given Under the conditions, after undergoing a t-step forward diffusion process, the following was obtained. The probability distribution. To preserve the coefficients of the original signal at time step t, The cumulative product coefficients from time step 0 to t. Let be the identity matrix with data dimension d.

[0132] The backsampling process can be represented as:

[0133] ;

[0134] in, For a clear image to be predicted, This is the denoised image at time step t-1. These are conditional signals, including geometric control features and perceived content features. This is for noise estimation.

[0135] In this embodiment of the invention, by using the DDIM sampler, the number of sampling steps can be significantly reduced while preserving image clarity and detail texture, thus balancing deployment speed and quality.

[0136] like Figure 2 As shown, the zero-shot image deblurring method based on geometric prior and conditional diffusion provided in this embodiment of the invention specifically includes:

[0137] Obtain the image to be processed;

[0138] The image to be processed is input into a pre-trained deep image recovery network to obtain an initial intermediate image;

[0139] The initial intermediate image is input into the depth estimation network, which performs bias estimation, calculates the overall difference, and determines whether the overall difference is greater than the adaptive threshold. If it is greater, iterative optimization is performed, the depth estimation confidence map is calculated, and selective backflow is performed using the depth estimation confidence map to optimize the intermediate image and obtain the updated image. If the overall difference is less than or equal to the adaptive threshold, a geometric prior map is generated.

[0140] The geometric prior image is input into the conditional diffusion model, and the geometric control features in the geometric prior image are extracted by the control network. The perceptual content features of the initial intermediate image are extracted by the pre-trained model encoder. The image to be processed, the geometric control features, and the perceptual content features are input into the U-Net encoder-decoder module, and the encoder-decoder module outputs the noise estimate. The image is then denoised by the denoising diffusion implicit model to obtain the deblurred image.

[0141] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a zero-shot image deblurring system based on geometric priors and conditional diffusion, comprising:

[0142] Image acquisition module 31 is used to acquire the image to be processed;

[0143] The prior image generation module 32 is used to recover the image to be processed to obtain an initial intermediate image, and generate a geometric prior image based on the initial intermediate image;

[0144] The image restoration module 33 is used to input the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model.

[0145] Based on the above embodiments, the zero-sample image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention, wherein the prior map generation module is specifically used for:

[0146] Using the initial intermediate image as the current image, monocular depth estimation is performed on the image to be processed and the current image respectively to determine the first depth map of the image to be processed and the second depth map of the current image;

[0147] Calculate the overall difference between the first depth map of the image to be processed and the second depth map of the current image, and determine an adaptive threshold based on the overall difference;

[0148] If the overall difference is greater than the adaptive threshold, then based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, the current image is guided to be updated to obtain the updated image;

[0149] The updated image is used as the current image, and the above process is iteratively repeated until the overall difference is less than or equal to the adaptive threshold or a preset number of iterations is reached. The second depth map of the current image is then used as the geometric prior map.

[0150] Based on the above embodiments, the zero-sample image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention, wherein the prior map generation module is specifically used for:

[0151] Based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, a depth estimation confidence map is calculated.

[0152] Based on the depth estimation confidence map, target regions in the current image with confidence levels below a set threshold are identified. Within these target regions, the current image is guided to be updated based on the second depth map of the current image, resulting in the updated image.

[0153] Based on the above embodiments, the zero-sample image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention, wherein the prior map generation module is specifically used for:

[0154] The second depth map of the current image and the current image are input into the guiding filter to obtain the output result of the guiding filter;

[0155] Based on the output, the current image, and the depth estimation confidence map, the updated image is determined.

[0156] Based on the above embodiments, the zero-sample image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention, wherein the prior map generation module is specifically used for:

[0157] Calculate the mean and standard deviation of the overall differences;

[0158] The adaptive threshold is determined based on the mean and the standard deviation.

[0159] Based on the above embodiments, the zero-sample image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention, wherein the prior map generation module is specifically used for:

[0160] Based on the second depth map of the current image and the second depth map of the previous image, calculate the first depth consistency between the current image and the previous image;

[0161] Based on the image to be processed and the second depth map of the current image, calculate the second depth consistency between the current image and the image to be processed;

[0162] The depth estimation confidence map is calculated based on the first depth consistency and the second depth consistency.

[0163] Based on the above embodiments, the zero-shot image deblurring system based on geometric prior and conditional diffusion provided in this embodiment of the invention includes a control network, a pre-trained model encoder, an encoding and decoding module, and a denoising diffusion implicit model.

[0164] The image restoration module is specifically used for:

[0165] The geometric prior map is input into the control network to obtain the geometric control features output by the control network;

[0166] The initial intermediate image is input into the pre-trained model encoder to obtain the perceptual content features output by the pre-trained model encoder;

[0167] The image to be processed, the geometric control features, and the perceptual content features are input into the encoding and decoding module to obtain the noise estimate output by the encoding and decoding module;

[0168] The noise estimate is input into the denoising diffusion implicit model to obtain the deblurred image output by the denoising diffusion implicit model.

[0169] Specifically, the functions of each module in the zero-sample image deblurring system based on geometric priors and conditional diffusion provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-mentioned method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0170] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the zero-sample image deblurring method based on geometric prior and conditional diffusion provided in the above embodiments.

[0171] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the zero-sample image deblurring method based on geometric prior and conditional diffusion provided in the above embodiments.

[0173] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the zero-sample image deblurring method based on geometric prior and conditional diffusion provided in the above embodiments.

[0174] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-shot image deblurring method based on geometric priors and conditional diffusion, characterized in that, include: Obtain the image to be processed; The image to be processed is restored to obtain an initial intermediate image, and a geometric prior map is generated based on the initial intermediate image; The image to be processed, the initial intermediate image, and the geometric prior image are input into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model. The step of generating a geometric prior map based on the initial intermediate image includes: Using the initial intermediate image as the current image, monocular depth estimation is performed on the image to be processed and the current image respectively to determine the first depth map of the image to be processed and the second depth map of the current image; Calculate the overall difference between the first depth map of the image to be processed and the second depth map of the current image, and determine an adaptive threshold based on the overall difference; If the overall difference is greater than the adaptive threshold, then based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, the current image is guided to be updated to obtain the updated image; The updated image is used as the current image, and the above process is iterated until the overall difference is less than or equal to the adaptive threshold or the preset iteration round is reached. The second depth map of the current image is used as the geometric prior map. The conditional diffusion model includes a control network, a pre-trained model encoder, an encoding / decoding module, and a denoising diffusion implicit model. The step of inputting the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model includes: The geometric prior map is input into the control network to obtain the geometric control features output by the control network; The initial intermediate image is input into the pre-trained model encoder to obtain the perceptual content features output by the pre-trained model encoder; The image to be processed, the geometric control features, and the perceptual content features are input into the encoding and decoding module to obtain the noise estimate output by the encoding and decoding module; The noise estimate is input into the denoising diffusion implicit model to obtain the deblurred image output by the denoising diffusion implicit model.

2. The zero-shot image deblurring method based on geometric prior and conditional diffusion according to claim 1, characterized in that, The step of guiding the update of the current image based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image to obtain the updated image includes: Based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, a depth estimation confidence map is calculated. Based on the depth estimation confidence map, target regions in the current image with confidence levels below a set threshold are identified. Within these target regions, the current image is guided to be updated based on the second depth map of the current image, resulting in the updated image.

3. The zero-shot image deblurring method based on geometric prior and conditional diffusion according to claim 2, characterized in that, The step of guiding an update of the current image based on the second depth map of the current image to obtain the updated image specifically includes: The second depth map of the current image and the current image are input into the guiding filter to obtain the output result of the guiding filter; Based on the output, the current image, and the depth estimation confidence map, the updated image is determined.

4. The zero-shot image deblurring method based on geometric prior and conditional diffusion according to claim 1, characterized in that, The step of determining the adaptive threshold based on the overall difference includes: Calculate the mean and standard deviation of the overall differences; The adaptive threshold is determined based on the mean and the standard deviation.

5. The zero-shot image deblurring method based on geometric prior and conditional diffusion according to claim 2, characterized in that, The step of calculating a depth estimation confidence map based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image includes: Based on the second depth map of the current image and the second depth map of the previous image, calculate the first depth consistency between the current image and the previous image; Based on the image to be processed and the second depth map of the current image, calculate the second depth consistency between the current image and the image to be processed; The depth estimation confidence map is calculated based on the first depth consistency and the second depth consistency.

6. A zero-shot image deblurring system based on geometric priors and conditional diffusion, characterized in that, include: The image acquisition module is used to acquire the image to be processed. The prior image generation module is used to recover the image to be processed to obtain an initial intermediate image, and generate a geometric prior image based on the initial intermediate image; The image restoration module is used to input the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model. The step of generating a geometric prior map based on the initial intermediate image includes: Using the initial intermediate image as the current image, monocular depth estimation is performed on the image to be processed and the current image respectively to determine the first depth map of the image to be processed and the second depth map of the current image; Calculate the overall difference between the first depth map of the image to be processed and the second depth map of the current image, and determine an adaptive threshold based on the overall difference; If the overall difference is greater than the adaptive threshold, then based on the image to be processed, the second depth map of the current image, and the second depth map of the previous image of the current image, the current image is guided to be updated to obtain the updated image; The updated image is used as the current image, and the above process is iterated until the overall difference is less than or equal to the adaptive threshold or the preset iteration round is reached. The second depth map of the current image is used as the geometric prior map. The conditional diffusion model includes a control network, a pre-trained model encoder, an encoding / decoding module, and a denoising diffusion implicit model. The step of inputting the image to be processed, the initial intermediate image, and the geometric prior image into the conditional diffusion model to obtain the deblurred image corresponding to the image to be processed output by the conditional diffusion model includes: The geometric prior map is input into the control network to obtain the geometric control features output by the control network; The initial intermediate image is input into the pre-trained model encoder to obtain the perceptual content features output by the pre-trained model encoder; The image to be processed, the geometric control features, and the perceptual content features are input into the encoding and decoding module to obtain the noise estimate output by the encoding and decoding module; The noise estimate is input into the denoising diffusion implicit model to obtain the deblurred image output by the denoising diffusion implicit model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the zero-sample image deblurring method based on geometric priors and conditional diffusion as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the zero-sample image deblurring method based on geometric priors and conditional diffusion as described in any one of claims 1-5.

Images