Dynamic blind image restoration method based on diffusion model

Abstract

The invention provides a dynamic blind image restoration method based on a diffusion model, and belongs to the technical field of image restoration. The method disclosed by the invention is based on a DPM (Diffusion Model) architecture, and mainly comprises three parts, namely a DCP (Degradation Content Perceptron), a Previewer and an Aggregator, which are used for realizing high-quality (HQ) recovery of a low-quality (LQ) image together, and the DPM architecture mainly comprises a DCP (Degradation Content Perceptron), a Previewer and an Aggregator which are used for realizing high-quality (HQ) recovery of the low-quality (LQ) image; according to the method, real-time generation type reference is utilized to maintain alignment with the generative model prior in the repair reasoning process, generation conditions are dynamically adjusted, unknown degradation scenes are effectively processed, and the method is excellent in quantitative index and visual quality through wide evaluation and experimental verification and has wide application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and in particular relates to a dynamic blind image restoration (BIR) method based on a diffusion model. Background Art

[0002] In the field of image restoration technology, the core task is to restore high-quality (HQ) visual details from low-quality (LQ) images. This technology has extremely important applications in many fields. In the field of social media, it can optimize the content shared by users, thereby improving the user experience. In the fields of autonomous driving and robotics, it is a key technology to improve the adaptability of the system in complex environments, helping the system to better identify the surrounding environment. In the field of object detection, when the detection environment is poor, image restoration technology can preprocess the input image to improve the accuracy of object detection. However, the image degradation process inevitably loses a lot of information. This means that a single low-quality image may correspond to multiple reasonable restoration results, making image restoration an ill-posed problem. In the blind image restoration (BIR) scenario, the problem is even more complicated because the model faces unknown image degradation conditions during testing, which places extremely high demands on the model's generalization ability.

[0003] To address the image restoration problem, researchers have proposed various methods, but each has limitations. Reference-based image restoration (Reference-IR) models rely on other high-quality images with similar content and richer visual details to modulate the features of low-quality images. In practical applications, obtaining such matching high-quality reference images is not easy, which limits the widespread application of this method. Generative methods achieve single-image restoration by directly learning the distribution of high-quality images. The principle is to first encode the input image into a hidden variable z, which is then used as a generative condition to sample high-quality images from the learned distribution p(y|z). Because the input distribution changes during image degradation, it is difficult for the encoder to accurately obtain the hidden variables. This makes generative methods prone to hallucinations, resulting in artifacts in the restored image, which affects image quality. To improve the robustness of existing methods, some researchers have attempted to train the model on richer and more diverse synthetic degradation data to learn more different types of degradation patterns. Others have introduced discrete feature codebooks, attempting to improve restoration performance by matching the features of low-quality images with the codebook. These methods can only alleviate the problem to a certain extent, but cannot fundamentally solve the difficulties faced by unknown inputs in blind image restoration. Therefore, it is necessary to explore new solutions.

[0004] The diffusion model (DPM), an emerging generative model, has made significant progress in the field of visual generation. It generates data from Gaussian noise through an iterative denoising approach, typically employing a neural network with a UNet architecture to predict the noise added at each inference step. Compared to variational autoencoders (VAEs) and models based on generative adversarial networks (GANs), DPMs demonstrate superior pattern coverage and generation quality without requiring adversarial training. By introducing additional inputs, DPMs can learn a variety of conditional distributions, a prominent application of which is text-to-image (T2I) generation. Leveraging the flexibility of text inputs and the abundance of text-image training data, DPMs demonstrate superior visual quality and rich diversity in generated images, laying the foundation for numerous subsequent outstanding research in visual generative models. Researchers are also actively exploring the potential of DPMs in the field of blind image restoration. Some studies have attempted to design specialized architectures and train DPMs from scratch, while others have opted to add additional modules to pre-trained T2I models, leveraging their extensive prior knowledge to achieve blind image restoration. In practical applications, when high-quality images with similar content to the image to be restored are available, such as photos in an album or video frames, reference-based methods have attracted widespread attention. These methods employ regression models to learn how to transfer features from high-quality images to low-quality images, thereby enhancing the recovery of image details. Therefore, the study of a dynamic blind image restoration method based on a diffusion model is of great significance. Summary of the Invention

[0005] In response to the above problems existing in the existing technology, the present invention proposes a dynamic blind image restoration method based on the diffusion model, called the INSTANTIR (Instant-reference Image Restoration) method. Based on the DPM architecture, it mainly includes three parts: a degradation content perceiver (DCP), an instant recovery previewer (Previewer) and an aggregator (Aggregator), which together achieve high-quality (HQ) restoration of low-quality (LQ) images.

[0006] The technical solutions of the present invention are as follows:

[0007] A dynamic blind image restoration method based on a diffusion model is characterized in that the method is based on a DPM architecture including a degraded content sensor, an instant restoration previewer, and an aggregator, and the specific steps are as follows:

[0008] Step S1: The degraded content perceptron extracts robust features c of low-quality LQ images through the pre-trained visual encoder lq, and adopts a learnable feature modulator Resampler containing a cross attention block for feature modulation. The modulation process introduces the adaptive layer normalization LayerNorm related to the time step t:

[0009]

[0010] in and Calculated by time step t, z is the diffusion variable and t is the time step;

[0011] Step S2: Based on the robust features, generate a repair reference image z through an instant recovery previewer r ;

[0012] Step S3: The reference image z is aggregated by an aggregator r With low quality LQ image z l Encoding the conditional input generated for the diffusion network repair;

[0013] Step S4: Adopting an adaptive restoration algorithm, taking the relative difference δ between the instant restoration previewer output and the denoised prediction as an indicator of input quality, dynamically adjusting the generation conditions according to δ, and gradually generating a restored image; wherein the calculation formula of the difference index δ is:

[0014]

[0015] in is the standard denoising prediction value, Ψ(·) is the generated reference feature output by the instant recovery previewer; the conditional signal strength of the aggregator is controlled by the δ value to achieve adaptive image restoration in degraded scenes.

[0016] Furthermore, the visual encoder adopts DINOv2.

[0017] Furthermore, the instant recovery previewer is a generative model obtained from the pre-trained text graph model through consistency distillation, and its loss function is:

[0018]

[0019] Where Ψ is the Wensheng graph model, z s 、z t are the diffusion variables at time steps s and t respectively.

[0020] Furthermore, the input H of the aggregator layer l is l Features that are split into previews in the spatial dimension and the potential features of the input LQ image And fusion is performed through spatial feature transformation (SFT), the formula is: in are two affine transformation parameters calculated based on the LQ latent feature map of this layer.

[0021] Furthermore, in the adaptive recovery algorithm:

[0022] When the δ value is greater than the threshold, the conditional signal of the aggregator is enhanced to preserve the original image details;

[0023] When the δ value is less than the threshold, the weight of the generated reference is weakened to prevent the generated path from diverging.

[0024] The technical effects of the present invention are as follows:

[0025] This paper presents a dynamic blind image restoration method based on a diffusion model. This method utilizes a real-time generative reference to maintain alignment with the generative model prior during the restoration inference process, dynamically adjusts the generation conditions, and effectively handles unknown degradation scenarios. Extensive evaluation and experimental validation have demonstrated that this method performs well in terms of quantitative metrics and visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The model structure and flow chart of the method of the present invention;

[0027] Figure 2 This is a comparison chart of the restoration quality of an embodiment of the present invention compared with other different methods; it can be seen that the restoration result of the present invention has richer and more reasonable detail texture;

[0028] Figure 3 This is a visualization diagram of the instant recovery previewer in an embodiment of the present invention; it can be seen that the introduction of generative repair reference can significantly improve detail repair. DETAILED DESCRIPTION

[0029] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.

[0030] This paper proposes a dynamic blind image restoration method based on a diffusion model, abbreviated as INSTANTIR. The model architecture and reasoning process flow chart are as follows: Figure 1 The model architecture includes a degradation content perceptron (DCP) for extracting robust visual features of LQ images, an instant restoration previewer (Previewer) for generating reference images based on image features, and an aggregator (Aggregator) for integrating LQ images with reference images as restoration generation conditions.

[0031] By maintaining alignment with the generative prior, leveraging knowledge from large-scale pre-trained models to handle unknown degradations, and dynamically adjusting generation conditions through instant preview of intermediate outputs, the present invention also develops an adaptive recovery algorithm suitable for the proposed model structure, achieving adaptive and controllable recovery. The specific steps of the present method include:

[0032] (1) Degraded Content Perceptron (DCP) uses the pre-trained visual encoder DINOv2 to extract LQ robust features c lq DINOv2 can maintain a high zero-shot classification accuracy in the face of various image degradations (such as noise, blur, and JPEG artifacts), and the high-level information it extracts can provide semantic guidance for the back-diffusion process. For further optimization, DCP modulates the representation of DINOv2 through a learnable feature modulator Resampler. Specifically, DCP uses a cross-attention block to process the output of DINOv2. At the same time, DCP introduces adaptive layer normalization (LayerNorm) based on the current inference time step t, modulating the output according to the time step t, and the calculation formula is:

[0033]

[0034] in and Calculated at time step t, z is the diffusion variable and t is the time step.

[0035] (2) Instant Restoration Previewer uses LQ image features extracted by DINOv2 lq As a condition, generate the repair reference image z r The instant recovery previewer is a generative model obtained by consistent distillation from the pre-trained text graph model, and the training loss function is

[0036]

[0037] Where Ψ is the Wensheng graph model, z s 、z t are the diffusion variables at time steps s and t respectively.

[0038] (3) Aggregator generates references ( Figure 1 z on r ) and LQ reference ( Figure 1 z on l) is integrated as the conditional input of the restoration generation (diffusion network). The aggregated condition can prevent the reasoning process from diverging. The main challenge of the blind image restoration task is the offset of the input distribution. The method INSTANTIR of the present invention aligns the Aggregator with the large-scale pre-trained generative model to prevent the generation path of the restoration from diverging. The input H of the first layer of the aggregator is l Features that are split into previews in the spatial dimension and the potential features of the input LQ image And fusion is performed through spatial feature transformation (SFT), the formula is

[0039]

[0040] in are two affine transformation parameters calculated based on the LQ potential feature map of this layer.

[0041] (4)INSTANTIR processes LQ images in two complementary ways, which may produce samples that deviate from the target HQ image when facing severe degradation. By analyzing the trajectory of the restored preview in the reverse process and comparing it with the denoised prediction, it is found that the relative difference between the preview and the ordinary denoised prediction can be used as an indicator of input quality. The specific indicator calculation formula is:

[0042]

[0043] in is the standard denoising prediction value, derived from the denoising formula; Ψ(·) is the generated reference feature output by the instant recovery previewer. A larger δ indicates higher input quality; in this case, the conditional signal of the aggregator should be amplified to preserve the fine-grained information of the original input. A smaller δ prevents INSTANTIR from diverging from the generated reference at the beginning. Based on this, the adaptive restoration (AdaRes) algorithm is designed. During the algorithm, δ is combined with the generated reference and denoising process to adjust according to different time steps t, gradually generating high-quality restored images.

[0044] The method of the present invention was used to conduct a comparative test with the existing method. In terms of qualitative comparison, Figure 2 As shown in the figure, through the restoration sample display of real-world LQ images, INSTANTIR uses the preview mechanism to reduce the hallucination phenomenon and can generate images with clear details and global semantic consistency, which is better than other comparison models in restoration effect; Figure 2In the first row, InstantIR is the only model among existing methods that successfully repairs 4 faces. At the same time, it also successfully restores realistic, delicate, and logical texture details in the second and third rows. Using common image quality indicators such as MUSIQ and CLIPIQA, the image quality indicators of InstantIR after repair on public super-resolution datasets are improved by up to 22% compared with existing methods. In addition, ablation experiments were conducted to verify the impact of different reference types on the restoration effect, such as the importance of using an instant restoration previewer to align the generation trajectory of the restoration task with the prior distribution of the generation model, and qualitatively compare the results of the above experiments. Figure 3 shown.

[0045] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A dynamic blind image restoration method based on a diffusion model, characterized in that: The method is based on the DPM architecture, which includes a degraded content sensor, an instant recovery previewer, and an aggregator. The specific steps are as follows: Step S1: The degraded content perceptron extracts robust features c of low-quality LQ images through the pre-trained visual encoder lq , and adopts a learnable feature modulator Resampler containing a cross attention block for feature modulation. The modulation process introduces the adaptive layer normalization LayerNorm related to the time step t: in and Calculated by time step t, z is the diffusion variable and t is the time step; Step S2: Based on the robust features, generate a repair reference image z through an instant recovery previewer r ; Step S3: The reference image z is aggregated by an aggregator r With low quality LQ image z l Encoding the conditional input generated for the diffusion network repair; Step S4: Adopting an adaptive restoration algorithm, taking the relative difference δ between the instant restoration previewer output and the denoised prediction as an indicator of input quality, dynamically adjusting the generation conditions according to δ, and gradually generating a restored image; wherein the calculation formula of the difference index δ is: in is the standard denoising prediction value, Ψ(·) is the generated reference feature output by the instant recovery previewer; The conditional signal strength of the aggregator is controlled by the δ value to achieve adaptive image restoration in degraded scenes.

2. The method according to claim 1, wherein The visual encoder adopts DINOv2.

3. The method according to claim 1, wherein The instant recovery previewer is a generative model obtained by consistent distillation from the pre-trained text graph model, and its loss function is: Where Ψ is the Wensheng graph model, z s 、z t are the diffusion variables at time steps s and t respectively.

4. The method according to claim 1, wherein The input H of the aggregator layer l l Features that are split into previews in the spatial dimension and the potential features of the input LQ image And fusion is performed through spatial feature transformation (SFT), the formula is: where α l 、 are two affine transformation parameters calculated based on the LQ latent feature map of this layer.

5. The method according to claim 1, wherein In the adaptive recovery algorithm: When the δ value is greater than the threshold, the conditional signal of the aggregator is enhanced to preserve the original image details; When the δ value is less than the threshold, the weight of the generated reference is weakened to prevent the generated path from diverging.

Images