Multi-degraded image restoration method based on self-adaptive prompt

Abstract

The invention provides a multi-degraded image restoration method based on self-adaptive prompt. The method comprises a two-stage hidden space diffusion model, and various degraded images can be flexibly processed. Specifically, the model generates a degradation cue by learning the difference between a degraded image and a clean image, and encodes the cue information in a hidden space. In the first stage, a degradation prompt is obtained through a degradation adaptive prompt learning mechanism as a key feature for guiding the diffusion model in the second stage to carry out image restoration. Finally, prompt information is integrated into a Fourier transform restoration network, and accurate restoration of different types of degraded images can be realized. The method not only improves the accuracy of image restoration, but also significantly reduces the calculation complexity, and is suitable for image restoration tasks in various real world scenes, including denoising, deblurring, defogging, rain removing, snow removing, raindrop removing and low illumination enhancement.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular to an image restoration method. Specifically, the present invention focuses on a multi-degraded image restoration technology based on adaptive prompting, aiming to improve the image restoration effect under various degradation conditions. Background Art

[0002] In the field of image restoration, many methods have been proposed in recent years to handle complex restoration tasks, each with its own advantages and disadvantages. The following are several major image restoration methods and their characteristics:

[0003] 1. Image restoration algorithm based on convolutional neural network (CNN)

[0004] CNN methods perform image restoration by learning local features of images, and have achieved remarkable results in various image restoration tasks, such as denoising, deblurring, and rain removal. For example, MPRNet progressively restores images through multiple stages, and each stage fuses intermediate information to obtain rich contextual features. MAXIM uses a UNet-based hierarchical structure and a spatially gated multi-layer perceptron to support long-range interactions, thereby achieving better image restoration results. However, these CNN-based methods mainly rely on local receptive fields, which may lead to artifacts when processing severely degraded images.

[0005] 2. Image restoration algorithm based on generative adversarial network (GAN)

[0006] Generative adversarial networks generate high-quality images through adversarial training of generators and discriminators, and have been widely used in image restoration tasks. For example, HRGAN

[31] uses generative adversarial networks to remove raindrops and can generate realistic raindrop-free images. Although GAN performs well in generating image details and realism, its training process is complicated, mode collapse may occur, and it requires a lot of computing resources, which affects the restoration effect.

[0007] 3. Image decoupling algorithm based on self-supervised learning

[0008] The Transformer architecture was originally used for sequence modeling tasks in natural language processing and was introduced to the field of computer vision due to its powerful feature representation capabilities. Transformer-based models can leverage the self-attention mechanism to model long-range dependencies and have achieved remarkable performance in a variety of image restoration tasks. For example, Restormer [reduces the number of operations while maintaining computational efficiency by computing scaled dot product attention in the feature depth domain. Uformer utilizes locally enhanced windowed Transformer blocks to improve the ability to capture local context. Compared to CNN-based methods, these Transformer-based methods are able to improve the performance of image restoration, but are still limited in recovering image details.

[0009] 4. Image decoupling algorithm based on unsupervised learning

[0010] The diffusion model is an emerging generative model that simulates the gradual transition from data distribution to noise distribution and reconstructs a clear image in the inverse process. For example, SR3 uses a denoising diffusion probability model for image super-resolution reconstruction, achieving better performance than methods based on generative adversarial networks. WeatherDiff restores images taken under severe weather conditions and uses a guided denoising process on overlapping blocks. However, these methods become time-consuming during inference due to the need for multiple image blocks and a large number of sampling steps.

[0011] In summary, current image restoration algorithms perform well when dealing with the types of degradation seen during training, but their restoration performance usually degrades significantly when faced with unknown or rare degradation conditions. In particular, in practical applications, images may encounter a variety of complex degradation conditions, such as blur, noise, weather effects, etc. These unfavorable factors will seriously affect the image quality and have a negative effect on advanced applications. Although methods based on CNN, GAN, Transformer and diffusion models have made some progress in the field of image restoration, they are still limited in adaptability and flexibility. Summary of the invention

[0012] In view of the above problems, the present invention proposes a multi-degraded image restoration method based on adaptive hinting, and comprises the following steps:

[0013] 1. A method for restoring a multi-degraded image based on adaptive hinting, characterized by comprising the following steps:

[0014] S1, constructs a degradation adaptive cue learning mechanism, which adaptively generates degradation cues by exploring the differences between degraded images and clean images, and encodes the cue information into the latent space as the key feature for processing various image degradation types;

[0015] S2, constructs a latent space diffusion model, which can adaptively capture key degradation features from degraded images using only degraded images, and directly estimate the cues similar to the degradation cues generated by S1;

[0016] S3, constructs a cue-guided Fourier transform inpainting network to integrate the cues extracted in S2 to enhance the model's ability to represent global frequency features and local spatial information;

[0017] S4, build an adaptive cue-based image inpainting network, dynamically fine-tune the Fourier transform inpainting network using the cues generated in S3, and train the model with a large amount of training data;

[0018] S5, input the degraded image into the constructed image restoration network to complete the accurate restoration of the image.

[0019] 2. The method for restoring a multi-degraded image based on adaptive prompting according to claim 1, wherein the step of constructing a degradation adaptive prompting learning mechanism in step S1 specifically comprises the following steps:

[0020] S11, receiving a pair of degraded images and corresponding clean images, concatenating the degraded images and the clean images along the channel dimension to form an input feature map, which can be expressed as;

[0021] F = Concat(I GT ,I LQ ) (1)

[0022] S12, input the S11 feature map into the pre-trained degradation adaptive cue learning mechanism;

[0023] S13, the degradation adaptive cue learning mechanism extracts the feature differences between the degraded image and the clean image through a series of convolutional layers and nonlinear activation functions;

[0024] S14, inputting the extracted feature differences into a prompt learning module, and the prompt learning module generates a degradation prompt vector through adaptive weighting and normalization operations;

[0025] S15, encoding the generated degradation hint vector into a latent space to form key discriminant information for feature extraction and processing in the subsequent image restoration process.

[0026] p=DAPLM(F) (2)

[0027] 3. The method for restoring a multi-degraded image based on adaptive hinting according to claim 1, wherein step S2 specifically comprises:

[0028] S21, first uses the latent space diffusion model, i.e., the degradation cue estimator, using only the degraded image as input;

[0029] S22, degradation hint estimator Through the forward diffusion process, we set p = p 0 As the sampling starting point of the diffusion process, Gaussian noise is gradually added to the degraded image until it is completely converted into a Gaussian noise distribution. This part can be described as:

[0030]

[0031] Where t=1...T, T is the total number of iterations. t represents the noise characteristics of the tth step, and N is a Gaussian distribution. Through iterative derivation, the previous formula is rewritten as:

[0032]

[0033] S23, in the reverse process, our goal is to generate degraded prior information from a pure Gaussian distribution. The purpose of the reverse process of the diffusion model is to estimate the posterior distribution from the following formula:

[0034]

[0035] The degradation cue estimator takes pure Gaussian noise as the starting point, uses conditional information, gradually removes the noise and recovers cue information similar to the degradation cue obtained in the S1 stage as conditional guidance to optimize the diffusion process. This part can be described as:

[0036]

[0037] S25, after T iterations, the degradation hint estimator outputs the final estimated degradation hint, which is used in the subsequent image restoration process. The optimization process is:

[0038]

[0039] 4. The method for restoring a multi-degraded image based on adaptive hinting according to claim 1, wherein step S3 is specifically:

[0040] S31, construct a hint-guided Fourier transform restoration network, which is designed to integrate the degradation hints extracted in step S2. The hint-guided Fourier transform restoration network embeds the degradation hints into multiple layers of the network to enhance the model's feature representation ability for the global frequency features and local spatial information of the image; specifically, the obtained degradation prior p is embedded into the given input feature F in Targeted guidance for different image restoration tasks:

[0041]

[0042] In the formula represents element-wise multiplication, Norm(·) represents layer normalization, and W l represents the linear layer, F n1 ,F in denote the input and output feature maps respectively.

[0043] S32, converts the input degraded image into the frequency domain through Fourier transform, and converts the input Projected onto the query Q, key K, and value V. In addition, we perform fast Fourier transform on the obtained features Q and K to estimate the correlation between Q and K in the frequency domain. We estimate the total frequency feature and then use the residual operation to obtain the output feature F gf , this part can be expressed as:

[0044]

[0045]

[0046] Where F(·) represents FFT, represents the conjugate transpose operation, F -1 (·) represents inverse FFT.

[0047] S33, construct a mechanism to aggregate local spatial domain information and embed the degradation prior into the input F sa This part of the operation is the same as before. Subsequently, 1×1 point-by-point convolution and 3×3 deconvolution are used to aggregate the information of different channels and spatially adjacent pixels, and the information of spatially adjacent pixels and different channels is aggregated. At the same time, the gating mechanism of aggregated features is explored to strengthen information encoding. Finally, the aggregated local features are estimated, and then the residual operation is used to obtain the output feature F out . This part can be expressed as:

[0048]

[0049] I HQ =W c F ls +F in (14)

[0050] S34, build an image restoration network based on adaptive cues, which combines the degradation cues obtained in steps S1 and S2. The image restoration network is trained using a large amount of training data containing multiple types of degradation, and the training data includes degraded images and their corresponding high-quality clean images. During the training process, the network parameters are dynamically adjusted to optimize the restoration effect to ensure that the network can adapt to different types of image degradation; the difference between the restored image and the clean image is measured by the loss function, and the optimization algorithm is used to minimize the loss function to improve the restoration quality. After sufficient training, an image restoration model that can handle multiple types of degradation is obtained, which can quickly and accurately restore the input degraded image in the inference stage.

[0051] Beneficial effects: Compared with the prior art, the present invention provides a multi-degraded image restoration method based on adaptive hints, which can handle various types of image degradation in an open environment and produce the following beneficial effects:

[0052] 1. Strong versatility: The existing technology lacks flexibility and adaptability in dealing with various uncertain image degradations. The present invention effectively adapts to and deals with various types of degradation including blur, noise, weather effects, etc. by introducing adaptive hints and latent space diffusion models. This method significantly improves the quality of image restoration, especially in dealing with boundary details and reducing artifacts.

[0053] 2. Computational efficiency: Traditional diffusion models require a large number of iterative steps to restore clean images, consuming a lot of computing resources. This invention significantly reduces the computational complexity of the model by using hints represented by low-dimensional latent vectors. During the inference process, this method can restore a SPA+ image in less than 0.09 seconds, demonstrating excellent real-time processing capabilities.

[0054] 3. Unified restoration framework: Compared with the method that requires training a model separately for each degradation type, this paper proposes a unified two-stage restoration framework) that can handle multiple degradation tasks simultaneously without retraining the network for each degradation type. This approach simplifies the training process, reduces the computational and memory burden, and makes the model easier to deploy in practical application scenarios.

[0055] 4. Wide application: The present invention has good robustness and universality, and can perform effective image restoration in a variety of application environments. Whether it is processing the degradation type seen during training or unknown new degradation, the present invention can provide high-quality restoration results and is suitable for various computer vision tasks, such as autonomous driving, medical imaging, security monitoring and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1This is a flow chart of training the image restoration model of the present invention.

[0057] Figure 2 A hint-guided Fourier transform inpainting network flow chart was constructed for the present invention.

[0058] Figure 3 The high-fidelity restoration effect of the present invention in various image restoration tasks, including (a) snowfall, (b) denoising, (c) raindrop removal, (d) defogging, (e) denoising, (f) deblurring, and (g) low-light enhancement. The three columns in each group of images represent the degraded images, the restored clean images, and the original images.

[0059] Figure 4 and Figure 5 This is a visualization result diagram of the multi-degraded image restoration method based on adaptive hints of the present invention and other methods.

[0060] Figure 6 and Figure 7 This is a comparison chart of the results of the adaptive hint-based multi-degraded image restoration method of the present invention and other methods on the Snow100K-S, Snow100K-K, Outdoor-Rain, RainDrop, Rain100L, SOTS, BSD68, GoPro and LOL datasets. DETAILED DESCRIPTION

[0061] The adaptive cue-based multi-degraded image restoration method of the present invention constructs a two-stage latent space diffusion model and example-driven restoration technology, so that the model can effectively restore multiple degradation types in the open world according to the degradation features in the image, thereby significantly improving the image restoration effect in actual scenes.

[0062] The invention will be further described below in conjunction with specific embodiments.

[0063] Embodiment 1:

[0064] like Figure 1 As shown in the first stage, the specific process of training degradation adaptive cue prior learning is as follows: First, collect image data pairs under different degradation conditions, including blur, noise, etc. These image pairs are sent to the pre-trained degradation adaptive cue learning mechanism, which adaptively generates degradation cues by learning the difference between degraded images and clean images, and encodes the cue information into the latent space to obtain key discriminative information; the generated cues are then sent to the cue-guided Fourier transform restoration network, which restores the degraded image based on these cue information, and compares the restored result with the clean image to calculate the loss, and optimizes the degradation adaptive cue learning mechanism through back propagation.

[0065] like Figure 1 As shown in the second stage, the specific process of training the latent space diffusion model guided by the degradation adaptive cue is as follows: First, we initialize the weights of the degradation adaptive cue learning mechanism obtained by pre-training in the first stage, and these weights are used to generate degradation cues. Then, we start the diffusion process with the degradation cues generated by the degradation adaptive cue learning mechanism as the starting point, gradually introduce Gaussian noise into it, and simulate the evolution of data from a clear state to a noisy state.

[0066] In the iterative denoising stage, we gradually recover clear image features from noisy data through multiple iterations. In each iteration, the denoising network estimates and removes noise, while using the degradation information of the degradation adaptive cue learning mechanism as a conditional guide to optimize the diffusion process. This process optimizes the network parameters by minimizing the difference between the reconstruction error and the noise, thereby improving the image restoration quality. In addition, we use the generated degradation cues to dynamically fine-tune PGFTR to adapt to the specific characteristics of different degraded images, further improving the detail quality and overall naturalness of the restored image. In the loss calculation and optimization stage, we compare the restored features with the original high-quality image, calculate the reconstruction loss, and optimize the denoising network parameters to minimize the loss function.

[0067] like Figure 2 As shown, the specific process of constructing the cue-guided Fourier transform restoration network of the present invention is as follows: first, a degraded image is input. Subsequently, the key degradation cues are extracted from the degraded image using the pre-trained degradation adaptive cue learning mechanism, which contain important information about the type and degree of image degradation. These degradation cues are used to construct a cue-guided Fourier transform restoration network, which consists of multiple cue-guided Fourier transformer blocks, each of which focuses on learning and modeling long-range dependencies and local details. The input degradation is first subjected to feature extraction through a series of convolutional layers, and then the features are converted from the spatial domain to the frequency domain using the fast Fourier transform to prepare for further processing.

[0068] In the frequency domain, the extracted degradation cues are embedded into the cue-guided Fourier transformer block to guide the network to perform targeted restoration on images with different degradation types. This process involves adjusting and optimizing the frequency features with the goal of restoring the details and quality of the image. After being processed by a series of cue-guided Fourier transformer blocks, the restored features are converted from the frequency domain back to the spatial domain using the inverse fast Fourier transform to obtain a preliminary restored clean image. The loss function, such as the mean square error or the structural similarity index, is calculated by comparing the restored clean image with the clean image, and the network weights are updated using the back-propagation algorithm to optimize the performance of the cue-guided Fourier transform restoration network.

[0069] The present invention provides a method for restoring multiple degraded images based on adaptive prompts. By inputting the degraded image and the corresponding high-definition clean image into a two-stage latent space diffusion model, this method enables the model to effectively handle various potential image degradation types according to adaptive prompts, including but not limited to denoising, deblurring, rain removal, defogging, raindrop removal, low-light enhancement, etc. By integrating the adaptive prompt mechanism into the model, the restoration effect of images of unknown degradation types in actual scenes is greatly improved. In particular, in the face of multiple degradation conditions occurring simultaneously, the method of the present invention can provide a more accurate and efficient image restoration solution.

[0070] Figure 4 and Figure 5 This is a visualization result diagram of the multi-degraded image restoration method based on adaptive hints of the present invention and other methods. It can be seen that the multi-degraded image restoration method based on adaptive hints has better effects than other methods.

[0071] Figure 6 and Figure 7 The figure is a comparison chart of the results of the multi-degraded image restoration method based on adaptive prompts of the present invention and other methods on Snow100K-S, Snow100K-K, Outdoor-Rain, RainDrop, Rain100L, SOTS, BSD68, GoPro and LOL datasets. It can be seen that the multi-degraded image restoration method based on adaptive prompts is more accurate than other methods.

[0072] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0073] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for restoring multi-degraded images based on adaptive hints, characterized in that: The following steps are involved: S1, constructs a degradation adaptive cue learning mechanism, which adaptively generates degradation cues by exploring the differences between degraded images and clean images, and encodes the cue information into the latent space as the key feature for processing various image degradation types; S2, constructs a latent space diffusion model, which can adaptively capture key degradation features from degraded images using only degraded images, and directly estimate the cues similar to the degradation cues generated by S1; S3, constructs a cue-guided Fourier transform inpainting network to integrate the cues extracted in S2 to enhance the model's ability to represent global frequency features and local spatial information; S4, build an adaptive cue-based image inpainting network, dynamically fine-tune the Fourier transform inpainting network using the cues generated in S3, and train the model with a large amount of training data; S5, input the degraded image into the constructed image restoration network to complete the accurate restoration of the image.

2. The method for restoring a multi-degraded image based on adaptive hinting as claimed in claim 1, characterized in that: The step S1 of constructing the degradation adaptive prompt learning mechanism specifically includes the following steps: S11, receiving a pair of degraded images and corresponding clean images, concatenating the degraded images and the clean images along the channel dimension to form an input feature map, which can be expressed as; F=Concat(I GT ,I LQ ) (1) S12, input the S11 feature map into the pre-trained degradation adaptive cue learning mechanism; S13, the degradation adaptive cue learning mechanism extracts the feature differences between the degraded image and the clean image through a series of convolutional layers and nonlinear activation functions; S14, inputting the extracted feature differences into a prompt learning module, and the prompt learning module generates a degradation prompt vector through adaptive weighting and normalization operations; S15, encoding the generated degradation hint vector into a latent space to form key discriminant information for feature extraction and processing in the subsequent image restoration process. p = DAPLM(F) (2).

3. The method for restoring a multi-degraded image based on adaptive hinting as claimed in claim 1, characterized in that: The step S2 is specifically as follows: S21, first uses the latent space diffusion model, i.e., the degradation cue estimator, using only the degraded image as input; S22, degradation hint estimator Through the forward diffusion process, we take p=p0 as the sampling starting point of the diffusion process, and gradually add Gaussian noise to the degraded image until it is completely converted into a Gaussian noise distribution. This part can be described as: Where t=1...T, T is the total number of iterations. t represents the noise characteristics of the tth step, and N is a Gaussian distribution. Through iterative derivation, the previous formula is rewritten as: S23, in the reverse process, our goal is to generate degraded prior information from a pure Gaussian distribution. The purpose of the reverse process of the diffusion model is to estimate the posterior distribution from the following formula: The degradation cue estimator takes pure Gaussian noise as the starting point, uses conditional information, gradually removes the noise and recovers cue information similar to the degradation cue obtained in the S1 stage as conditional guidance to optimize the diffusion process. This part can be described as: S25, after T iterations, the degradation hint estimator outputs the final estimated degradation hint, which is used in the subsequent image restoration process. The optimization process is:

4. The method for restoring a multi-degraded image based on adaptive hinting as claimed in claim 1, characterized in that: The step S3 is specifically as follows: S31, construct a hint-guided Fourier transform restoration network, which is designed to integrate the degradation hints extracted in step S2. The hint-guided Fourier transform restoration network embeds the degradation hints into multiple layers of the network to enhance the model's feature representation ability for the global frequency features and local spatial information of the image; specifically, the obtained degradation prior p is embedded into the given input feature F in Targeted guidance for different image restoration tasks: In the formula represents element-wise multiplication, Norm(·) represents layer normalization, and W l represents the linear layer, F n1 ,F in denote the input and output feature maps respectively. S32, converts the input degraded image into the frequency domain through Fourier transform, and converts the input Projected onto the query Q, key K, and value V. In addition, we perform fast Fourier transform on the obtained features Q and K to estimate the correlation between Q and K in the frequency domain. We estimate the total frequency feature and then use the residual operation to obtain the output feature F gf , this part can be expressed as: Where F(·) represents represents the conjugate transpose operation, F -1 (·) represents inverse FFT. S33, construct a mechanism to aggregate local spatial domain information and embed the degradation prior into the input F sa This part of the operation is the same as before. Subsequently, 1×1 point-by-point convolution and 3×3 deconvolution are used to aggregate the information of different channels and spatially adjacent pixels, and the information of spatially adjacent pixels and different channels is aggregated. At the same time, the gating mechanism of aggregated features is explored to strengthen information encoding. Finally, the aggregated local features are estimated, and then the residual operation is used to obtain the output feature F out . This part can be expressed as: S34, build an image restoration network based on adaptive cues, which combines the degradation cues obtained in steps S1 and S2. The image restoration network is trained using a large amount of training data containing multiple types of degradation, and the training data includes degraded images and their corresponding high-quality clean images. During the training process, the network parameters are dynamically adjusted to optimize the restoration effect to ensure that the network can adapt to different types of image degradation; the difference between the restored image and the clean image is measured by the loss function, and the optimization algorithm is used to minimize the loss function to improve the restoration quality. After sufficient training, an image restoration model that can handle multiple types of degradation is obtained, which can quickly and accurately restore the input degraded image in the inference stage.

Images