A method for blurry image diffusion restoration with background-guided constraints
By introducing a background guidance module and a diffusion generation module into the SwinIR network and correcting the MSE loss function, a blurred image restoration network is constructed, which solves the problems of over-smoothing and loss of detail in complex image degradation and achieves a more efficient image restoration effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HENAN UNIVERSITY
- Filing Date
- 2024-12-28
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies are not ideal for restoring complex degraded images, often resulting in over-smoothing, loss of detail and texture, and limited generalization ability.
A blurred image diffusion restoration method integrating background guidance constraints is proposed. By adding a background guidance module and a diffusion generation module to the SwinIR network and modifying the MSE loss function, a blurred image restoration network is constructed. The background guidance module and the diffusion generation module are used to generate more details and contour information.
It improves the realism and detail preservation of images, significantly reduces the problem of over-smoothing edges, enhances the generalization ability of the model, and improves the accuracy and efficiency of image restoration.
Smart Images
Figure CN119831877B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence image processing technology, specifically to a method for blurry image diffusion restoration by incorporating background-guided constraints. Background Technology
[0002] With the development of information technology, digital image processing technology has been applied in many fields. Digital image restoration is an important branch of this field, playing a role in medical imaging, remote sensing monitoring, agricultural analysis, and image recognition.
[0003] Image restoration aims to recover high-quality images from low-quality observations. Typical tasks include denoising, deblurring, dehazing, deraining, and super-resolution reconstruction. Traditional image restoration research typically assumes that the image degradation process is known and relatively simple (e.g., Gaussian noise, bicubic downsampling), and has proposed various image restoration algorithms based on this assumption. However, these algorithms have limited generalization ability and struggle to handle complex real-world degradation problems. Furthermore, traditional methods often focus on pixel-level errors, employing direct gradient constraints or simple loss functions (e.g., mean squared error, MSE), which may lead to over-smoothing of the image, resulting in loss of detail and texture, and ultimately, poor performance when dealing with complex degraded images. Summary of the Invention
[0004] To address the technical problem of poor image restoration results, this invention proposes a blurred image diffusion restoration method incorporating background-guided constraints. First, to improve the accuracy and realism of the restoration, the similarity between the generated image and the background of the noisy image is used to guide the restoration process, forming background-guided constraints. This improves the problem of unrealistic restored images caused by overly smooth backgrounds. Furthermore, to preserve more details and contour information in the image, a pre-trained diffusion generation module is used to generate more image details based on the restored image. Finally, a blurred image restoration network incorporating background-guided constraints is proposed, forming a novel blurred image restoration method.
[0005] In a first aspect, the present invention provides a method for restoring blurred image diffusion by incorporating background-guided constraints, the method comprising:
[0006] Obtain the image to be restored;
[0007] The blurred image restoration network is pre-trained to restore the image to be restored. The blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network and modifying its MSE loss function.
[0008] The training process of a blurred image restoration network includes:
[0009] Obtain a clear dataset, blur it, and generate a degraded blurred image;
[0010] A blurred image restoration network is constructed based on the SwinIR network, the modified MSE loss function, the background guidance module, and the pre-trained diffusion generation module. The modified MSE loss function is a loss function that adds background guidance as a constraint to the MSE loss function. The SwinIR network includes a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module.
[0011] Using degraded blurred images as training samples, clear images from the clear dataset as training labels, and the modified MSE loss function as the loss function during training, the constructed blurred image restoration network is trained to obtain the trained blurred image restoration network.
[0012] Optionally, the blurred image restoration network constructed based on the SwinIR network, the modified MSE loss function, the background guidance module, and the pre-trained diffusion generation module includes:
[0013] A background guidance module is introduced before the high-resolution image reconstruction module included in the SwinIR network; the background guidance module calculates the blurred image... and restored image The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through iterative iteration.
[0014] Replace the MSE loss function of the SwinIR network structure with the modified MSE loss function;
[0015] A pre-trained diffusion generation module is introduced, and the resulting network is determined to be a blurred image restoration network.
[0016] Optionally, the image input to the diffusion generation module is subjected to final restoration, including:
[0017] The restored image input to the diffusion generation module The final result is obtained using Stable Diffusion; in the above process, IR ControlNet is used to guide the final restored image. Compared to the original clear image The local pixel information is the same, thus outputting the final restored image. .
[0018] Optionally, the background guidance module calculates a blurred image. and restored image The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through iterative iteration, including:
[0019] For blurry images and restored image By adding different degrees of Gaussian blur, the corresponding representations of the Gaussian-blurred images with different blur kernels are as follows:
[0020] ,
[0021] in, This represents Gaussian blur under different blur kernels. The size of the fuzzy kernel;
[0022] Gradient calculation is performed on the obtained Gaussian blurred image, and the corresponding gradient information is represented as follows:
[0023]
[0024] in, This indicates that gradient calculations are performed on the image. This represents the image after Gaussian blurring of the input blurred image. This represents the image after Gaussian blurring of the restored image obtained from the blurred image restoration network;
[0025] Calculate its gradient difference Norms, as a guide to the context, are represented as follows:
[0026] .
[0027] Optionally, background guidance is added as a constraint to the MSE loss function to obtain a modified MSE loss function, including:
[0028] The background guiding constraint and MSE are used together as the loss function of the blurred image restoration network, and different weight coefficients are set for the gradient difference of the image under different Gaussian blur kernels to obtain the total loss function, which is the modified MSE loss function:
[0029]
[0030] in, express The square of the norm, Represents the restored image. Indicates a clear image. This represents the trade-off coefficients under different fuzzy kernels. This indicates that gradient calculations are performed on the image. This represents the image after Gaussian blurring of the input blurred image. This represents the image after Gaussian blurring of the restored image obtained from the blurred image restoration network.
[0031] The present invention has the following beneficial effects:
[0032] First, it improves the image's ability to restore the main subject. Specifically, the constructed background guidance module can provide direction for image restoration on the background, allowing the image to retain the desired subject information as much as possible during the restoration process, rather than restoring all content in the image. This invention can more accurately restore the image's main subject information during the restoration process, significantly reducing the problem of overly smooth edges, while the restored image is more realistic.
[0033] Secondly, the trained network is easy to use; pre-trained model weights can be used during inference, thus improving restoration efficiency in practice. Accurate and effective restoration of blurred images has aided downstream tasks in many fields.
[0034] In summary, this invention cleverly constructs a background guidance module and combines it with MSE to serve as a new loss function for image restoration. This not only improves the realism and richness of image details in various image restoration tasks, but also enhances the generalization ability of the model. Attached Figure Description
[0035] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a flowchart of the blurred image diffusion restoration method with fused background guidance constraints according to the present invention;
[0037] Figure 2 This is a flowchart illustrating the training process of the blurred image restoration network of the present invention.
[0038] Figure 3 This is a schematic diagram of the structure of the blurred image restoration network of the present invention;
[0039] Figure 4 This is a comparison chart of the image restoration results of the present invention. Detailed Implementation
[0040] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0042] refer to Figure 1 The flowchart illustrates some embodiments of the blurred image diffusion restoration method with fused background-guided constraints according to the present invention. This blurred image diffusion restoration method with fused background-guided constraints includes the following steps:
[0043] Step S1: Obtain the image to be restored.
[0044] The image to be restored can be a blurred image that needs to be restored.
[0045] In some embodiments, the image captured by the camera can be recorded as the image to be restored.
[0046] Step S2: The image to be restored is restored using a pre-trained blurred image restoration network.
[0047] The blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network, and modifying its MSE loss function. The diffusion generation module is also known as the diffusion model. The diffusion model is a technique based on Markov processes and has been applied in fields such as computer vision, natural language processing, multimodal learning, time series analysis, and adversarial learning.
[0048] In some embodiments, the image to be restored can be input into a blurred image restoration network, and the image to be restored can be achieved through the blurred image restoration network.
[0049] refer to Figure 2 This illustrates the training process of a blurred image restoration network, which may include the following steps:
[0050] Step 201: Obtain a clear dataset, blur it, and generate a degraded blurred image.
[0051] In some embodiments, a clear dataset similar to a blurred image can be sought. Various types of blurring processes are applied to generate a degraded blurred image. This results in a sharp-blurred image pair. Then, it is divided into training set and validation set according to a certain ratio.
[0052] As an example, this step may include the following steps:
[0053] The first step is to obtain a clear dataset related to the blurry image. .
[0054] The second step involves applying various blurring methods to the clear dataset to provide as many blurring types as possible, thereby obtaining the corresponding blurred images. The resulting fuzzy dataset is divided into a training set and a validation set according to a predetermined ratio.
[0055] The third step is to use make-list.py to convert the training and validation set data into different formats and save them as text files, thus obtaining the file types required for training the model.
[0056] Step 202: Construct a blurred image restoration network based on the SwinIR network, the modified MSE loss function, the background guidance module, and the pre-trained diffusion generation module.
[0057] The modified MSE loss function is based on the MSE loss function, with background guidance added as a constraint. The SwinIR (Image Restoration Based on Swin Transformer) network mainly includes: a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module.
[0058] In some embodiments, a background guidance module is constructed based on the SwinIR network structure and used as part of the constraints. Then, a pre-trained diffusion generation module is used to generate relevant details and contour information of the image. The network is trained using images from data preprocessing, and the loss function value of the network model is calculated for each iteration until the loss function value converges to a minimum. This yields the weights of the trained model and the restored image. This forms the restoration model used in this invention, also known as the blurred image restoration network. The trained model is loaded, and the blurred image is restored using the restoration model formed in this invention, thus completing the image restoration task.
[0059] As an example, this step may include the following steps:
[0060] The first step involves introducing a background guidance module before the high-resolution image reconstruction module included in the SwinIR network. This background guidance module is designed to constrain the reconstructed image. Background and blurred image The background is made as similar as possible to the subject, thus drawing more attention to the blurred image. Specifically, the background guidance module calculates the background image... and restored image The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to characterize the background gradient difference between the images before and after restoration. The model parameters are adjusted iteratively to improve the quality of image restoration. Specifically, this may include the following sub-steps:
[0061] The first sub-step involves processing the blurred image. and restored image By adding different degrees of Gaussian blur, the corresponding representations of the Gaussian-blurred images with different blur kernels are as follows:
[0062] ,
[0063] in, This represents Gaussian blur under different blur kernels. The size of the fuzzy kernel.
[0064] The second sub-step involves calculating the gradient of the Gaussian blurred image to obtain the corresponding gradient information.
[0065]
[0066] in, This indicates that gradient calculations are performed on the image. This represents the image after Gaussian blurring of the input blurred image. This represents the image after Gaussian blurring of the restored image obtained from the blurred image restoration network.
[0067] The third sub-step is to calculate the gradient difference. Norms, as a guide to the context, are represented as follows:
[0068] .
[0069] The second step is to replace the MSE loss function of the SwinIR network structure with the modified MSE loss function, and then continuously iterate and optimize until the loss function converges, thus saving the weight parameters of the image restoration model.
[0070] It should be noted that using only MSE loss as the loss function of the restoration model may result in oversmoothing or cartoonish imagery. To mitigate this, background guidance is incorporated as a constraint, working in conjunction with MSE as the model's loss function. Different levels of Gaussian blur and varying weights are also applied to ensure that the image restoration process focuses more on the subject's information. Using the constructed loss function (based on MSE loss and background guidance constraints), dynamic adjustments are made, and the gradient descent algorithm is used for iterative optimization until convergence is achieved.
[0071] For example, by adding background guidance as a constraint to the MSE loss function, the modified MSE loss function can be obtained by using the background guidance constraint and MSE together as the loss function of the blurred image restoration network, and setting different weight coefficients for the gradient difference of the image under different Gaussian blur kernels. Therefore, the total loss function, which is the modified MSE loss function, is as follows:
[0072]
[0073] in, express The square of the norm, Represents the restored image. Indicates a clear image. This represents the trade-off coefficients under different fuzzy kernels. This indicates that gradient calculations are performed on the image. This represents the image after Gaussian blurring of the input blurred image. This represents the image after Gaussian blurring of the restored image obtained from the blurred image restoration network.
[0074] The model parameters are adjusted by iteratively training the system, repeatedly calculating the loss function and updating the parameters, until the loss function converges.
[0075] The third step is to introduce a pre-trained diffusion generation module and determine the final network as a blurred image restoration network.
[0076] It should be noted that the pre-trained diffusion generation model enables the restored image to generate more details and contours. Based on the pre-trained diffusion generation module, the model is fine-tuned according to the required restoration type to complete the image restoration process. Specifically, the final restoration of the image input to the diffusion generation module can include: the restored image input to the diffusion generation module... The final result, or the final restored image, is obtained using Stable Diffusion. In the above process, IR ControlNet is used to guide the final restored image. Compared to the original clear image The local pixel information is the same, thus outputting the final restored image. .
[0077] Step 203: Using the degraded blurred image as the training sample, the clear image in the clear dataset as the training label, and the modified MSE loss function as the loss function in the training process, the constructed blurred image restoration network is trained to obtain the trained blurred image restoration network.
[0078] It should be noted that the structure of a blurred image restoration network can be as follows: Figure 3 As shown, the comparison of image restoration results can be seen as follows: Figure 4 As shown.
[0079] In summary, this invention proposes a blurred image diffusion restoration method that integrates background-guided constraints. This method combines the powerful image generation capabilities of diffusion models with the restoration method that integrates background-guided constraints to form a new blurred image restoration method. It aims to improve image clarity while maintaining image authenticity as much as possible, and can better promote the development of image restoration technology.
[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for restoring blurred image diffusion by incorporating background-guided constraints, characterized in that, Includes the following steps: Obtain the image to be restored; The blurred image restoration network is pre-trained to restore the image to be restored. The blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network and modifying its MSE loss function. The training process of a blurred image restoration network includes: Obtain a clear dataset, blur it, and generate a degraded blurred image; A blurred image restoration network is constructed based on the SwinIR network, the modified MSE loss function, the background guidance module, and the pre-trained diffusion generation module. The SwinIR network includes a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module. The blurred image restoration network, constructed based on the SwinIR network, a modified MSE loss function, a background guidance module, and a pre-trained diffusion generation module, includes: introducing a background guidance module before the high-resolution image reconstruction module included in the SwinIR network; the background guidance module calculates the blurred image... and restored image The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term. The model parameters are adjusted through iterative iteration to obtain the corrected MSE loss function. Specifically, the background constraint and MSE are used together as the loss function of the blurred image restoration network, and different weight coefficients are set for the gradient difference of the image under different Gaussian blur kernels to obtain the total loss function, which is the corrected MSE loss function. A pre-trained diffusion generation module is introduced to generate the details and contour information of the image, and the final network is determined as the blurred image restoration network. Using degraded blurred images as training samples, clear images from the clear dataset as training labels, and the modified MSE loss function as the loss function during training, the constructed blurred image restoration network is trained to obtain the trained blurred image restoration network. The corrected MSE loss function is: in, express The square of the norm, Represents the restored image. Indicates the original, clear image. These represent the weight coefficients under different fuzzy kernels. This indicates that gradient calculations are performed on the image. This represents the image after Gaussian blurring of the input blurred image. This represents the image after Gaussian blurring of the restored image obtained from the blurred image restoration network; Calculate its gradient difference Norms, as a guide to the context, are represented as follows: 。 2. The method for restoring blurred image diffusion by fusing background-guided constraints according to claim 1, characterized in that, The final restoration of the image input to the diffusion generation module includes: The restored image input to the diffusion generation module The final result is obtained using Stable Diffusion; in the above process, IR ControlNet is used to guide the final restored image. Compared to the original clear image The local pixel information is the same, thus outputting the final restored image. .