Blurred image diffusion restoration method fusing background guide constraint

Abstract

The invention relates to the technical field of artificial intelligence image processing, in particular to a blurred image diffusion restoration method fusing background guide constraints, which comprises the steps of fusing the background guide constraints into restoration loss, and realizing restoration of a blurred image by combining a pre-trained diffusion generation module. In order to improve the accuracy and authenticity of restoration, the similarity between the background of the generated image and the background of the noise image is utilized to guide the restoration process to form background guide constraints, and the problem that the restored image is unreal due to the fact that the background is too smooth in the restoration process is solved. In order to maintain more details and contour information in the image, on the basis of the restored image, a pre-trained diffusion generation module is utilized to generate more details of the image. Finally, a blurred image restoration network fusing background guide constraints is provided, image degradation caused by factors such as noise and shooting can be effectively relieved, and the image quality is improved. The subject of the image can be made clearer by restoring the image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing technology, and in particular to a fuzzy image diffusion restoration method integrating background guidance constraints. Background Art

[0002] With the development of information technology, digital image processing technology has been applied in many fields. Digital image restoration is an important branch of this field, playing a role in medical imaging, remote sensing monitoring, agricultural analysis and image recognition.

[0003] Image restoration aims to restore high-quality images from low-quality observed images. Typical tasks include denoising, deblurring, defogging, deraining, and super-resolution reconstruction. Traditional image restoration research usually assumes that the image degradation process is known and relatively simple (such as Gaussian noise, bicubic downsampling, etc.), and a variety of image restoration algorithms are proposed based on this. However, the generalization ability of these algorithms is limited and it is difficult to deal with complex real degradation problems. In addition, traditional methods focus more on pixel-level errors and use direct gradient constraints or simple loss functions (such as mean square error MSE), which may lead to over-smoothing of images, loss of details and textures, and unsatisfactory results when processing complex degraded images. Summary of the invention

[0004] In order to solve the technical problem of poor image restoration effect, the present invention proposes a fuzzy image diffusion restoration method that integrates background guidance constraints. First, in order to improve the accuracy and authenticity of the restoration, the similarity between the generated image and the background of the noise image is used to guide the restoration process to form a background guidance constraint, which improves the problem of unrealistic restored images caused by excessive background smoothing during the restoration process. Furthermore, in order to maintain more details and contour information in the image, the pre-trained diffusion generation module is used on the basis of the restored image to generate more image details. Finally, a fuzzy image restoration network that integrates background guidance constraints is proposed, forming a new fuzzy image restoration method.

[0005] In a first aspect, the present invention provides a blurred image diffusion restoration method integrating background guidance constraints, the method comprising:

[0006] Obtaining the image to be restored;

[0007] The image to be restored is restored through the pre-trained blurred image restoration network, wherein the blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network and correcting its MSE loss function;

[0008] The training process of the blurred image restoration network includes:

[0009] Obtain a clear data set, blur it, and generate a degraded blurred image;

[0010] Based on the SwinIR network, the modified MSE loss function, the background guidance module and the pre-trained diffusion generation module, a blurred image restoration network is constructed. The modified MSE loss function is a loss function that adds background guidance as a constraint on the basis of the MSE loss function. The SwinIR network includes: a shallow feature extraction module, a deep feature extraction module and a high-definition image reconstruction module.

[0011] The degraded blurred images are used as training samples, the clear images in the clear data set are used as training labels, and the modified MSE loss function is used as the loss function in the training process. The constructed blurred image restoration network is trained to obtain a trained blurred image restoration network.

[0012] Optionally, the blurred image restoration network is constructed based on the SwinIR network, the modified MSE loss function, the background guidance module and the pre-trained diffusion generation module, including:

[0013] The background guidance module is introduced before the HD image reconstruction module included in the SwinIR network; the background guidance module calculates the blurred image I blur and the restored image I pred The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through loop iteration;

[0014] Replace the MSE loss function of the SwinIR network structure with the modified MSE loss function;

[0015] A pre-trained diffusion generation module is introduced, and the final network is determined as a blurred image restoration network.

[0016] Optionally, the image input to the diffusion generation module is finally restored, including:

[0017] The restored image I input to the diffusion generation module pred The final result is obtained by using Sable Diffusion; in the above process, IR ControlNet is used to guide the final restoration image. Compared with the original clear image I clean The local pixel information is the same, so the final restored image is output

[0018] Optionally, the background guidance module calculates the blurred image I blur and the restored image I predThe gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through loop iteration, including:

[0019] For blurred image I blur and the restored image I pred Adding different degrees of Gaussian blur, the corresponding representations of the images after Gaussian blur under different blur kernels are:

[0020] G δ I blur , G δ I pred ,

[0021] Among them, G δ represents Gaussian blur under different blur kernels, δ is the size of blur kernel;

[0022] The gradient of the image after Gaussian blur is calculated, and the corresponding gradient information is represented as follows:

[0023]

[0024] in, Indicates the gradient operation of the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network;

[0025] Calculate the gradient difference L 1 Norm, as a guide for the background, is expressed as:

[0026]

[0027] Optionally, background guidance is added as a constraint on the basis of the MSE loss function to obtain a modified MSE loss function, including:

[0028] The background guidance constraint and MSE are used together as the loss function of the blurred image restoration network, and different weight coefficients are set for the gradient difference of the image under different Gaussian blur kernels. The total loss function, that is, the modified MSE loss function is:

[0029]

[0030] in, Indicates L 2 The square of the norm, I pred represents the restored image, I clean represents a clear image, βδ represents the trade-off coefficient under different blur kernels, Indicates the gradient operation of the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred Represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network.

[0031] The present invention has the following beneficial effects:

[0032] First, the main body restoration capability of the image is improved. Specifically, the constructed background guidance module can provide a restoration direction for the restored image on the background, so that the image retains the desired main body information as much as possible during the restoration process, rather than restoring all the contents in the image. The present invention can more accurately restore the main body information of the image during the restoration process, significantly reduce the problem of over-smoothing of the edges, and the restored image is more realistic.

[0033] Second, the trained network can be easily used, and the pre-trained model weights can be used during inference, which improves the efficiency of actual restoration. Accurate and effective restoration of blurred images helps downstream tasks in many fields.

[0034] In summary, the present invention cleverly constructs a background guidance module and combines it with MSE as a new loss function for image restoration. In various image restoration tasks, it not only improves the image realism and rich details, but also enhances the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flow chart of the blurred image diffusion restoration method integrating background guidance constraints of the present invention;

[0037] Figure 2 A flow chart of the training process of the blurred image restoration network of the present invention;

[0038] Figure 3 It is a structural schematic diagram of the blurred image restoration network of the present invention;

[0039] Figure 4 This is a comparison chart of the image restoration results of the present invention. DETAILED DESCRIPTION

[0040] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0041] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0042] refer to Figure 1 , shows the process of some embodiments of the blurred image diffusion restoration method integrating background guidance constraints according to the present invention. The blurred image diffusion restoration method integrating background guidance constraints comprises the following steps:

[0043] Step S1, obtaining an image to be restored.

[0044] The image to be restored may be a blurred image to be restored.

[0045] In some embodiments, the image captured by the camera may be recorded as the image to be restored.

[0046] Step S2, restoring the image to be restored by using a pre-trained blurred image restoration network.

[0047] Among them, the blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network and modifying its MSE loss function. The diffusion generation module is also called the diffusion model. The diffusion model is a technology based on Markov process generation and has been applied to computer vision, natural language processing, multimodal learning, time series analysis, and adversarial learning.

[0048] In some embodiments, the image to be restored may be input into a fuzzy image restoration network, and the image to be restored may be restored through the fuzzy image restoration network.

[0049] refer to Figure 2 , shows the process corresponding to the training process of the blurred image restoration network, which may include the following steps:

[0050] Step 201 , obtaining a clear data set, blurring it, and generating a degraded blurred image.

[0051] In some embodiments, a clear dataset similar to the blurred image may be sought. clean , perform various types of blur processing on it to generate the degraded blurred image I blur , thus forming a clear-blurred image pair I clean -I blur ,Then, divide it into training set and validation set according to a certain ratio.

[0052] As an example, this step may include the following steps:

[0053] The first step is to obtain a clear dataset I related to the blurred image clean .

[0054] The second step is to use multiple types of blur methods on the clear data set to make it have as many blur types as possible and obtain the corresponding blurred image I blur The fuzzy data set is divided into a training set and a validation set according to the set ratio.

[0055] The third step is to use make-list.py to convert the format of the training set and validation set data and save them as text files to obtain the file type required for training the model.

[0056] Step 202, constructing a blurred image restoration network based on the SwinIR network, the modified MSE loss function, the background guidance module and the pre-trained diffusion generation module.

[0057] Among them, the modified MSE loss function is a loss function that adds background guidance as a constraint on the basis of the MSE loss function. The SwinIR (Swin Transformer-based image restoration) network mainly includes: shallow feature extraction module, deep feature extraction module and high-definition image reconstruction module.

[0058] In some embodiments, based on the SwinIR network structure, a background guidance module is constructed and used as part of the constraint. Then, the pre-trained diffusion generation module is used to generate relevant details and contour information of the image. The network is trained with the image in the data preprocessing, and the loss function value of the network model of each iteration is calculated until the loss function value converges to the minimum, and the weight of the training model is saved and the restored image I is obtained. pred , thereby forming the restoration model used in the present invention, which is also recorded as a blurred image restoration network. The trained model is loaded, and the blurred image is restored using the restoration model formed by the present invention to complete the image restoration task.

[0059] As an example, this step may include the following steps:

[0060] In the first step, a background guidance module is introduced before the HD image reconstruction module included in the SwinIR network, wherein the background guidance module is designed to constrain the restored image I pred Background and blurred image I blur The background guidance module calculates the blurred image I blur and the restored image I pred The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to characterize the background gradient difference of the image before and after restoration. The model parameters are adjusted through cyclic iteration to improve the quality of image restoration. Specifically, the following sub-steps may be included:

[0061] The first sub-step is to blur the image I blur and the restored image I pred Adding different degrees of Gaussian blur, the corresponding representations of the images after Gaussian blur under different blur kernels are:

[0062] G δ I blur , G δ I pred ,

[0063] Among them, G δ It represents Gaussian blur under different blur kernels, and δ is the size of the blur kernel.

[0064] In the second sub-step, the gradient of the image after Gaussian blur is calculated, and the corresponding gradient information is represented as follows:

[0065] ▽G δ I blur ,▽G δ I pred ,

[0066] Among them, ▽ represents the gradient operation of the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred Represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network.

[0067] The third sub-step is to calculate the gradient difference L 1 Norm, as a guide for the background, is expressed as:

[0068]

[0069] In the second step, the MSE loss function of the SwinIR network structure is replaced with the modified MSE loss function, and the weight parameters of the image restoration model are saved through continuous iterative optimization until the loss function converges.

[0070] It should be noted that when only MSE loss is used as the loss function of the restoration model, over-smoothing or image cartooning may occur. In order to alleviate this phenomenon, background guidance is used as a constraint together with MSE as the loss function of the model, and different levels of Gaussian blur and different weights are set to ensure that more attention is paid to the subject information during the image restoration process. Using the constructed loss function (the loss function based on MSE loss and background guidance constraint), the gradient descent algorithm is used to iterate and optimize continuously through dynamic adjustment and changes to reach a convergence state.

[0071] For example, adding background guidance as a constraint on the basis of the MSE loss function to obtain a modified MSE loss function may include: using the background guidance constraint and the MSE as the loss function of the blurred image restoration network, and setting different weight coefficients for the gradient difference of the image under different Gaussian blur kernels, so as to obtain the total loss function, that is, the modified MSE loss function is:

[0072]

[0073] in, Indicates L 2 The square of the norm, I pred represents the restored image, I clean represents a clear image, β δ represents the trade-off coefficient under different blur kernels, Indicates the gradient operation of the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred Represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network.

[0074] The model parameters are adjusted through iterative training, and the loss function is repeatedly calculated and the parameters are updated until the loss function converges.

[0075] The third step is to introduce the pre-trained diffusion generation module and determine the final network as the blurred image restoration network.

[0076] It should be noted that the use of the pre-trained diffusion generation model enables the restored image to generate more details and contours. Based on the pre-trained diffusion generation module, the model is fine-tuned according to the required restoration type to complete the image restoration process. Specifically, the final restoration of the image input to the diffusion generation module may include: pred Use Sable Diffusion to get the final result, which is the final restored image In the above process, IR ControlNet is used to guide the final restoration of the image. Compared with the original clear image I clean The local pixel information is the same, so the final restored image is output

[0077] Step 203, using the degraded blurred image as a training sample, using the clear image in the clear data set as a training label, and using the modified MSE loss function as the loss function in the training process, the constructed blurred image restoration network is trained to obtain a trained blurred image restoration network.

[0078] It should be noted that the structure of the blurred image restoration network can be as follows Figure 3 As shown, the image restoration results can be compared as shown Figure 4 shown.

[0079] In summary, the present invention proposes a blurred image diffusion restoration method that integrates background guidance constraints. It combines the powerful image generation capability of the diffusion model with the restoration method that integrates background guidance constraints to form a new blurred image restoration method, which aims to improve the image clarity while maintaining the image authenticity as much as possible, and can better promote the development of image restoration technology.

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A fuzzy image diffusion restoration method integrating background guidance constraints, characterized in that: The following steps are involved: Obtaining the image to be restored; The image to be restored is restored through the pre-trained blurred image restoration network, wherein the blurred image restoration network is a neural network obtained by adding a background guidance module and a diffusion generation module to the SwinIR network and correcting its MSE loss function; The training process of the blurred image restoration network includes: Obtain a clear data set, blur it, and generate a degraded blurred image; Based on the SwinIR network, the modified MSE loss function, the background guidance module and the pre-trained diffusion generation module, a blurred image restoration network is constructed. The modified MSE loss function is a loss function that adds background guidance as a constraint on the basis of the MSE loss function. The SwinIR network includes: a shallow feature extraction module, a deep feature extraction module and a high-definition image reconstruction module. The degraded blurred images are used as training samples, the clear images in the clear data set are used as training labels, and the modified MSE loss function is used as the loss function in the training process. The constructed blurred image restoration network is trained to obtain a trained blurred image restoration network.

2. The method for fuzzy image diffusion restoration integrating background guidance constraint according to claim 1, characterized in that: The blurred image restoration network is constructed based on the SwinIR network, the modified MSE loss function, the background guidance module and the pre-trained diffusion generation module, including: The background guidance module is introduced before the HD image reconstruction module included in the SwinIR network; the background guidance module calculates the blurred image I blur and the restored image I pred The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through loop iteration; Replace the MSE loss function of the SwinIR network structure with the modified MSE loss function; A pre-trained diffusion generation module is introduced, and the final network is determined as a blurred image restoration network.

3. The method for fuzzy image diffusion restoration integrating background guidance constraint according to claim 1, characterized in that: The image input to the diffusion generation module is finally restored, including: The restored image I input to the diffusion generation module pred The final result is obtained by using Sable Diffusion; in the above process, IR ControlNet is used to guide the final restoration image. Compared with the original clear image I clean The local pixel information is the same, so the final restored image is output 4. The method for fuzzy image diffusion restoration integrating background guidance constraint according to claim 2, characterized in that: The background guidance module calculates the blurred image I blur and the restored image I pred The gradient difference under different Gaussian blur weights and different Gaussian blur kernels is used as a constraint term to adjust the model parameters through loop iteration, including: For blurred image I blur and the restored image I pred Adding different degrees of Gaussian blur, the corresponding representations of the images after Gaussian blur under different blur kernels are: G δ I blur ,G δ I pred , Among them, G δ represents Gaussian blur under different blur kernels, δ is the size of blur kernel; The gradient of the image after Gaussian blur is calculated, and the corresponding gradient information is represented as follows: in, Indicates the gradient operation of the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network; Calculate the L1 norm of its gradient difference as a guide for the background, and its corresponding expression is:

5. The method for fuzzy image diffusion restoration integrating background guidance constraint according to claim 1, characterized in that: On the basis of the MSE loss function, background guidance is added as a constraint to obtain the modified MSE loss function, including: The background guidance constraint and MSE are used together as the loss function of the blurred image restoration network, and different weight coefficients are set for the gradient difference of the image under different Gaussian blur kernels. The total loss function, that is, the modified MSE loss function is: in, represents the square of the L2 norm, I pred represents the restored image, I clean represents the original clear image, β δ represents the trade-off coefficient under different blur kernels, ▽ represents the gradient operation on the image, G δ I blur Represents the image after Gaussian blurring of the input blurred image, G δ I pred Represents the image after Gaussian blurring of the restored image obtained by the blurred image restoration network.

Images