A consistency model image restoration method and system based on range-null space decomposition

Abstract

The application provides a consistency model image restoration method based on range-null space decomposition, and the method is based on the processing of the consistency model for different image restoration tasks through the construction of the degradation matrix and the pseudo-inverse for the specific image restoration task, the picture range-null space decomposition, the null space refinement, and the actual implementation of the iterative sampling process. The image restoration task is processed by the consistency model through the range-null space decomposition, and the degradation matrix is designed for different tasks to realize the processing of different image restoration tasks. After the consistency model is introduced through the range-null space, the sampling process is designed according to the characteristics of the fast sampling of the consistency model to avoid the generation of artifacts. Because of the fast sampling characteristics of the consistency model itself, the sampling time cost is greatly reduced compared with other iterative generation models. And through the range-null space decomposition and other methods, the prior knowledge of the consistency model is fully utilized to generate high-quality results.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image restoration, and relates to a consistency model image restoration method and system based on range-null space decomposition. BACKGROUND

[0002] Image restoration aims to restore high-quality images from degraded images. Because of the non-well-posedness of the image restoration task, image restoration is often challenging, and image restoration technology has wide application value. Therefore, image restoration has been a long-term research problem in academia and industry.

[0003] In the current image restoration task, the main method is to use a high-performance generative model prior, and to make the high-quality prior of the generative model directly used in the image restoration task by reasonable design of the image sampling process. As long as the sampling method is designed to fully utilize the performance of the generative model prior, the higher the performance of the generative model, the higher the performance of the image restoration. In this method, the diffusion model is widely used as the generative model prior because of its high performance. However, the diffusion model has the problem of relying on iterative sampling, resulting in slow generation speed. To solve this problem, some people propose a consistency model based on the diffusion model. The consistency model is derived from the diffusion model but has faster generation speed and smaller performance loss compared with the diffusion model. However, there is no sampling method for the image restoration task based on the consistency model. Therefore, how to fully utilize the consistency prior and use it for the image restoration task to quickly generate high-quality degeneration results, and explore the possibility of using the consistency model for different image restoration tasks based on the known degeneration operator, it is suggested that a unified framework for solving the image restoration task has important significance.

[0004] With the emergence of generative models, such as diffusion models, generative adversarial networks and flow-based models, some researchers focus on using high-quality generative models to solve the image restoration problem. There are mainly two directions, the first direction is to train a generative model for a specific image to produce a more reasonable perceptual restoration image, however, the degradation network trained by this method is deterministic, often introduces artifacts that do not exist in the original clear image, and has poor scalability. The second direction uses well-performing generative models, such as denoising diffusion models or BigGAN, etc. The most critical part of this direction is to design a sampling method to fully utilize the prior knowledge of high-quality generative models to complete the image restoration task. Current sampling methods using pre-trained generative models for image restoration tasks, such as integrating deterministic predictors during the sampling process to generate samples ([1] Jay Whang et al. "Deblurring via Stochastic Refinement". In: CVPR. IEEE, 2022, PP. 16272-16282), DDRM effectively handles various conditions of different inverse problems by combining variational objectives with diffusion models ([2] Bahjat Kawar et al. "Denoising Diffusion Restoration Models". In: NeurlPS. 2022), GDP generates restoration images with target semantics by integrating degraded image sets into the inverse denoising process of denoising diffusion models ([3] Ben Fei et al. "Generative Diffusion Prior for Unified Image Restoration and Enhancement" n: CVPR. IEEE, 2023, pp. 9935-9946), DDNM ([4] Yang Song et al. "Consistency Models". In: ICML. Vol. 202. Proceedings of Machine) achieves zero-shot image restoration based on diffusion models through range-null space decomposition, these methods fully utilize diffusion models to achieve high-quality image restoration, but rely on the iterative sampling process of diffusion models, resulting in slow image restoration process. Unlike the step-by-step iterative sampling of diffusion models, consistency models generate high-quality samples by directly mapping noise to data, theoretically supporting fast one-step generation, while still allowing multi-step sampling to calculate the quality of samples, which is an ideal high-quality generative model for image restoration, but there is no image restoration method for consistency models yet. SUMMARY

[0005] To address the shortcomings of existing technologies, this invention proposes a consistency model-based image restoration method. This method decouples the consistency and realism optimization processes of images through range-null space decomposition, thereby introducing a consistency model. By rationally designing the null space thinning and noise distribution addition processes, high-quality results from the consistency model are generated. Furthermore, a backtracking method is used to address the issue of poor realism in specific situations such as average pooling super-resolution tasks and image inpainting under large mask conditions.

[0006] This invention proposes an image restoration method based on a consistency model using range-null space decomposition. The method, through the construction of degradation matrices and pseudo-inverses for specific image restoration tasks, image range-null space decomposition, null space thinning, and iterative sampling processes, utilizes a consistency model to process different image restoration tasks, including:

[0007] Step 1: Initialize relevant parameters, defining the degraded image as y and the target image as x. The image degradation process in the image restoration task is abstracted into a degradation matrix A, and the pseudo-inverse A' of the matrix is ​​obtained using mathematical methods.

[0008] Step 2: Initialize the noise in the degraded image by adding noise to the degraded image y. (Result of the noise addition) The result is x T This ensures that the noise distribution conforms to the consistency model. t represents the current time step, σ represents the variance of the added noise at each step, and ∈ represents the minimum variance of the added noise.

[0009] Step 3: Begin the iterative process, using a consistency model to process the noisy, degraded image x. t Denoising is performed to generate a preliminary prediction x for the target image. 0|t =f θ (x t ,σ t ); t represents the current time step.

[0010] Step 4: Provide the preliminary prediction for x 0|t Adding noise, the result of adding noise This ensures that the noise distribution conforms to the consistency model.

[0011] Step 5: Adjust the noise-adding results Perform range-null space decomposition to obtain the results. x t-1 x, as the starting point of the next iteration t Continue the iteration process; A' represents the pseudo-inverse matrix.

[0012] Step 6: The loop ends, and x1 is returned as the denoising result.

[0013] In the application, the initialized parameters include: setting the number of iterations and the time point of each iteration, the total number of iterations is T, and the noise variance sequence is T ≥σ T-1 ≥…≥σ1.The pre-trained consistency model f θ (x,t) is obtained, wherein the input x represents a picture that needs to be denoised, t represents the noise intensity of the picture, the noise used herein is a Gaussian noise by default, and the output result is a denoised picture.

[0014] In the application, for the image degradation process, it can be abstracted as y=Ax, wherein x represents a real image, y represents a degraded image, and A represents a degradation matrix of different image restoration tasks.For example, for an n times super-resolution task, It can be designed as an average pooling operation, and the value is

[0015] The degradation matrix A is designed according to different image restoration tasks, and the pseudo-inverse matrix A' is obtained by using the singular value decomposition (SVD) method, and the pseudo-inverse matrix A' satisfies the property A=AA'A.

[0016] In the application, the intensity of each noise depends on the current time point, and the noise gradually increases with the increase of the time point, and the mean value of the noise is always 0, and the relationship between the time point and the noise variance is t =t n+1 .

[0017] In the application, the result of the range-null space decomposition is x0'=A -1 y+(I-A -1 A)x, so that the obtained result x0' satisfies Ax0'=A(A'y+(I-A'A)x)=AA'y+A(I-A'A)x=AA'y+Ax-AA'Ax=Ax=y, thereby keeping the iteration result consistent with the degraded image.

[0018] Wherein, as long as the range space A'y of Ax0' is always unchanged, the result can be guaranteed to be consistent with the degraded image, so that the consistency and authenticity can be decoupled, the fixed range space is used to guarantee the consistency, and the iteration of the consistency model is used to guarantee the authenticity of the degraded image.

[0019] In the application, according to the definition of the consistency model itself, the difference between the multi-step iteration of the diffusion model is that the consistency model can generate a result within a single-step iteration, and the problem is that the consistency model directly generates a result from the noisy x tA noiseless x0 is generated. According to the conventional idea, the range-null space decomposition should be directly performed on the prediction result of each iteration of the consistency model, and then the noise is added for the next iteration, so that the decomposition of the null space and the decomposition of the range space at each step are based on the noiseless condition, but because of the fast iteration characteristics of the consistency model itself, if the conventional idea is followed, the inference result of the next consistency model is the null space refinement result of the last step, and the role of iterative sampling is not fully played, and the final result will produce artifacts. Therefore, the sampling process is redesigned in the application, the result predicted by the consistency model is first added with noise, and then the null space refinement is performed, the randomness is introduced through the range-null space decomposition under different noise scales, and the consistency model at each step will perform inference on the new target scale, fully play the role of iterative sampling, and produce high-quality results without artifacts.

[0020] In the application, in order to solve the problem that the average pooling super-resolution task and the image restoration under the condition of large mask produce poor results, a backtracking method is added in the scheme, and a specified number of inferences are performed before each calculation of the next step, so that a better "past" is generated at each step, and the global coordination of the image restoration task result is improved.

[0021] Based on the above method, the application further provides a consistency model image restoration system based on range-null space decomposition, comprising a memory and a processor; the memory stores a computer program, and when the computer program is executed by the processor, the method described above is realized.

[0022] The application introduces a consistency model to process the image restoration task through range-null space decomposition, designs a degradation matrix for different tasks to realize the processing of different image restoration tasks. The consistency model is introduced through range-null space, and the sampling process is designed according to the fast sampling characteristics of the consistency model to avoid generating artifacts. Because of the fast sampling characteristics of the consistency model itself, compared with other iterative generation models, the sampling time cost is greatly reduced. And through the range-null space decomposition method, the prior knowledge of the consistency model is fully utilized to generate high-quality results. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creating any inventive labor.

[0024] Figure 1 To generate artifacts without adjusting the order of noise addition and consistency model inference.

[0025] Figure 2 A flow chart of a consistency model image restoration method based on range-null space decomposition.

[0026] Figure 3 An example diagram for a deblurring task. DETAILED DESCRIPTION

[0027] The application will be further described in conjunction with the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the application are the general knowledge and common sense in the art, and the application does not have special restrictions.

[0028] Figure 3 In the formula, A'y represents the fixed range space at each step, DDNM represents the artifact result generated by iterating the unadjusted noise sequence, and CNRM represents the high-quality result generated after adjusting the sequence and adding noise.

[0029] The application discloses a consistency model image restoration method based on range-null space decomposition, which comprises the following steps: iterative sampling is performed through a consistency model, and prior knowledge of the consistency model is applied to an image restoration task by using null space refinement. In the iterative sampling process, a degradation matrix and its pseudo-inverse are constructed for different image restoration tasks, the image range space and the null space of different tasks are decomposed to decouple the consistency and authenticity of the image. The consistency of the image and the image to be repaired is ensured through the range space, and the prior knowledge of the consistency model is used through the null space refinement to ensure the authenticity of the result image. In each iteration, the image predicted by the consistency model is first subjected to noise processing, and then range-null space decomposition of the image is performed, so that the artifacts caused by the fast reasoning of the consistency model are avoided. A backtracking method is used to solve the problem of poor result authenticity in specific cases such as image restoration under the conditions of average pooling super-resolution task and large mask.

[0030] The first aspect of the application mainly optimizes that by range-null space decomposition of the degraded image, the pre-trained consistency model is introduced into different image restoration tasks by fixing the range space and iteratively refining the null space without additional training. According to the conventional idea, the range-null space decomposition should be directly performed on the prediction result of each iteration of the consistency model, and then the noise is added for the next iteration, so that the decomposition of the null space and the decomposition of the range space at each step are based on the noise-free condition. However, because of the rapid iteration characteristics of the consistency model itself, if the next consistency model inference result is the null space refinement result of the last step according to the conventional idea, the role of iterative sampling is not fully played, and the final result will produce artifacts. Therefore, the sampling process is redesigned in the application, the noise is added to the prediction result of the consistency model at each iteration, and then the null space refinement is performed, and the randomness is introduced by range-null space decomposition under different noise scales. Each step of the consistency model will perform inference on a new target scale, fully play the role of iterative sampling, and produce high-quality results without artifacts.

[0031]

[0032] The second aspect of the application is for average pooling super-resolution tasks, image inpainting tasks under large mask conditions, and the like, and the resulting results are poor. In these cases, the range space content A'y is too local and cannot guide the iterative inference process of the consistency model to produce globally coordinated results. It is considered that a backtracking method can be introduced, and before calculating the next inference result, a specified number of inferences are performed, so that a better "past" is produced at each step, and a more coordinated "future" is produced. In order to facilitate use, an additional hyperparameter l is allocated to control the number of repetitions at each backtracking.

[0033]

[0034]

[0035] The third aspect of the application is to design corresponding degradation matrix A and corresponding pseudo-inverse matrix A' for different image restoration tasks, and the degradation matrix and the pseudo-inverse matrix satisfy A=AA'A. Many tasks can be easily constructed manually without the help of complex Fourier transform or SVD. The following mainly introduces three examples: for the image coloring task, A can be designed as a pixel operator It converts each RGB channel pixel [R G B] to a gray value The pseudo-inverse A' is manually constructed as

[111] . For the n-fold image super-resolution task, the degradation matrix can be constructed as an average pooling operation, And Thus, the n*n block of the picture can be average-pooled into a value, and the application can construct the pseudo-inverse A' = [1...1] T And Considering that A is a composite operation composed of many sub-operations, that is, A = A1...A n , the corresponding pseudo-inverse matrix A' = A n '...A1' can be constructed. This can actually be used to solve the complex degenerate image restoration task, such as converting the complex image restoration task into a combination of several degenerate operations, and then finding the corresponding pseudo-inverse to achieve the repair of the complex image restoration task.

[0036] In the first aspect of the application, the degraded image is decomposed by range-null space, the range space is fixed, and the null space is iteratively refined, without additional training, to introduce a pre-trained consistency model into different image restoration tasks. The second aspect uses a backtracking method to produce a globally coordinated result to address the problem of poor results produced by the average pooling super-resolution task and the image repair task under the condition of a large mask. The third aspect can handle complex image restoration tasks by reasonably designing the degradation matrix and the pseudo-inverse of the degradation matrix.

[0037] The application provides a sampling method for deblurring image restoration tasks. Similarly, for different tasks, the degradation matrix and the inverse of the degradation matrix can be manually constructed to achieve processing of different image restoration tasks. The overall iterative process is as shown in Figure 2 , and the technical details are as follows:

[0038] Step 1: Construct a blur degradation matrix for three blur scenarios, including Gaussian blur, uniform blur, and non-anisotropic blur, which are defined as follows:

[0039] · Gaussian blur: a method of image blur processing using a Gaussian function. The characteristic of Gaussian blur is that the blur degree is continuously changed in space, which can generate a smooth blur effect on the image. Its mathematical expression is where σ is the standard deviation of the Gaussian function, which determines the degree of blur. In the actual degradation process, the Gaussian kernel blur kernel size is set to 5 and the Gaussian kernel width is set to 10.

[0040] · Uniform blur: also known as average blur, it is a method of blur processing by taking the average value of each pixel and its neighborhood pixels in the image. A fixed size blur kernel is usually used, and all elements in the kernel have equal values. Its mathematical expression is where I(x,y) is the original image, and k is half the size of the convolution kernel. In the actual degradation process, the convolution kernel size is set to 9.

[0041] • Anisotropic blur: Unlike isotropic blur (e.g., Gaussian blur), anisotropic blur is not uniform in the degree of blurring in each direction. This method is often used to preserve edges and important details in an image while applying a stronger blur in other areas. In the actual degradation process, the blur kernel size is 9, and the kernel width of each axis is 20 and 1, respectively.

[0042] Step two, based on the degradation matrix (blur kernel) of the three blur scenarios, the pseudo-inverse of the degradation matrix in the three scenarios is calculated respectively using the singular value decomposition method. The pseudo-inverse is calculated by singular value decomposition. First, the matrix A is decomposed into A = UΣV T , where U is an m x m orthogonal matrix, Σ is an m x n diagonal matrix, and V is an n x n orthogonal matrix. Then, construct the pseudo-inverse matrix Σ', take the reciprocal of the non-zero singular values of the diagonal matrix Σ and construct an n x m matrix Σ'. For non-zero singular values, calculate their reciprocal and place them in the corresponding position of Σ'. The rest is filled with zeros. Finally, use the results of SVD and the constructed Σ' to calculate the pseudo-inverse matrix A' = VΣ'U T .

[0043] Step three, select LSUN-Bedroom 256x256 and LSUN-Cat 256x256 datasets respectively for blur operation. The degraded images of the example are as follows Figure 3 The second column. The pre-trained consistency model used is based on two datasets respectively ([5] Yang Song et al. “Consistency Models”. In: ICML. Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 32211-32252.).

[0044] Step four, initialize the image and set the iteration parameters T = 9, U = σ T ≥ σ T-1 ≥…≥ σ1 = ∈, ρ = 7, ∈ = 0.002, U = 80.0. Start the iteration process, assuming that the intermediate output at this time is x t , according to Algorithm 1 to perform image restoration process, details as follows:

[0045] • Use the consistency model to predict the corresponding noise-free image x t according to the intermediate output x 0|t

[0046] • Add noise, the target distribution is Obtain the result Make it conform to the noise distribution of the time point trajectory, distinguish from other similar generation model sampling methods, and put the noise in the range-zero space decomposition after each iteration process.

[0047] · Range-zero space decomposition is performed, the A' fixed range space is used, the null space is refined, and the result x of the next step is obtained t-1

[0048] Step five, finally obtain x0 as the final image restoration result

[0049] The present application provides a construction system of degradation matrix and its pseudo-inverse for different image restoration tasks, so that the sampling method of the application can be applied to different image restoration fields, as follows:

[0050] · n times super-resolution task: And So the n x n block of the picture can be averaged into a value, and the pseudo-inverse A' = [1...1] can be constructed in the same way. T And

[0051] · Deblurring task: use the blur kernel corresponding to Gaussian blur, uniform blur and anisotropic blur as the degradation matrix, and use the singular value decomposition method to obtain the corresponding pseudo-inverse.

[0052] · Mask-based image inpainting task: use text mask, random pixel mask and hand-drawn mask as the degradation matrix (mask part is 0 and other part is 1), for the mask-based image inpainting task, the degradation matrix and its pseudo-inverse are equal.

[0053] · Coloring task: A can be designed as a pixel operator It converts each RGB channel pixel [R G B] to a gray value The pseudo-inverse A' is manually constructed as [1 1 1]

[0054] The protection content of the present application is not limited to the above embodiments. Changes and advantages that can be thought of by those skilled in the art without departing from the spirit and scope of the application are included in the present application, and are protected by the appended claims.

Claims

1. A consistency model image restoration method based on range-null space decomposition, characterized in that, The method comprises the following steps: Step 1: initialize related parameters, define the degradation image as , and the target image as ; abstract the degradation process of the image in the image restoration task into a degradation matrix , and find the pseudo-inverse of the matrix ; the degradation matrix is designed according to different image restoration tasks, and the pseudo-inverse matrix is obtained by using the singular value decomposition method, and the pseudo-inverse matrix satisfies the property ; Step 2: denoising the degraded image adding noise, a result of the adding noise to make it satisfy a noise distribution of a consistency model; a variance of noise of each step of adding noise, representing a minimum variance of noise of adding noise Step 3: Start iterative process, use consistency model on noisy degraded image Perform denoising, produce preliminary prediction for target image ;​ Step 4: refining the initial prediction noisy, noisy result to make it satisfy the noise distribution of the consistency model; Step 5: On the result of the noise addition Perform range-null space decomposition to get the result , As the start of the next iteration process Continue the iteration process; Indicates the pseudo-inverse matrix; the range-null space decomposition decouples consistency and authenticity, uses the fixed range space to ensure consistency, and uses the iteration of the consistency model to ensure the authenticity of the degenerated image; The method further comprises: Step 6: Loop ends, return As a result of denoising.

2. The method of claim 1, wherein, The initialized parameters include: setting the number of iterations and the time point of each iteration, and the total number of iterations is a noise variance sequence ; pre-training a consistency model , wherein the input represents a picture that needs to be denoised, represents the noise intensity of the picture; the noise uses Gaussian noise, and the output result is a denoised picture.

3. The method of claim 1, wherein, The image degradation process is abstracted as where, denotes the target image, denotes the degraded image, denotes the degradation matrix for different image restoration tasks: for n times super-resolution task, is designed as an average pooling operation, with values .

4. The method of claim 1, wherein, The strength of each added noise depends on the current time point As the time point decreases, the noise gradually decreases, and the mean of the noise is always 0, .

5. The method of claim 1, wherein, Range - null space decomposition results Thus, the resulting results are guaranteed Satisfy Thus, the consistency of the iteration results with respect to the degenerate image is maintained.

6. The method of claim 1, wherein, Adding backtracking: each time before calculating the next step, the specified number of inferences are performed, so that each step produces a better "past", improving the global coordination of the image restoration task results. Comprise:

7. A range-null space decomposition based consistency model image restoration system, comprising: A memory and a processor; The memory has stored thereon a computer program which, when executed by the processor, implements the method of any one of claims 1-6. ​

Images