A high-resolution gray-scale image layering coloring method based on zero-value domain decomposition

Abstract

This invention discloses a high-resolution grayscale image layered colorization method based on zero-domain decomposition. This method employs a layered image processing strategy. First, the input high-resolution grayscale image is downsampled, and the downsampled result is colorized using a text image model to obtain a low-resolution color result. Then, detail information from the high-resolution grayscale image and color information from the low-resolution color result are extracted based on zero-domain decomposition. Finally, the extracted color information is corrected using prior knowledge from a diffusion model, while the extracted detail information is injected to obtain the high-resolution color result. This invention, based on zero-domain decomposition, proposes color image information decomposition and diffusion model-based color information correction in the grayscale image layered colorization process, which can be used to generate color results highly consistent with the detail information in the high-resolution grayscale image. The color image information decomposition proposed in this invention is easily integrated with existing layered grayscale image colorization methods.

Description

Technical Field

[0001] This invention relates to the field of image colorization, and mainly to a high-resolution grayscale image layer colorization method based on zero-range decomposition. Background Technology

[0002] Grayscale image colorization has always been a hot topic in computer vision, involving adding color to grayscale images to produce more vivid outputs. Grayscale image colorization has wide applications, including colorizing old photographs, old movies, and line drawings. Current state-of-the-art grayscale image colorization methods leverage the powerful prior knowledge of text-to-image models trained on large-scale datasets to predict the color distribution of semantic information in grayscale images and use semantic information from text prompts for colorization. However, the use of text-to-image models places very high demands on computational resources. As the image size increases, the required GPU memory also increases. Therefore, these methods face challenges in processing high-resolution images.

[0003] Methods to reduce the computational resource requirements of text-to-image models mainly focus on improving model structure and adjusting processing strategies. For improving model structure, the latent diffusion model (Rombach R, Blattmann A, Lorenz D, Esser Pand Ommer B. High-resolution image synthesis with latent diffusion modes. In CVPR, 2022) transfers the diffusion process to the latent space of a variational autoencoder, reducing the computational resource requirements by lowering the dimensionality of the data to be processed. While this method is effective, the improved network still has excessively high computational resource requirements. It cannot directly process high-resolution images under limited computational resources. Regarding adjusting processing strategies (Jiménez...),... Barbero (Mixture of diffusers for scene composition and high resolution image generation. In arXiv, 2023) employs a noise-weighted fusion method based on block noise prediction, reducing resource requirements by decreasing the data size the network needs to process each time. However, block noise prediction suffers from a limited receptive field, resulting in a lack of global color uniformity in the final result. In practical applications, existing methods generally adopt a layered image colorization strategy based on super-resolution, reducing computational resource requirements by decreasing the image size that the text image model needs to process. These methods first generate a low-resolution color result, and then use an advanced super-resolution model to enlarge it to obtain a high-resolution color result. However, these methods neglect the utilization of detail information in high-resolution grayscale images, resulting in significant differences in detail between the generated color image and the original grayscale image. Furthermore, existing super-resolution models produce incorrect detail information when processing specific objects, such as faces and text. Summary of the Invention

[0004] Objective: To address the problems existing in the background technology, this invention provides a high-resolution grayscale image layered colorization method based on zero-domain decomposition. This method, based on zero-domain decomposition, decomposes image information into grayscale detail information and color information. It then fuses the detail information from the high-resolution grayscale image with the color information from the low-resolution color result to generate a high-resolution color image. The main steps are as follows: Based on zero-domain decomposition, detail information is extracted from the high-resolution grayscale image, and color information is extracted from the low-resolution color result. Prior knowledge from a diffusion model is used to correct the extracted color information. Then, the extracted detail information is injected, and the high-resolution color image is generated by iteratively executing the denoising process of the diffusion model.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A high-resolution grayscale image layering colorization method based on zero-range decomposition includes the following steps:

[0007] Step S1: Input a high-resolution grayscale image y;

[0008] Step S2: Downsample the high-resolution grayscale image y, and use a text image model to colorize the downsampled result to obtain a low-resolution color result x. lr ;

[0009] Step S3: Extract detailed information from the high-resolution grayscale image y and the low-resolution color result x based on zero-range decomposition. lrColor information in;

[0010] Step S4: Introduce a diffusion model. During the denoising process of the diffusion model, use the predicted clean image to evaluate the low-resolution color result x. lr The color information in the image is corrected, and the detail information in the high-resolution grayscale image y is injected to obtain a new, clean prediction image.

[0011] Step S5: Determine whether the denoising process of the diffusion model is complete. If not, add noise according to the principle of the diffusion model and return to step S4. If complete, proceed to step S6.

[0012] Step S6: Output high-resolution color results.

[0013] Preferably, the implementation process of step S2 is as follows:

[0014] Step S2.1: Use image average pooling to downsample the high-resolution grayscale image to obtain a low-resolution grayscale image y. lr , is represented as:

[0015] y lr =A SR y

[0016] Where A sR For image average pooling downsampling operation, y is the high-resolution grayscale image;

[0017] Step S2.2: Convert the low-resolution grayscale image y lr In addition to textual prompts containing color descriptions as conditions, the text-image model is input to generate a low-resolution color result x. lr The noise prediction process used by the text image model in the denoising stage can be represented as follows:

[0018]

[0019] in It is a variable, ∈ θ This is the noise prediction network for the text image model, where θ is the network parameter and x is the noise prediction network. t Let t be the latent spatial variable of the text-image model at the current time step. This indicates that the condition is an empty set, i.e., unconditional input. The noise is predicted based on unconditional input, where s is a scaling factor and c is the condition, i.e., the low-resolution grayscale image y. lr And text prompts containing color descriptions, ∈ θ (x t ,t,c) is noise obtained based on conditional prediction;

[0020] Iteration yields low-resolution color results xlr :

[0021]

[0022] The Scheduler is the scheduling strategy used by the text-image model in the denoising stage. When t-1 = 0, the output x0 is used as the low-resolution color result x. lr .

[0023] Preferably, the implementation process of step S3 is as follows:

[0024] Step S3.1: Define the image coloring problem:

[0025] y′=A C x

[0026] Where y′ is the grayscale image, x is the color image, and A C It is the image grayscale conversion operator;

[0027] Step S3.2: Decompose the color image x based on zero-range decomposition:

[0028]

[0029] in For A C The generalized inverse operation, where I is the identity matrix;

[0030] satisfy as well as

[0031] Wherein, ≡ represents an identity; Viewed as projecting the color image x onto A C The range space; Viewed as projecting the color image x onto A C The zero space;

[0032] Because y′=A C x, therefore Consider the detailed information contained in the color image x. Consider the color information contained in the color image x;

[0033] Step S3.3: According to step 3.2, the process of extracting detail information Detail from the high-resolution grayscale image y is represented as follows:

[0034]

[0035] Step S3.4: According to step 3.2, the low-resolution color result x lrThe process of extracting color information (Color) from the data is represented as follows:

[0036]

[0037] Preferably, the implementation process of step S4 is as follows:

[0038] Step S4.1: Introduce a pre-trained pixel-level diffusion model. During the denoising process of the pixel-level diffusion model, adjust x′ according to the current time step t. t Noise prediction is performed to obtain the predicted noise ∈ θ ′(x′ t ,t),x′ t Let t be the latent spatial variable of the pixel-level diffusion model at the current time step, and t be the current time step. Using ∈ θ ′(x t ,t) for x′ t Denoising is performed to obtain the predicted clean image x. 0|t , is represented as:

[0039]

[0040] Where α t and σ t It is the scaling factor, ∈ θ ′ is the noise prediction network of the pixel-level diffusion model described above;

[0041] Step S4.2: From the predicted clean image x 0|t Extracting color information x|t :

[0042]

[0043] Step S4.3: Based on zero-range decomposition, utilize color information (Color) x|t The Color obtained in step S3.4 is corrected to obtain high-resolution color information. This process can be represented as:

[0044]

[0045] in This is the image mean upsampling operator; For A SR Generalized inverse operation;

[0046] Step S4.4: Based on zero-range decomposition, inject the detail information obtained in step S3.3 into the high-resolution color information. In the process, a new predicted clean image is obtained. The process is represented as:

[0047]

[0048] Preferably, in step S5, it is determined whether the denoising process of the diffusion model has been completed. If not, the denoising process is implemented according to the principle of the diffusion model as follows:

[0049]

[0050] Where α t-1 and σ t-1 It is the scaling factor, η t It is an adjustable noise fusion factor, where ∈ is the noise sampled from a standard Gaussian distribution.

[0051] Beneficial effects:

[0052] (1) This invention proposes a color image information decomposition method based on zero-range decomposition for the layered colorization process of high-resolution grayscale images, decomposing the color image into grayscale detail information and color information. In existing layered image colorization methods, grayscale detail information is only used in the generation stage of low-resolution color results, while high-resolution color results are generated entirely by the super-resolution model based on the low-resolution color results. This results in significant differences in detail between the final high-resolution color results and the input high-resolution grayscale image. In contrast, the color image decomposition method proposed in this invention based on range null space can extract the detail information of high-resolution grayscale images and use it in the image layered colorization process. Combined with the correction of color information by the diffusion model, it avoids the inconsistency in detail between the high-resolution color results and the high-resolution grayscale image.

[0053] (2) The color image decomposition based on zero-range decomposition proposed in this invention can be easily applied to existing layered image coloring methods. Because these methods do not utilize the detailed information in the high-resolution grayscale image during high-resolution color image generation, the generated color results contain erroneous grayscale detail information. The color image decomposition proposed in this invention can extract the color information. By fusing the extracted color information with the detailed information in the high-resolution grayscale image to generate a high-resolution color image, the inconsistency in detail between the high-resolution color result and the high-resolution grayscale image can also be reduced. Attached Figure Description

[0054] Figure 1 This is a flowchart of the high-resolution grayscale image layering and colorization method based on zero-domain decomposition provided by the present invention;

[0055] Figure 2 This is a simplified algorithm framework diagram for generating low-resolution color results from high-resolution grayscale images provided by the present invention;

[0056] Figure 3 This is a simplified algorithm framework diagram for extracting grayscale detail information and color information based on color image decomposition provided by the present invention;

[0057] Figure 4 This is a simplified algorithm framework diagram provided by the present invention for correcting color information in low-resolution color results using a predicted clean image. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] This invention provides a high-resolution grayscale image layering and colorization method based on zero-range decomposition, the specific principle of which is as follows: Figure 1 As shown, it includes the following steps:

[0060] Step S1: Input a high-resolution grayscale image;

[0061] Step S2: Downsample the high-resolution grayscale image, and use a text image model to colorize the downsampled result to obtain a low-resolution color result. Specifically, refer to... Figure 2 .

[0062] Step S2.1: The high-resolution grayscale image is processed by image average pooling downsampling to obtain a low-resolution grayscale image y. lr , is represented as:

[0063] y lr =A SR y

[0064] Where A SR For image average pooling downsampling operation, y is the high-resolution grayscale image;

[0065] Step S2.2: Convert the low-resolution grayscale image y lr And text-image models that take color-description text as conditional input generate low-resolution color results x. lr The noise prediction process used by the text image model in the denoising stage can be represented as follows:

[0066]

[0067] Where ∈ θ This is the noise prediction network for the text image model, where t is the current time step and x is the noise prediction network. t These are the latent spatial variables of the text-image model at the current time step. This indicates that the conditional input is an empty set. The noise is based on unconditional prediction, s is an adjustable scaling factor, and c is the low-resolution grayscale image y in the conditional input. lr And the condition variable obtained from the text hint containing color description, ∈ θ (x t ,t,c) is noise obtained based on conditional prediction;

[0068] Based on the predicted noise The following process is performed iteratively to obtain a low-resolution color result x. lr :

[0069]

[0070] The Scheduler is the scheduling strategy used in the text-image denoising process. When t-1 = 0, the output x0 is used as the low-resolution color result x. lr ;

[0071] Step S3: Extract detail information from the high-resolution grayscale image and color information from the low-resolution color result based on zero-domain decomposition. Specifically, refer to... Figure 3 .

[0072] Step S3.1: Define the image coloring problem:

[0073] y = A C x

[0074] Where y is the grayscale image, x is the color image, and A C It is the image grayscale conversion operator;

[0075] A C The matrix form can be represented as Its function is to convert the pixel values ​​[r, g, b] of the RGB channels. T Convert to grayscale

[0076] Step S3.2: Decompose x based on zero-range decomposition:

[0077]

[0078] in For A C The generalized inverse operation, where I is the identity matrix;

[0079] The matrix form can be represented as [1, 1, 1];

[0080] satisfy as well as

[0081] Wherein, ≡ represents an identity; Viewed as projecting x onto A C The range space; Viewed as projecting x onto A C The zero space;

[0082] Because y = A C x, therefore Consider the grayscale detail information contained in x. Consider the color information contained in x;

[0083] Step S3.3, the extraction process of detail information from the high-resolution grayscale image, is represented as follows:

[0084]

[0085] Step S3.4, the process of extracting color information (Color) from the low-resolution color results, is represented as follows:

[0086]

[0087] Where x lr This is the low-resolution color result obtained based on step S2.

[0088] Step S4: In the denoising process of the diffusion model, the color information in the low-resolution color result is corrected using the predicted clean image, and the detail information from the high-resolution grayscale image is injected to obtain a new predicted clean image. Specifically, refer to... Figure 4 .

[0089] Step S4.1: Introduce a pre-trained pixel-level diffusion model. During the denoising process of the pixel-level diffusion model, the noisy data x is processed according to the current time step t. t Perform noise prediction to obtain the predicted noise ∈ θ (x t ,t), using ∈ θ (x t ,t) for x t Denoising is performed to obtain the predicted clean image x. 0|t Represented as:

[0090]

[0091] Where α t and σ t It is a predefined scaling factor, ∈ θ It is the noise prediction network of the pixel-level diffusion model described above;

[0092] Step S4.2, from x 0|t Extracting color information x|t :

[0093]

[0094] Step S4.3: Based on zero-range decomposition, using Color... x|t Information correction is performed on the color to obtain high-resolution color information. This process can be represented as:

[0095]

[0096] in This is the image mean upsampling operator; For A SR Generalized inverse operation;

[0097] Step S4.4: Based on the zero-range decomposition, inject the detail information obtained in step S3 into... In the process, a new predicted clean image is obtained. The process is represented as:

[0098]

[0099] Step S5: Determine whether the denoising process of the diffusion model is complete. If not, add noise according to the principle of the diffusion model and return to step S4. The noise addition process is implemented as follows:

[0100]

[0101] Where α t-1 and σ t-1 It is a pre-set scaling factor, η t It is an adjustable noise fusion factor, where ∈ is the noise sampled from a standard Gaussian distribution;

[0102] Step S6: Output high-resolution color results.

[0103] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A high-resolution grayscale image layering and colorization method based on zero-range decomposition, characterized in that, Includes the following steps: Step S1: Input a high-resolution grayscale image y; Step S2: Downsample the high-resolution grayscale image y, and use a text image model to colorize the downsampled result to obtain a low-resolution color result. ; Step S3: Extract detailed information from the high-resolution grayscale image y and the low-resolution color result based on zero-range decomposition. Color information in; Step S4: Introduce a diffusion model. In the denoising process of the diffusion model, use the predicted clean image to evaluate the low-resolution color results. The color information in the image is corrected, and the detail information in the high-resolution grayscale image y is injected to obtain a new, clean prediction image. Step S5: Determine whether the denoising process of the diffusion model is complete. If not, add noise according to the principle of the diffusion model. If complete, proceed to step S6. Step S6: Output high-resolution color results; The implementation process of step S2 is as follows: Step S2.1: Use image average pooling to downsample the high-resolution grayscale image y to obtain a low-resolution grayscale image. , represented as: ， in This is an image average pooling downsampling operation; Step S2.2: Convert the low-resolution grayscale image... The text image model is input with textual cues containing color descriptions as conditions to generate low-resolution color results. The noise prediction process used by the text image model in the denoising stage can be represented as follows: ， in It is a variable. It is the noise prediction network of the text image model. These are the latent spatial variables of the text-image model at the current time step. It is the current time step. This indicates that the condition is an empty set, i.e., unconditional input. The noise is obtained based on unconditional input prediction. It is the scaling factor, and c is the condition, namely the low-resolution grayscale image. And text prompts containing color descriptions, It is noise obtained based on conditional prediction; Iteration yields low-resolution color results : ， in This is the scheduling strategy used by the text image model in the denoising stage. When, output As a low-resolution color result ; The implementation process of step S3 is as follows: Step S3.1: Define the image coloring problem: ， in It is a grayscale image. It is a color image. It is the image grayscale conversion operator; Step S3.2: Decompose the color image based on the zero-range decomposition. Disassembly: ， in for Generalized inverse operation, It is the identity matrix; satisfy as well as ; in, It means that it is always equal to; View as a color image Projected to The range space; View as a color image Projected to The zero space; because Therefore, Viewed as a color image The details contained therein Viewed as a color image The color information contained therein; Step S3.3: According to step 3.2, high-resolution grayscale image Detailed information The extraction process is represented as follows: ， Step S3.4: Based on step 3.2, low-resolution color results. The process of extracting color information (Color) from the data is represented as follows: ； The implementation process of step S4 is as follows: Step S4.1: Introduce a pre-trained pixel-level diffusion model. During the denoising process of the pixel-level diffusion model, based on the current time step... right Noise prediction is performed to obtain predicted noise. , These are the latent spatial variables of the pixel-level diffusion model at the current time step. It is the current time step, using right Denoising is performed to obtain a clean predicted image. , represented as: ， in and It is a scaling factor. It is the noise prediction network of the pixel-level diffusion model described above; Step S4.2: From the predicted clean image Extract color information : ， Step S4.3: Based on zero-range decomposition, utilize color information The result obtained in step S3.4 Information correction is performed to obtain high-resolution color information. The process is represented as: ， in This is the image mean upsampling operator; for Generalized inverse operation; Step S4.4: Based on zero-range decomposition, inject the detail information obtained in step S3.3 into the high-resolution color information. In the process, a new predicted clean image is obtained. The process is represented as: 。 2. The high-resolution grayscale image layering and colorization method based on zero-range decomposition according to claim 1, characterized in that, In step S5, it is determined whether the denoising process of the diffusion model has been completed. If not, the denoising process is implemented according to the principle of the diffusion model as follows: ， in and It is a scaling factor. It is an adjustable noise fusion factor. It is noise sampled from a standard Gaussian distribution.

Images