Structure sensing image super-resolution method

Abstract

The invention discloses an image super-resolution method based on structure perception. The method is based on an existing image super-resolution diffusion model. Firstly, the whole reasoning process is divided into an early stage and an intervention stage. In an early stage, a reconstructed image is screened based on the structural similarity with a low-resolution image, and a clear image with the most similar structure is obtained as structure embedding. In the intervention stage, through control condition embedding, the reasoning process is jointly guided by structure embedding and low-resolution images, and generation of high-fidelity structure information is promoted. Meanwhile, through image detail embedding, structure embedding is inserted into a clean reconstructed image in each time step according to the degradation degree of a low-resolution image, and possible false details are inhibited. According to the method, control condition embedding and image detail embedding in the inference process of the image super-resolution diffusion model are provided based on the structure information of the reconstructed image in the early stage of the inference process, and the method can be used for generating the reconstructed image with high fidelity of the structure and fewer false details.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution, and mainly to a structure-aware image super-resolution method. Background Art

[0002] Image super-resolution has always been a hot topic in computer vision, which involves extracting information from low-resolution images and reconstructing high-resolution images with clear structures. Image super-resolution has a wide range of applications, including medical diagnosis, unmanned driving, image and video compression, etc. The latest image super-resolution methods use the powerful prior knowledge of the text-to-image diffusion model trained on large-scale datasets to predict the missing details and existing blur, noise and other problems in low-resolution images, and use the semantic information extracted from the low-resolution images to fill in the details and remove blur, noise and other problems. However, the detail information generated by the text-to-image diffusion model is inconsistent with the semantics of the low-resolution image, so these methods are challenging to generate high-fidelity structures and details.

[0003] Methods to improve the fidelity of structure and details generated by image super-resolution diffusion models mainly study improving model structure and adjusting processing strategies. For improving model structure, PASD (Yang T, Wu R, Ren P, Xie X, Zhang L. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. In ECCV, 2024) introduces a pre-trained text extractor, and uses a pixel-level attention cross module to extract text cues from low-resolution images for high-resolution image reconstruction, generating detail information that is more consistent with the semantics of low-resolution images. Although this method is effective, the introduced pre-trained text extractor cannot identify problems such as blur and noise in low-resolution images. When the low-resolution image is severely damaged, semantically incorrect details will be generated. SeeSR (Wu R, Yang T, Sun L, Zhang Z, Li S, Zhang L. Seesr: Towards semantics-aware real-world image super-resolution. In CVPR, 2024) uses data with different degrees of blur and noise problems to train a degradation-aware text extractor, and uses the trained text extractor to provide text hints and semantic features for image reconstruction, ensuring that correct semantic details can be generated when the low-resolution image is severely damaged. For the adjustment processing strategy, DiffBIR (Lin X, He J, Chen Z, Lyu Z, Dai B, Yu F, Ouyang W, Qiao Y, Dong C. Diffbir: Towards blind image restoration with generative diffusion prior. InarXiv, 2023) adopts a two-stage processing method. By introducing a preprocessing stage, a generative adversarial network model for dealing with blur, noise and other problems is used to pre-restore the low-resolution image. In the detail reconstruction stage, the prior knowledge of the text image diffusion model is used to fill in the missing detail information of the restored image. However, the model used in the preprocessing stage needs to be specially trained, which leads to an increase in training costs. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a structure-aware image super-resolution method, which is based on the existing image super-resolution diffusion model, and screens out reconstructed images with clear structures from the early stage to intervene in the reasoning process and generate high-fidelity structures and details. The main approach is that, in the early stage, the structural embedding is obtained based on the structural similarity with the low-resolution image. In the intervention stage, the generation of high-fidelity structural information is promoted by controlling the conditional embedding. At the same time, through the image detail embedding, according to the degree of degradation of the low-resolution image, at each time step, the structural embedding is inserted into the clean reconstructed image to suppress possible false details.

[0005] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A structure-aware image super-resolution method comprises the following steps:

[0007] Step S1: Input low-resolution image I LR ;

[0008] Step S2: Divide the entire reasoning process into the early stage T E and intervention phase T I ;

[0009] Step S3: At the early stage T E , based on the low-resolution image I LR The structural similarity of the reconstructed image is screened and the clear image with the most similar structure is obtained as the structural embedding Z SE ;

[0010] Step S4: In the intervention phase T I , by controlling the conditional embedding, using the structure embedding Z SE Together with low-resolution images, it guides the reasoning process and promotes the generation of high-fidelity structural information;

[0011] Step S5: In the intervention phase T I , through image detail embedding, according to the low-resolution image I LR The degree of degradation, at each time step t, embeds the structure into Z SE Inserted into the clean reconstructed image to suppress possible false details;

[0012] Step S6, judging whether the whole reasoning process is completed;

[0013] Step S7, outputting a reconstructed high-resolution image;

[0014] Preferably, the implementation process of step S2 is:

[0015] Step S2.1: According to the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage T E and intervention phase T I , expressed as:

[0016] T=T E +T I

[0017] Where T is the number of inference steps predefined by the existing image super-resolution diffusion model;

[0018] Preferably, the implementation process of step S3 is:

[0019] Step S3.1, define the noise prediction process of the existing image super-resolution diffusion model:

[0020] ∈ t =∈ θ (Z t ,ε(I LR ),t)

[0021] Where t is the current time step, ∈ t is the prediction noise at time step t, ∈ θ is the noise prediction network of the existing image super-resolution diffusion model, Z t is the noisy latent spatial image at time step t, ε is the encoder of the existing image super-resolution diffusion model, ε(I LR ) is the low-resolution image I LR The corresponding latent space image;

[0022] Step S3.2: According to step S3.1, based on the predicted noise ∈ at the time step t t For the noisy latent space image Z at time step t t To denoise:

[0023] Z 0|t =Sampler(Z t ,∈ t ,t)

[0024] Where Z 0|t is the predicted clean latent space image at time step t, Sampler is the sampler of the existing image super-resolution diffusion model;

[0025] Step S3.3: According to step 3.2, the reconstructed image I at time step t t It is expressed as:

[0026]

[0027] in It is the decoder of the existing image super-resolution diffusion model;

[0028] Step S3.4: According to step 3.3, the reconstructed image I at time step t t Compared with the low-resolution image I LR The calculation process of structural similarity is expressed as:

[0029] S t =SSIM(I t ,SR(I LR ))

[0030] Where S t is the structural similarity result at time step t, SSIM is a public structural similarity index algorithm, SR(I LR ) is to adjust the low-resolution image I LR To reconstruct the image I t The size is consistent;

[0031] Step S3.5: For the early stage T E At each time step t, perform steps S3.1-3.4 to obtain the structural similarity result S t Store it in the container Buffer and filter out the maximum value S max , judge S t With S max Are they consistent? If they are consistent, then the predicted clean latent space image Z at the time step t is 0|t Embed Z as a structure SE ;

[0032] Preferably, the implementation process of step S4 is:

[0033] Step S4.1: During the intervention phase T I , using the structure described to embed Z SE For each time step t, the noisy latent space image Z t Noise prediction is performed, expressed as:

[0034]

[0035] in is the structural embedding Z at time step t SE The resulting prediction noise;

[0036] Step S4.2: Obtain the original predicted noise according to the noise prediction process of the existing image super-resolution diffusion model

[0037] Step S4.3: Using the data obtained in step S4.1 The original prediction noise obtained in step S4.2 Constraints are applied to obtain the noise that is actually removed It is expressed as:

[0038]

[0039] Where S max In the early stage E The maximum value of the structural similarity result obtained by the last calculation;

[0040] Step S4.4: According to step S4.3, use For the noisy latent space image Z at time step t t Denoising to get the predicted clean latent space image Z 0|t , expressed as:

[0041]

[0042] Preferably, the implementation process of step S5 is:

[0043] Step S5.1: Based on the low-resolution image I LR The degradation degree sets the image insertion ratio w t , expressed as:

[0044]

[0045] Step S5.2: According to step S5.1, in the intervention phase T I At each time step t, the structure is embedded in Z SE Inserted into the clean reconstructed image, it is expressed as:

[0046] Z' 0|t =w t Z SE +(1-w t )Z 0|t

[0047] where Z' 0|t The clean latent space image after intervention;

[0048] Preferably, in step S6, it is determined whether the entire reasoning process is completed. If it is not completed, noise is added according to the principle of the existing image super-resolution diffusion model, which is expressed as:

[0049]

[0050] Preferably, in step S7, the implementation process of outputting the high-resolution reconstructed image is:

[0051]

[0052] Among them IHR is the output high-resolution reconstructed image, and Z0 is the predicted clean latent space image obtained after the entire inference process, that is, when t = 0;

[0053] Beneficial effects:

[0054] (1) Based on the existing image super-resolution diffusion model, the present invention proposes a structure-aware reconstructed image screening in the early stage of the reasoning process, and uses the screened reconstructed image that is most consistent with the low-resolution image structure and clearer to embed control conditions and guide the subsequent reasoning process. In the reasoning process of the existing image super-resolution diffusion model, the information used for guidance comes from the structure and details of the low-resolution image and the semantic cues extracted therefrom, while the blur and noise in the low-resolution image may be over-interpreted by the model's generation prior, resulting in false detail information in the final reconstructed image. In contrast, the reconstructed image screened by the present invention is highly consistent with the low-resolution image structure, and contains less blur, noise and other problems, which can provide clearer guidance for the reasoning process and avoid the generation of false details.

[0055] (2) The structure-aware image detail embedding proposed in the present invention can be easily applied to existing diffusion models based on image super-resolution. Due to empirical priors and hallucinations, these models continuously generate false detail information during the reasoning process. The image detail embedding proposed in the present invention can use the empirical priors of the reconstructed image suppression model containing fewer false details to reduce the generation of false details in the subsequent reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the structure-aware image super-resolution method provided by the present invention;

[0057] Figure 2 It is a brief algorithm framework diagram of the structure-aware image super-resolution method provided by the present invention;

[0058] Figure 3 It is a brief algorithm framework diagram of structure-aware screening provided by the present invention;

[0059] Figure 4 It is a brief algorithm framework diagram of control condition embedding and image detail embedding provided by the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The present invention provides a high-resolution grayscale image layered coloring method based on zero-value domain decomposition. The specific principle is as follows: Figure 1 As shown, the following steps are included:

[0062] Step S1: Input low-resolution image I LR ;

[0063] Step S2: Divide the entire reasoning process into the early stage T E and intervention phase T I Specifically, refer to Figure 2 .

[0064] Step S2.1: According to the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage T E and intervention phase T I , expressed as:

[0065] T=T E +T I

[0066] Where T is the number of inference steps predefined by the existing image super-resolution diffusion model;

[0067] Step S3: At the early stage T E , based on the low-resolution image I LR The structural similarity of the reconstructed image is screened and the clear image with the most similar structure is obtained as the structural embedding Z SE Specifically, refer to Figure 3 .

[0068] Step S3.1, define the noise prediction process of the existing image super-resolution diffusion model:

[0069] ∈ t =∈ θ (Z t ,ε(I LR ),t)

[0070] Where t is the current time step, ∈ t is the prediction noise at time step t, ∈ θ is the noise prediction network of the existing image super-resolution diffusion model, Z t is the noisy latent spatial image at time step t, ε is the encoder of the existing image super-resolution diffusion model, ε(I LR ) is the low-resolution image I LR The corresponding latent space image;

[0071] Step S3.2: According to step S3.1, based on the predicted noise ∈ at the time step t t For the noisy latent space image Z at time step tt To denoise:

[0072] Z 0|t =Sampler(Z t ,∈ t ,t)

[0073] Where Z 0|t is the predicted clean latent space image at time step t, Sampler is the sampler of the existing image super-resolution diffusion model;

[0074] Step S3.3: According to step 3.2, the reconstructed image I at time step t t It is expressed as:

[0075]

[0076] in It is the decoder of the existing image super-resolution diffusion model;

[0077] Step S3.4: According to step 3.3, the reconstructed image I at time step t t Compared with the low-resolution image I LR The calculation process of structural similarity is expressed as:

[0078] S t =SSIM(I t ,SR(I LR ))

[0079] Where S t is the structural similarity result at time step t, SSIM is a public structural similarity index algorithm, SR(I LR ) is to adjust the low-resolution image I LR To reconstruct the image I t The size is consistent;

[0080] Step S3.5: For the early stage T E At each time step t, perform steps S3.1-3.4 to obtain the structural similarity result S t Store it in the container Buffer and filter out the maximum value S max , judge S t With S max Are they consistent? If they are consistent, then the predicted clean latent space image Z at the time step t is 0|t Embed Z as a structure SE .

[0081] Step S4: In the intervention phase T I , by controlling the conditional embedding, using the structure embedding Z SETogether with the low-resolution image, it guides the reasoning process and promotes the generation of high-fidelity structural information. Figure 4 .

[0082] Step S4.1: During the intervention phase T I , using the structure described to embed Z SE For each time step t, the noisy latent space image Z t Noise prediction is performed, expressed as:

[0083]

[0084] in is the structural embedding Z at time step t SE The predicted noise is obtained.

[0085] Step S4.2: Obtain the original predicted noise according to the noise prediction process of the existing image super-resolution diffusion model

[0086] Step S4.3: Using the data obtained in step S4.1 The original prediction noise obtained in step S4.2 Constraints are applied to obtain the noise that is actually removed It is expressed as:

[0087]

[0088] Where S max In the early stage E The maximum value of the structural similarity result obtained by the last calculation;

[0089] Step S4.4: According to step S4.3, use For the noisy latent space image Z at time step t t Denoising to get the predicted clean latent space image Z 0|t , expressed as:

[0090]

[0091] Step S5: In the intervention phase T I , through image detail embedding, according to the low-resolution image I LR The degree of degradation, at each time step t, embeds the structure into Z SE Inserted into the clean reconstructed image to suppress possible spurious details. Specifically, refer to Figure 4 .

[0092] Step S5.1: Based on the low-resolution image I LR The degradation degree sets the image insertion ratio w t , expressed as:

[0093]

[0094] Step S5.2: According to step S5.1, in the intervention phase T I At each time step t, the structure is embedded in Z SE Inserted into the clean reconstructed image, it is expressed as:

[0095] Z' 0|t =w t Z SE +(1-w t )Z 0|t

[0096] where Z' 0|t Clean latent space image after intervention;

[0097] Step S6: Determine whether the entire inference process is completed. If not completed, add noise according to the principle of the existing image super-resolution diffusion model, expressed as:

[0098]

[0099] The implementation process of step S7, outputting the reconstructed high-resolution image is as follows:

[0100]

[0101] Among them I HR is the output high-resolution reconstructed image, and Z0 is the predicted clean latent space image obtained when the entire reasoning process is completed, that is, t = 0.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A structure-aware image super-resolution method, characterized in that: The following steps are involved: Step S1: Input low-resolution image I LR ; Step S2: Divide the entire reasoning process into the early stage T E and intervention phase T I ; Step S3: At the early stage T E , based on the low-resolution image I LR The structural similarity of the reconstructed image is screened and the clear image with the most similar structure is obtained as the structural embedding Z SE ; Step S4: In the intervention phase T I , by controlling the conditional embedding, using the structure embedding Z SE Together with the low-resolution image, it guides the reasoning process and produces a clean reconstructed image. Step S5: In the intervention phase T I , through image detail embedding, according to the low-resolution image I LR The degree of degradation, at each time step t, embeds the structure into Z SE Inserting into the clean reconstructed image to suppress possible false details; Step S6, judging whether the whole reasoning process is completed; Step S7: output the reconstructed high-resolution image.

2. The structure-aware image super-resolution method according to claim 1, characterized in that: The implementation process of step S2 is: Step S2.1: According to the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage T E and intervention phase T I , expressed as: T=T E +T I Where T is the number of inference steps predefined by the existing image super-resolution diffusion model.

3. The structure-aware image super-resolution method according to claim 1, characterized in that: The implementation process of step S3 is: Step S3.1, define the noise prediction process of the existing image super-resolution diffusion model: ∈ t =∈ θ (Z t ,ε(I LR ),t) Where t is the current time step, ∈ t is the prediction noise at time step t, ∈ θ is the noise prediction network of the existing image super-resolution diffusion model, Z t is the noisy latent spatial image at time step t, ε is the encoder of the existing image super-resolution diffusion model, ε(I LR ) is the low-resolution image I LR The corresponding latent space image; Step S3.2: According to step S3.1, based on the predicted noise ∈ at the time step t t For the noisy latent space image Z at time step t t To denoise: Z 0|t =Sampler(Z t ,∈ t ,t) Where Z 0|t is the predicted clean latent space image at time step t, Sampler is the sampler of the existing image super-resolution diffusion model; Step S3.3: According to step 3.2, the reconstructed image I at time step t t It is expressed as: Where D is the decoder of the existing image super-resolution diffusion model; Step S3.4: According to step 3.3, the reconstructed image I at time step t t Compared with the low-resolution image I LR The calculation process of structural similarity is expressed as: S t =SSIM(I t ,SR(I LR )) Where S t is the structural similarity result at time step t, SSIM is a public structural similarity index algorithm, SR(I LR ) is to adjust the low-resolution image I LR To reconstruct the image I t The size is consistent; Step S3.5: For the early stage T E At each time step t, perform steps S3.1-3.4 to obtain the structural similarity result S t Store it in the container Buffer and filter out the maximum value S max , judge S t With S max Are they consistent? If they are consistent, then the predicted clean latent space image Z at the time step t is 0|t Embed Z as a structure SE .

4. The structure-aware image super-resolution method according to claim 3, characterized in that: The implementation process of step S4 is: Step S4.1: During the intervention phase T I , using the structure described to embed Z SE For each time step t, the noisy latent space image Z t Noise prediction is performed, expressed as: in is the structural embedding Z at time step t SE The resulting prediction noise; Step S4.2: Obtain the original predicted noise according to the noise prediction process of the existing image super-resolution diffusion model Step S4.3: Using the data obtained in step S4.1 The original prediction noise obtained in step S4.2 Constraints are applied to obtain the noise that is actually removed It is expressed as: Where S max In the early stage E The maximum value of the structural similarity result obtained by the last calculation; Step S4.4: According to step S4.3, use For the noisy latent space image Z at time step t t Denoising to get the predicted clean latent space image Z 0|t , expressed as:

5. The structure-aware image super-resolution method according to claim 4, characterized in that: The implementation process of step S5 is: Step S5.1: Based on the low-resolution image I LR The degradation degree sets the image insertion ratio w t , expressed as: Step S5.2: According to step S5.1, in the intervention phase T I At each time step t, the structure is embedded in Z SE Inserted into the clean reconstructed image, it is expressed as: WITH' 0|t =in t WITH SE +(1-in t )WITH 0|t where Z' 0|t Clean latent space image after intervention.

6. The structure-aware image super-resolution method according to claim 5, characterized in that: Step S6 determines whether the entire inference process is completed. If not, noise is added according to the principle of the existing image super-resolution diffusion model, which is expressed as:

7. The structure-aware image super-resolution method according to claim 6, characterized in that: The implementation process of step S7 is: Among them I HR is the output high-resolution reconstructed image, and Z0 is the predicted clean latent space image obtained when the entire reasoning process is completed, that is, t = 0.

Images