A structure-aware image super-resolution method

Abstract

This invention discloses a structure-aware image super-resolution method. This method is based on existing image super-resolution diffusion models. First, the entire inference process is divided into an early stage and an intervention stage. In the early stage, reconstructed images are selected based on their structural similarity to low-resolution images, and the clearest image with the most similar structure is used as the structure embedding. In the intervention stage, through controlled conditional embedding, the structure embedding and the low-resolution image jointly guide the inference process, promoting the generation of high-fidelity structural information. Simultaneously, through image detail embedding, according to the degradation degree of the low-resolution image, the structure embedding is inserted into the clean reconstructed image at each time step, suppressing possible false details. This invention, based on the structural information of the reconstructed image in the early stage of the inference process, proposes controlled conditional embedding and image detail embedding in the inference process of the image super-resolution diffusion model, which can be used to generate reconstructed images with high structural fidelity and fewer false details.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution, and mainly to a structure-aware image super-resolution method. Background Technology

[0002] Image super-resolution has always been a hot topic in computer vision, involving extracting information from low-resolution images and reconstructing high-resolution images with clear structure. Its applications are wide-ranging, including medical diagnosis, autonomous driving, and image and video compression. Current state-of-the-art image super-resolution methods leverage the powerful prior knowledge of text-to-image diffusion models trained on large-scale datasets to predict missing details and issues like blur and noise in low-resolution images, and then use semantic information extracted from the low-resolution image for detail filling and blur and noise removal. However, the detail information generated by text-to-image diffusion models often suffers from inconsistencies with the semantics of the low-resolution image, thus posing a challenge to generating high-fidelity structures and details.

[0003] Methods to improve the fidelity of generated structures and details in image super-resolution diffusion models mainly focus on improving model structure and adjusting processing strategies. Regarding improving model structure, PASD (Yang T, Wu R, Ren P, Xie X, Zhang L. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. In ECCV, 2024) introduces a pre-trained text extractor. Through pixel-level attention cross-modules, textual cues extracted from low-resolution images are used for high-resolution image reconstruction, generating detail information that is more semantically consistent with the low-resolution image. While this method is effective, the introduced pre-trained text extractor cannot recognize problems such as blurring and noise in low-resolution images. In cases where the low-resolution image is severely damaged, it may generate semantically incorrect details. SeeSR (Wu R, Yang T, Sun L, Zhang Z, Li S, Zhang L. SeeSR: Towards semantics-aware real-world image super-resolution. In CVPR, 2024) trains a degradation-aware text extractor using data with varying degrees of blur and noise. This trained text extractor provides textual cues and semantic features for image reconstruction, ensuring the generation of correct semantic details even in severely damaged low-resolution images. For adjusting processing strategies, DiffBIR (Lin X, He J, Chen Z, Lyu Z, Dai B, Yu F, Ouyang W, Qiao Y, Dong C. Diffbir: Towards blind image restoration with generative diffusion prior. In arXiv, 2023) employs a two-stage processing method. A preprocessing stage is introduced, using a generative adversarial network model designed to address blur and noise issues to pre-restore low-resolution images. In the detail reconstruction stage, prior knowledge from the text-image diffusion model is used to fill in the missing details in the restored image. However, the models used in the preprocessing stage require specialized training, which increases the training cost. Summary of the Invention

[0004] Objective: To address the problems existing in the aforementioned background technology, this invention provides a structure-aware image super-resolution method. This method, based on existing image super-resolution diffusion models, selects structurally clear reconstructed images from an early stage to intervene in the inference process and generate high-fidelity structures and details. The main approach is as follows: In the early stage, structural embeddings are obtained based on structural similarity to low-resolution images. In the intervention stage, conditional embedding is controlled to promote the generation of high-fidelity structural information. Simultaneously, through image detail embedding, based on the degradation level of the low-resolution image, structural embeddings are inserted into the clean reconstructed image at each time step, suppressing potentially generated false details.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A structure-aware image super-resolution method includes the following steps:

[0007] Step S1: Input low-resolution image I LR ;

[0008] Step S2: Divide the entire reasoning process into an early stage T. E and intervention phase T I ;

[0009] Step S3, in the early stage T E Based on the low-resolution image I LR The structural similarity of the reconstructed images is used to select the clearest image with the most structural similarity as the structural embedding Z. SE ;

[0010] Step S4, during the intervention phase T I By controlling the embedding conditions, Z is embedded using the aforementioned structure. SE Together with low-resolution images, they guide the reasoning process and promote the generation of high-fidelity structural information;

[0011] Step S5, during the intervention phase T I By embedding image details, based on low-resolution image I LR The degree of degradation, at each time step t, embeds the structure into Z. SE Inserted into a clean reconstructed image, suppressing any false details that may be introduced;

[0012] Step S6: Determine whether the entire reasoning process has been completed;

[0013] Step S7: Output the reconstructed high-resolution image;

[0014] Preferably, the implementation process of step S2 is as follows:

[0015] Step S2.1: Based on the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage T. E and intervention phase T I , is represented as:

[0016] T = T E +T I

[0017] Where T is the number of inference steps predefined in the existing image super-resolution diffusion model;

[0018] Preferably, the implementation process of step S3 is as follows:

[0019] Step S3.1: Define the noise prediction process for the existing image super-resolution diffusion model:

[0020] ∈ t =∈ θ (Z t ,ε(I LR ),t)

[0021] Where t is the current time step, ∈ t It is the prediction noise at time step t, ∈ θ It is a noise prediction network for existing image super-resolution diffusion models, Z t It is the noisy latent space image at time step t, ε is the encoder of the existing image super-resolution diffusion model, ε(I LR ) is the low-resolution image I LR The corresponding latent space image;

[0022] Step S3.2: According to step S3.1, based on the predicted noise at time step t, ∈ t For the noisy latent space image Z at time step t t Denoising:

[0023] Z 0|t =Sampler(Z) t ,∈ t ,t)

[0024] Z 0|t It is the predicted clean latent space image at time step t, and Sampler is the sampler of the existing image super-resolution diffusion model;

[0025] Step S3.3: According to step 3.2, the reconstructed image I at time step t. t Represented as:

[0026]

[0027] in It is a decoder for existing image super-resolution diffusion models;

[0028] Step S3.4: Based on step 3.3, the reconstructed image I at time step t. t With the low-resolution image I LR The calculation process for structural similarity is expressed as follows:

[0029] S t =SSIM(I t ,SR(I LR ))

[0030] Where S t This is the structural similarity result at time step t. SSIM is a publicly available structural similarity index algorithm. SR(I) LR ) is to adjust the low-resolution image I. LR To and reconstruct image I t The dimensions are consistent;

[0031] Step S3.5, for the early stage T E At each time step t, perform the structural similarity results S obtained in steps S3.1-3.4. t Store the data in a container Buffer, and then filter out the maximum value S. max Determine S t With S max If they are consistent, then the predicted clean latent space image Z at time step t is obtained. 0|t As a structural embedding Z SE ;

[0032] Preferably, the implementation process of step S4 is as follows:

[0033] Step S4.1, during the intervention phase T I Z is embedded using the aforementioned structure. SE The noisy latent space image Z at each time step t t Noise prediction is represented as:

[0034]

[0035] in It is based on the structure embedding Z at time step t. SE The obtained prediction noise;

[0036] Step S4.2: Obtain the original predicted noise based on the noise prediction process of the existing image super-resolution diffusion model.

[0037] Step S4.3: Using the information obtained in step S4.1 The original prediction noise obtained in step S4.2 By applying constraints, the actual noise removed can be obtained. Represented as:

[0038]

[0039] Where S max In the early stage T E The maximum value of the structural similarity result obtained from the last calculation;

[0040] Step S4.4: According to step S4.3, utilize... For the noisy latent space image Z at time step t t Denoising yields a clean latent space image Z. 0|t , is represented as:

[0041]

[0042] Preferably, the implementation process of step S5 is as follows:

[0043] Step S5.1: Based on the low-resolution image I LR The degradation level is set by the image insertion ratio w t , is represented as:

[0044]

[0045] Step S5.2: Based on step S5.1, in the intervention phase T I At each time step t, the structure is embedded into Z. SE Inserted into a clean reconstructed image, it is represented as:

[0046] Z' 0|t =w t Z SE +(1-w t )Z 0|t

[0047] Where Z' 0|t Clean latent space image after intervention;

[0048] Preferably, in step S6, it is determined whether the entire inference process has been completed. If not, noise is added based on the existing image super-resolution diffusion model principle, as shown below:

[0049]

[0050] Preferably, in step S7, the process of outputting the high-resolution reconstructed image is as follows:

[0051]

[0052] Where IHR Z0 is the high-resolution reconstructed image output, and Z0 is the predicted clean latent space image obtained when the entire inference process is completed, i.e., when t=0.

[0053] Beneficial effects:

[0054] (1) Based on existing image super-resolution diffusion models, this invention proposes a structure-aware reconstructed image selection process in the early stages of inference. The selected reconstructed image, which is structurally most consistent with and clearer than the low-resolution image, is used for control condition embedding to guide subsequent inference. In the inference process of existing image super-resolution diffusion models, the guiding information comes from the structure, details, and semantic cues extracted from the low-resolution image. However, blurriness and noise in the low-resolution image may be over-interpreted by the model's generation prior, leading to false details in the final reconstructed image. In contrast, the reconstructed image selected in this invention has a high degree of structural consistency with the low-resolution image and contains fewer blurriness and noise issues, providing clearer guidance for the inference process and avoiding the generation of false details.

[0055] (2) The structure-aware image detail embedding proposed in this invention can be easily applied to existing image super-resolution diffusion models. Due to empirical priors and illusions, these models continuously generate false detail information during the inference process. However, the image detail embedding proposed in this invention can utilize the empirical priors of reconstructed image suppression models that contain fewer false details, thereby reducing the generation of false details in subsequent reconstructed images. Attached Figure Description

[0056] Figure 1 This is a flowchart of the structure-aware image super-resolution method provided by the present invention;

[0057] Figure 2 This is a simplified algorithm framework diagram of the structure-aware image super-resolution method provided by the present invention;

[0058] Figure 3 This is a simplified algorithm framework diagram of the structure-aware screening provided by the present invention;

[0059] Figure 4 This is a simplified algorithm framework diagram of the control condition embedding and image detail embedding provided by the present invention. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0061] This invention provides a high-resolution grayscale image layering and colorization method based on zero-range decomposition, the specific principle of which is as follows: Figure 1 As shown, it includes the following steps:

[0062] Step S1: Input low-resolution image I LR ;

[0063] Step S2: Divide the entire reasoning process into an early stage T. E and intervention phase T I Specifically, refer to Figure 2 .

[0064] Step S2.1: Based on the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage T. E and intervention phase T I , is represented as:

[0065] T = T E +T I

[0066] Where T is the number of inference steps predefined in the existing image super-resolution diffusion model;

[0067] Step S3, in the early stage T E Based on the low-resolution image I LR The structural similarity of the reconstructed images is used to select the clearest image with the most structural similarity as the structural embedding Z. SE Specifically, refer to Figure 3 .

[0068] Step S3.1: Define the noise prediction process for the existing image super-resolution diffusion model:

[0069] ∈ t =∈ θ (Z t ,ε(I LR ),t)

[0070] Where t is the current time step, ∈ t It is the prediction noise at time step t, ∈ θ It is a noise prediction network for existing image super-resolution diffusion models, Z t It is the noisy latent space image at time step t, ε is the encoder of the existing image super-resolution diffusion model, ε(I LR ) is the low-resolution image I LR The corresponding latent space image;

[0071] Step S3.2: According to step S3.1, based on the predicted noise at time step t, ∈ t For the noisy latent space image Z at time step tt Denoising:

[0072] Z 0|t =Sampler(Z) t ,∈ t ,t)

[0073] Z 0|t It is the predicted clean latent space image at time step t, and Sampler is the sampler of the existing image super-resolution diffusion model;

[0074] Step S3.3: According to step 3.2, the reconstructed image I at time step t. t Represented as:

[0075]

[0076] in It is a decoder for existing image super-resolution diffusion models;

[0077] Step S3.4: Based on step 3.3, the reconstructed image I at time step t. t With the low-resolution image I LR The calculation process for structural similarity is expressed as follows:

[0078] S t =SSIM(I t ,SR(I LR ))

[0079] Where S t This is the structural similarity result at time step t. SSIM is a publicly available structural similarity index algorithm. SR(I) LR ) is to adjust the low-resolution image I. LR To and reconstruct image I t The dimensions are consistent;

[0080] Step S3.5, for the early stage T E At each time step t, perform the structural similarity results S obtained in steps S3.1-3.4. t Store the data in a container Buffer, and then filter out the maximum value S. max Determine S t With S max If they are consistent, then the predicted clean latent space image Z at time step t is obtained. 0|t As a structural embedding Z SE .

[0081] Step S4, during the intervention phase T I By controlling the embedding conditions, Z is embedded using the aforementioned structure. SETogether with low-resolution images, they guide the reasoning process, promoting the generation of high-fidelity structural information. Specifically, refer to... Figure 4 .

[0082] Step S4.1, during the intervention phase T I Z is embedded using the aforementioned structure. SE The noisy latent space image Z at each time step t t Noise prediction is represented as:

[0083]

[0084] in It is based on the structure embedding Z at time step t. SE The obtained prediction noise.

[0085] Step S4.2: Obtain the original predicted noise based on the noise prediction process of the existing image super-resolution diffusion model.

[0086] Step S4.3: Using the information obtained in step S4.1 The original prediction noise obtained in step S4.2 By applying constraints, the actual noise removed can be obtained. Represented as:

[0087]

[0088] Where S max In the early stage T E The maximum value of the structural similarity result obtained from the last calculation;

[0089] Step S4.4: According to step S4.3, utilize... For the noisy latent space image Z at time step t t Denoising yields a clean latent space image Z. 0|t , is represented as:

[0090]

[0091] Step S5, during the intervention phase T I By embedding image details, based on low-resolution image I LR The degree of degradation, at each time step t, embeds the structure into Z. SE Inserted into a clean reconstructed image, suppressing any artificial details that may be introduced. Specifically, refer to... Figure 4 .

[0092] Step S5.1: Based on the low-resolution image I LR The degradation level is set by the image insertion ratio w t , is represented as:

[0093]

[0094] Step S5.2: Based on step S5.1, in the intervention phase T I At each time step t, the structure is embedded into Z. SE Inserted into a clean reconstructed image, it is represented as:

[0095] Z' 0|t =w t Z SE +(1-w t )Z 0|t

[0096] Where Z' 0|t Clean latent space image after intervention;

[0097] Step S6: Determine whether the entire inference process is complete. If not, add noise according to the existing image super-resolution diffusion model principle, as shown below:

[0098]

[0099] Step S7, the process of outputting the reconstructed high-resolution image, is as follows:

[0100]

[0101] Where I HR Z0 is the output high-resolution reconstructed image, and Z0 is the predicted clean latent space image obtained when the entire inference process is completed, i.e., when t=0.

[0102] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A structure-aware image super-resolution method, characterized in that, Includes the following steps: Step S1, inputting a low-resolution image ; Step S2, dividing the entire inference process into an early stage and an intervention stage ; Step S3, in the early stage Based on the low-resolution image mentioned above The structural similarity of the reconstructed images is used to select the clearest image with the most similar structure as the structural embedding. The specific implementation process is as follows: Step S3.1: Define the noise prediction process for the existing image super-resolution diffusion model: ； in It is the current time step. It is a time step Prediction noise at that time It is a noise prediction network for existing image super-resolution diffusion models. It is a time step Noisy latent space image at that time, It is the encoder of the existing image super-resolution diffusion model. The low-resolution image The corresponding latent space image; Step S3.2: Based on step S3.1, and using the aforementioned time step... Prediction noise at time time step Noisy latent space image Denoising: ； in It is a time step Predicted clean latent space image at time, It is a sampler for existing image super-resolution diffusion models; Step S3.3: According to step 3.2, the time step... Reconstructed images at time Represented as: ； in It is a decoder for existing image super-resolution diffusion models; Step S3.4: According to step 3.3, the time step... Reconstructed images at time With the aforementioned low-resolution image The calculation process for structural similarity is expressed as follows: ； in It is a time step Structural similarity results at time It is a publicly available structural similarity index algorithm. It is to adjust the low-resolution image mentioned above. To and reconstruct images The dimensions are consistent; Step S3.5, for the early stage At each time step t, perform the structural similarity results obtained in steps S3.1-3.

4. Store the data in a container Buffer and then filter out the maximum value. ,judge and If they are consistent, then the time steps will be... Predicted clean latent space image As structural embedding ; Step S4, during the intervention phase By controlling the embedding conditions, the structure is embedded. The inference process is guided by low-resolution images to generate clean reconstructed images; Step S5, during the intervention phase By embedding image details, based on low-resolution images The degree of degradation at each time step Embed the structure Inserted into the clean reconstructed image, suppressing any false details that may be introduced; Step S6: Determine whether the entire reasoning process has been completed; Step S7: Output the reconstructed high-resolution image.

2. The structure-aware image super-resolution method according to claim 1, characterized in that, The implementation process of step S2 is as follows: Step S2.1: Based on the changes in the structural information of the reconstructed image during the reasoning process, the entire reasoning process is divided into two stages, namely the early stage. and intervention phase , is represented as: ； in The number of inference steps is predefined for existing image super-resolution diffusion models.

3. The structure-aware image super-resolution method according to claim 1, characterized in that, The implementation process of step S4 is as follows: Step S4.1, during the intervention phase Embedded using the aforementioned structure For each time step Noisy latent space image Noise prediction is represented as: ； in It is a time step Time based on structure embedding The obtained prediction noise; Step S4.2: Obtain the original predicted noise based on the noise prediction process of the existing image super-resolution diffusion model. ; Step S4.3: Using the information obtained in step S4.1 The original prediction noise obtained in step S4.2 By applying constraints, the actual noise removed can be obtained. , is represented as: ； in In the early stages The maximum value of the structural similarity result obtained from the last calculation; Step S4.4: According to step S4.3, utilize... time step Noisy latent space image Denoising yields a clean latent space image for prediction. , is represented as: 。 4. The structure-aware image super-resolution method according to claim 3, characterized in that, The implementation process of step S5 is as follows: Step S5.1: Based on the low-resolution image Degradation level setting image insertion ratio , is represented as: ； Step S5.2: Based on step S5.1, in the intervention phase... Each time step Embed the structure Inserted into a clean reconstructed image, it is represented as: ； in Clean latent space image after intervention.

5. The structure-aware image super-resolution method according to claim 4, characterized in that, Step S6 determines whether the entire inference process is complete. If not, noise is added based on the existing image super-resolution diffusion model principle, as shown below: 。 6. The structure-aware image super-resolution method according to claim 5, characterized in that, The implementation process of step S7 is as follows: ； in It is the output high-resolution reconstructed image. It is to complete the entire reasoning process, that is When the time is right, the predicted clean latent space image is obtained.

Images