Real scene image super-resolution method and system based on mean transfer diffusion

Abstract

The invention belongs to the field of image super-resolution, and provides a real scene image super-resolution method and system based on mean transfer diffusion, and the method comprises the steps: obtaining a low-resolution image in a real scene, and enabling the low-resolution image to be projected in a potential space through a pre-trained variational auto-encoder, and obtaining the potential features of the low-resolution image; adding noise into the potential features of the low-resolution image to obtain an initial diffusion state; and on the basis of the initial diffusion state, de-noising is carried out by iteratively utilizing a pre-trained stable diffusion model, and each round of iteration is guided by taking the potential features of the low-resolution image as control signals, so that a super-resolution reconstructed image is obtained. The invention provides a mean migration diffusion process for realizing the consistency of training and testing. Based on the proposed mean migration diffusion process, a corresponding training strategy and a reasoning method are designed, and an existing stable diffusion model is finely adjusted by using the training strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution, and in particular relates to a real scene image super-resolution method and system based on mean shift diffusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Image super-resolution (SR) aims to recover high-resolution (HR) images from degraded low-resolution (LR) image observations. Starting with SRCNN, deep learning-based ISR has become popular. Various methods focusing on model design have been proposed to improve the accuracy of ISR reconstruction and achieve satisfactory performance on synthetic data with the same degradation. However, these SR methods have strict assumptions on the degradation process, such as bicubic downsampling and blurring with known parameters, which limits their effectiveness in dealing with complex and unknown degradations in the real world. Therefore, real-world super-resolution has gradually become a popular research direction in recent years.

[0004] Real-scene image super-resolution (Real-ISR) aims to restore low-resolution images with unknown and complex degradation in the real world to high-resolution versions with high quality. Some researchers have accomplished this task by synthesizing LR-HR image pairs similar to real-world data. Specifically, BSRGAN uses a random shuffled degradation modeling strategy, while Real-ESRGAN adopts a high-order degradation modeling process. Using training samples with more realistic degradation, both BSRGAN and Real-ESRGAN use GAN to reconstruct the desired HR images and achieve better visual perception results, but these GAN-based methods usually produce unnatural artifacts. Recent research based on diffusion models has further demonstrated superior performance on real-world image SR. Benefiting from the powerful generative priors of pre-trained text-to-image (T2I) diffusion models, such as stable diffusion (SD), Real-ISR methods based on pre-trained stable diffusion models often produce many details with excellent visual effects. StableSR uses SD for the first time to complete the Real-ISR task and demonstrates impressive detail generation capabilities. DiffBIR uses ControlNet to provide appropriate control signals for SD to improve the generation effect of the model. However, StableSR and DiffBIR only use images as control conditions, and neither considers the role of semantic information in the pre-trained T2I model. Therefore, PASD extracts low-level and high-level features of LR images and inputs them into the pre-trained SD model with a pixel-aware cross-attention module. SeeSR further explores the way to introduce semantic information, using anti-degradation label-style text hints and utilizing soft hints to guide the diffusion process, so that the model can produce more realistic texture details.

[0005] In other words, the existing super-resolution methods based on pre-trained SD models have the problem of inconsistency between training and testing. The inconsistency between training and testing of existing methods comes from the fact that the pre-trained stable diffusion model will encode high-resolution and low-resolution image pairs into the latent space, resulting in a huge mean difference, making it difficult for the model trained on noisy high-resolution images to adapt to the test with noisy low-resolution images as the diffusion starting point. These methods use the HR image latent features with noise as input during the training phase, and use the LR image latent features as input during the test phase. However, after the pre-trained SD model encodes the paired HR images and LR images into the latent space, there will be a significant difference in the feature mean between the two, which leads to obvious inconsistency between training and testing of existing methods. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a novel mean-shift diffusion process, which realizes the transfer of the distribution mean from the LR image to the HR image by replacing the mean of the Gaussian noise in the traditional diffusion process from zero to the weight coefficient multiplied by the residual of the LR image and the HR image, so that the inverse diffusion process during inference can naturally start from the noisy LR image and recover the corresponding high-quality HR image. This mean-shift diffusion supports the use of noisy LR images as the diffusion starting point, so during testing, only the potential features of the LR image with noise added need to be used as the input of the model, and a high-quality high-resolution image can be recovered through an iterative denoising process.

[0007] According to some embodiments, a first solution of the present invention provides a real scene image super-resolution method based on mean shift diffusion, which adopts the following technical solution:

[0008] A real scene image super-resolution method based on mean-shift diffusion, comprising:

[0009] Obtain low-resolution images in real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images;

[0010] Add noise to the latent features of the low-resolution image to obtain the initial diffusion state;

[0011] Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image.

[0012] Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

[0013] Furthermore, the training strategy of the stable diffusion model is specifically as follows:

[0014] Acquire a high-resolution image, and randomly degrade the high-resolution image to obtain a low-resolution image;

[0015] Use pre-trained variational autoencoders to obtain high-resolution image latent features and low-resolution image latent features;

[0016] Calculate the difference between the low-resolution latent features and the high-resolution image latent features as the residual;

[0017] Multiply the residual by the weight factor and add standard Gaussian noise to get the target noise;

[0018] The product of the target noise and the scale parameter is added to the potential features of the high-resolution image under a certain scale parameter to obtain the current state distribution;

[0019] The current state distribution is input into the noise prediction network model to predict the target noise, and the difference between the predicted target noise and the target noise is used as the loss to obtain the final trained noise prediction network model;

[0020] The final trained noise prediction network model is used as the pre-trained stable diffusion model.

[0021] Furthermore, the noise prediction network model is used to obtain the estimation of the potential features of the high-resolution image, specifically:

[0022]

[0023] Among them, the noise prediction network model ∈ θ (z t ,z lr ,t),z θ (z t ,z lr ,t) is the estimate of the potential features of the high-resolution image, and the distribution z at the tth time step t , t=1,2,3,……T, T is an integer, z lr is the potential feature of the low-resolution image, α t is the scale parameter in the SD diffusion process and θ is the model parameter.

[0024] Furthermore, the mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein:

[0025] Mean-shift diffusion forward process: replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the original high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the high-resolution image latent features to the low-resolution image latent features;

[0026] The reverse process of mean-shift diffusion is to replace the mean of Gaussian noise with the residual between the low-resolution image potential features and the high-resolution image potential features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image potential features to the high-resolution potential features.

[0027] Furthermore, in the mean transfer diffusion forward process, the distribution z of each time step t is t satisfy:

[0028]

[0029] Among them, z hr is the high-resolution image potential feature, z lr is the potential feature of the low-resolution image, α t and βt is the scale parameter in the SD diffusion process, satisfying α t ,β t >0 and w t Represents a time-dependent weight curve.

[0030] Furthermore, the reverse process of mean shift diffusion is as follows:

[0031] p θ (z t-1 |z t ,z lr )=α t-1 z θ +(w t-1 β t-1 -w t η t )(z lr -z θ )+η t ∈′ θ +σ t ∈ 1 ;

[0032] in, z θ =z θ (z t ,z lr ,t),∈′ θ =∈′ θ (z t ,z lr ,t), And t is a variable parameter, z lr is the potential feature of the low-resolution image, and the distribution z at the t-th time step t , the distribution z at the t-1th time step t-1 , α t-1 and β t-1 is the scale parameter in the SD diffusion process, satisfying α t-1 ,β t-1 >0 and w t represents a time-dependent weight curve; noise prediction network model ∈′ θ (z t ,z lr ,t),z′ θ (z t ,z lr ,t) as the high-resolution image potential feature z hr Estimate of ∈′ θ is the target noise, ∈ 1 is a randomly sampled standard Gaussian noise, and θ is a model parameter.

[0033] Furthermore, a time-dependent weight curve w t , specifically:

[0034]

[0035] Among them, γ is a hyperparameter that controls the speed of the distribution mean shift, and T is the time step.

[0036] According to some embodiments, a second solution of the present invention provides a real scene image super-resolution system based on mean shift diffusion, which adopts the following technical solution:

[0037] Real scene image super-resolution system based on mean-shift diffusion, including:

[0038] A latent feature acquisition module is configured to acquire a low-resolution image in a real scene, and use a pre-trained variational autoencoder to project the low-resolution image into a latent space to obtain a latent feature of the low-resolution image;

[0039] An initial diffusion state determination module is configured to add noise to the potential features of the low-resolution image to obtain an initial diffusion state;

[0040] The image super-resolution module is configured to iteratively use the pre-trained stable diffusion model for denoising based on the starting diffusion state, and use the potential features of the low-resolution image as a control signal to guide each round of iteration to obtain a super-resolution reconstructed image;

[0041] Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

[0042] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the real scene image super-resolution method based on mean-shift diffusion as described in the first aspect above.

[0044] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0045] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the real scene image super-resolution method based on mean shift diffusion as described in the first aspect above are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention proposes a mean-shift diffusion process, in which the distribution mean is transferred from the HR image to the LR image in the forward process, so that the diffusion reverse process during inference can naturally start from the noisy LR image and recover the corresponding high-quality HR image, solving the inconsistency between training and testing of the existing methods, so that the model can more effectively utilize the trained knowledge during testing and obtain better performance.

[0048] Based on the mean shift diffusion process, the present invention proposes a set of training strategies and a mean shift sampler in the inference process, which can be applied to all Real-ISR methods based on pre-trained SD and has strong versatility. After fine-tuning using the diffusion process, the model can achieve better performance with one-fifth of the previous sampling steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 This is a flow chart of a real scene image super-resolution method based on mean shift diffusion in an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of changes in the mean values ​​of the HR and LR image pairs in the pixel space during the forward diffusion process described in an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of changes in the mean of the HR and LR image pairs in the latent space during the forward diffusion process described in an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the distribution change of the mean shift diffusion process in the potential space in an embodiment of the present invention;

[0054] Figure 5 is the weight curve w in the embodiment of the present invention t Schematic diagram;

[0055] Figure 6 is a schematic diagram of the mean shift diffusion training process in an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of the mean shift diffusion inference stage in an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram comparing the results of the method according to the embodiment of the present invention and other existing methods on a Real data set. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0061] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0062] Embodiment 1

[0063] like Figure 1 As shown, this embodiment provides a real scene image super-resolution method based on mean shift diffusion. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0064] Obtain low-resolution images in real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images;

[0065] Add noise to the latent features of the low-resolution image to obtain the initial diffusion state;

[0066] Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image.

[0067] Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

[0068] The mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein:

[0069] Mean-shift diffusion forward process: replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the original high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the high-resolution image latent features to the low-resolution image latent features;

[0070] First, the diffusion process needs to start from the noisy LR latent features, which requires that the mean of the distribution is converted from the original HR latent features to the LR latent features during the diffusion process. According to the original diffusion process in stable diffusion (SD), at step t, the current state z t0 satisfy:

[0071] z t0 =α t z hr +β t ∈,t=1,2,…,T(1);

[0072] Among them, α t and β t is the scale parameter in the SD diffusion process, satisfying α t ,β t >0 and And z hr In the Real-ISR task, it is the HR latent feature. In the traditional diffusion process, the noise ∈ follows a standard Gaussian distribution, and its mean is completely determined by α t z hr Since the noise in the original diffusion process does not change the mean, the mean value is introduced as w t (z lr -z hr ) instead of the standard Gaussian noise ∈ to achieve the shift of the distribution mean. lr represents the LR potential feature, w t Represents a time-dependent weight curve.

[0073] The transformed distribution z t It can be expressed as follows:

[0074] z t =α t z hr +β t ∈′=α tz hr +w t β t (z lr -z hr )+β t ∈(2);

[0075] in, Obviously, when When t can be regarded as LR latent features plus standard Gaussian noise, which is easy to obtain during inference. t = 0, the mean shift diffusion process is converted to the traditional diffusion process, and the mean of the distribution is the HR potential feature. The weight curve w designed by the present invention t The form is as follows:

[0076]

[0077] Among them, γ is a hyperparameter that controls the speed of the mean shift. Finally, in the forward process of mean shift diffusion, the distribution z at each time step t is t satisfy:

[0078]

[0079] Based on this forward process, a new stable diffusion SD training strategy is designed for the Real-ISR task. First, the HR image latent features and LR image latent features are obtained by pre-training the VAE encoder (Variational Autoencoder, VAE), which are denoted as z and hr and z lr Then, calculate z hr and z lr The residual is denoted as z res =z lr -z hr . Then, z res Multiply by the weight factor w t , and add the randomly sampled standard Gaussian noise ∈ to get the target noise ∈′. Finally, β t ∈′ added to α t z hr In the current distribution z t , and input it into the noise prediction network model to predict the target noise. It can be understood that the noise prediction network model includes but is not limited to using the Unet network model.

[0080] The optimization goal of the training phase is:

[0081]

[0082] Based on the loss in the above formula (5), the general stable diffusion model is fine-tuned to obtain the final trained noise prediction network model, and the final trained noise prediction network model is used as the pre-trained stable diffusion model in the method described in this example.

[0083] Mean shift diffusion inverse process: replace the mean of Gaussian noise with the residual between the low-resolution image potential features and the high-resolution image potential features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image potential features to the high-resolution potential features;

[0084] After obtaining the mean shift forward diffusion process and its corresponding training strategy, the corresponding mean shift sampler is still needed in the inference stage. The forward diffusion process is regarded as a non-Markov chain similar to DDIM, so that the distribution z t-1 can be considered as the variance and the mean is z t ,z hr With z lr The normal distribution of the linear combination of . The formula is as follows:

[0085]

[0086] Among them, k t ,λ t and p t is the linear weighting coefficient, is the distribution z at each time step t-1 t-1 The estimated value of, the distribution z at the tth time step t , the distribution z at the t-1th time step t-1 By using the method of undetermined coefficients, we can solve for k t ,λ t and p t , its solution is expressed as follows:

[0087]

[0088] Considering that z cannot be obtained in practical applications hr , so a noise prediction network model ∈′ can be trained θ (z t ,z lr ,t) to obtain z′ θ (z t ,z lr ,t) as z hr The estimate is as follows:

[0089]

[0090] By combining the above formulas, the mean shift diffusion inverse process can be expressed as:

[0091] p θ (z t-1 |z t ,z lr )=α t-1 z θ +(w t-1 β t-1 -w t η t )(z lr -z θ )+η t ∈′ θ +σ t ∈ 1 (9);

[0092] in, z θ =z θ (z t ,z lr ,t),∈′ θ =∈′ θ (z t ,z lr ,t), And t is a variable parameter; the current state z at step t-1 during the diffusion process t-1 , α t-1 and β t-1 is the scale parameter in the SD diffusion process, satisfying α t-1 ,β t-1 >0 and w t represents a time-dependent weight curve; noise prediction network model ∈′ θ (z t ,z lr ,y),z′ θ (z t ,z lr ,t) as the high-resolution image potential feature z ht Estimate of ∈′ θ is the target noise, ∈ 1 is a randomly sampled standard Gaussian noise, and θ is a model parameter.

[0094] Finally, this embodiment customizes the mean shift sampler for inference based on the reverse process expression. t = 0 as the mean shift sampler in the inference process, that is, the DDIM version of the sampler.

[0095] In the inference stage, the present invention uses the potential features of the LR image to add noise as the diffusion starting point, and iteratively uses a sampler for denoising, and finally obtains a high-quality super-resolution result.

[0096] In the experiment, the present invention uses the most advanced real-scene super-resolution model SeeSR as the basic model architecture, and uses only 10 sampling steps to surpass the effect of the original SeeSR model with 50 sampling steps, which has significantly improved the inference speed and result quality.

[0097] The mean-shift diffusion process transfers the mean of the distribution from high-resolution image features to low-resolution image features in the forward diffusion process, and from low-resolution image features to high-resolution image features in the reverse diffusion process, while the mean of the distribution in the traditional diffusion process is always the high-resolution image. In the forward diffusion process, the mean of the distribution gradually transfers from high-resolution image features to low-resolution image features to solve the inconsistency of training and testing of existing methods; the generation capability of the pre-trained stable diffusion model is used to achieve high-quality image super-resolution effects, and the potential features of low-resolution images with noise are used as the diffusion starting point to restore the details of high-resolution images in the diffusion process; the mean-shift diffusion process is used to fine-tune the existing most advanced real-scene image super-resolution model, achieving a significant improvement in inference speed and super-resolution quality.

[0098] Example 1: Mean-Shift Diffusion

[0099] like Figure 2 and Figure 3 As shown in , the HR and LR image pairs have similar means in the pixel space, but when they are encoded into the latent space, a significant mean difference appears. As noise is continuously added during the forward diffusion process, this mean difference gradually decreases but remains significant. Therefore, the present invention proposes a mean transfer diffusion method such as Figure 4As shown in the figure, after being projected into the latent space, the high-resolution (HR) distribution (blue area) and the low-resolution (LR) distribution (orange area) show significant differences in distribution. In the traditional forward diffusion process, although the HR and LR distributions gradually approach each other, they still maintain a significant distance due to the difference in means, and finally fall into the green and red areas, respectively. Since the current Real-ISR models based on diffusion models follow the traditional diffusion process and predict zero-mean standard Gaussian noise, they have difficulty in recovering the HR distribution from the noisy LR distribution during the backward diffusion process, and finally move away from the purple area of ​​the HR distribution. In this study, we replace the mean of the Gaussian noise with the residual between the LR latent features and the HR latent features, and scale them by a weight factor. This strategy enables the distribution mean to naturally transition from the LR latent features to the HR latent features during the backward diffusion process. It is called the mean-shifted diffusion process. In this setting, the backward diffusion process restores the noisy LR distribution to a position closer to the HR distribution (such as Figure 4 In practice, the Figure 5 The weight curve w shown t , where the hyperparameter γ = 50.

[0100] Example 2: Experimental Setup

[0101] The present invention is trained on the top 10K face images of DIV2K, DIV8K, Flickr2K, OST, and FFHQ. In order to synthesize low-quality images paired with high-quality images, the images in the dataset are degraded using the degradation process of RealESRGAN and randomly cropped into 512×512 image blocks. For the network architecture, this embodiment uses the same network as the SeeSR model and fine-tunes the model weights based on the publicly released SeeSR model. The model is fine-tuned for 50k iterations using the Adam optimizer, with a batch size of 24 and a learning rate of 5×10 -6 , for training.

[0102] Example 3: Diffusion model training

[0103] 1. Model architecture:

[0104] -Use a diffusion model-based image restoration network, which is pre-trained on a large-scale dataset and accumulates rich prior knowledge.

[0105] 2. Training process:

[0106] -Use the images in the dataset as HR images and use the two-stage degradation process proposed by RealESRGAN to degrade the images to obtain LR versions.

[0107] -Training process such as Figure 6As shown in the figure, the potential representations of HR images and LR images are obtained by pre-training the VAE encoder, which are denoted as z hr and z lr . Use z lr Input as control signal to ControlNet. Calculate z hr and z lr The residual is denoted as z res =z lr -z hr . Then, z res Multiply by the weight factor w t , and add the randomly sampled standard Gaussian noise ∈ to get the target noise ∈′. Finally, β t ∈′ added to α t z hr In the current distribution z t , and input it into the noise prediction network model to predict the target noise. The loss of formula (5) is used between the output of the noise prediction network model and the target noise to obtain the final noise prediction network model, which is used as the pre-trained stable diffusion model.

[0108] The pre-trained stable diffusion model accumulates rich prior knowledge by pre-training on large-scale datasets to enhance the model's ability to generate and restore image details. The mean shift diffusion model training strategy is used to fine-tune the existing state-of-the-art real scene image super-resolution model based on the stable diffusion model, and the noisy low-resolution image features are used as the diffusion starting point. The denoising process is implemented according to the proposed inference sampler to gradually restore high-resolution image details.

[0109] Example 4: Image Restoration

[0110] 1. Input processing:

[0111] -Take the LR image as the overall input and project it into the latent space through the VAE encoder to obtain z lr , add T time step noise to it, and get the initial diffusion state z T , t=1,2,3,……T, T is an integer. Use z lr As a control signal input into ControlNet to guide the denoising process, such as Figure 7 shown.

[0112] 2. Recovery process:

[0113] -The diffusion model gradually restores the high-frequency details in the image. In each iteration, the LR image and the extracted semantic information are used as control signals to guide the restoration process.

[0114] 3. Output generation:

[0115] -Finally, a high-definition HR image is generated. The content in the restored image is consistent with the LR image, and the degradation in the LR image is removed, with rich details and high quality.

[0116] Example 5: Experimental verification

[0117] 1. Experimental setup:

[0118] -The DIV2K validation set and the RealSR, DRealSR, and LR200 datasets are selected as test datasets to compare the restoration effects of different methods.

[0119] - Use PSNR, SSIM, LPIPS and FID to measure image fidelity. For non-reference metrics, use CLIPIQ, MUSIQ, MANIQA, TOPIQ and QALIGN to measure image quality.

[0120] 2. Experimental results:

[0121] -Compared with the traditional SD-based Real-ISR method, the method of the present invention has significant improvements in reference-free indicators such as CLIPIQ, MUSIQ, MANIQA, TOPIQ and QALIGN.

[0122] -like Figure 8 As shown in the figure, the restored image shows excellent performance in terms of high-frequency detail generation ability and image content fidelity, especially on severely degraded images.

[0123] This embodiment aims to restore perceptually convincing high-resolution images based on low-resolution images with complex and diverse degradation in the real world. Many methods use the generation ability of pre-trained stable diffusion models to achieve super-resolution to obtain better super-resolution effects. However, due to the gap between training and testing, these methods are prone to produce blurred super-resolution effects. To address this problem, this embodiment proposes a diffusion process of mean shift to achieve consistency between model training and testing. Based on the diffusion process, a real scene image super-resolution method is proposed, which can achieve high-quality image super-resolution effects with a small number of diffusion steps. First, the limitations of existing methods based on pre-trained stable diffusion models are analyzed, the inconsistency problem of training and testing of existing methods is revealed, and a diffusion process of mean shift is proposed to achieve consistency between training and testing. Based on the proposed mean shift diffusion process, corresponding training strategies and inference methods are designed, and the existing model is fine-tuned using the training strategy. Experiments have shown that the proposed method can achieve higher quality image super-resolution effects in less inference time. This invention provides new ideas and methods for the field of real scene image super-resolution.

[0124] Embodiment 2

[0125] This embodiment provides a real scene image super-resolution system based on mean shift diffusion, including:

[0126] A latent feature acquisition module is configured to acquire a low-resolution image in a real scene, and use a pre-trained variational autoencoder to project the low-resolution image into a latent space to obtain a latent feature of the low-resolution image;

[0127] An initial diffusion state determination module is configured to add noise to the potential features of the low-resolution image to obtain an initial diffusion state;

[0128] The image super-resolution module is configured to iteratively use the pre-trained stable diffusion model for denoising based on the starting diffusion state, and use the potential features of the low-resolution image as a control signal to guide each round of iteration to obtain a super-resolution reconstructed image;

[0129] Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

[0130] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0131] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0133] Embodiment 3

[0134] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the real scene image super-resolution method based on mean shift diffusion as described in the first embodiment above are implemented.

[0135] Embodiment 4

[0136] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the real scene image super-resolution method based on mean transfer diffusion as described in the first embodiment are implemented.

[0137] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0142] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A real scene image super-resolution method based on mean-shift diffusion, characterized in that: include: Obtain low-resolution images in real scenes, and use pre-trained variational autoencoders to project the low-resolution images into the latent space to obtain the latent features of the low-resolution images; Add noise to the latent features of the low-resolution image to obtain the initial diffusion state; Based on the initial diffusion state, the pre-trained stable diffusion model is used iteratively for denoising, and the potential features of the low-resolution image are used as control signals to guide each round of iteration to obtain a super-resolution reconstructed image. Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

2. The real scene image super-resolution method based on mean shift diffusion according to claim 1, characterized in that: The training strategy of the stable diffusion model is specifically as follows: Acquire a high-resolution image, and randomly degrade the high-resolution image to obtain a low-resolution image; Use pre-trained variational autoencoders to obtain high-resolution image latent features and low-resolution image latent features; Calculate the difference between the low-resolution latent features and the high-resolution image latent features as the residual; Multiply the residual by the weight factor and add standard Gaussian noise to get the target noise; The product of the target noise and the scale parameter is added to the potential features of the high-resolution image under a certain scale parameter to obtain the current state distribution; The current state distribution is input into the noise prediction network model to predict the target noise, and the difference between the predicted target noise and the target noise is used as the loss to obtain the final trained noise prediction network model; The final trained noise prediction network model is used as the pre-trained stable diffusion model.

3. The real scene image super-resolution method based on mean shift diffusion according to claim 2, characterized in that: The noise prediction network model is used to obtain an estimate of the potential features of the high-resolution image, specifically: Among them, the noise prediction network model ∈′ θ (z t ,z lr ,t),z′ θ (z t ,z lr ,t) is the estimate of the potential features of the high-resolution image, and the distribution z at the tth time step t , t=1,2,3,……T, T is an integer, z lr is the potential feature of the low-resolution image, α t is the scale parameter in the SD diffusion process and θ is the model parameter.

4. The real scene image super-resolution method based on mean shift diffusion according to claim 1, characterized in that: The mean shift diffusion process includes a mean shift diffusion forward process and a mean shift diffusion reverse process, wherein: Mean-shift diffusion forward process: replace the mean of Gaussian noise with the residual between the low-resolution image latent features and the original high-resolution image latent features, and scale them according to the weight factor so that the distribution mean is transformed from the high-resolution image latent features to the low-resolution image latent features; The reverse process of mean-shift diffusion is to replace the mean of Gaussian noise with the residual between the low-resolution image potential features and the high-resolution image potential features, and scale them according to the weight factor so that the distribution mean is transformed from the low-resolution image potential features to the high-resolution potential features.

5. The real scene image super-resolution method based on mean shift diffusion according to claim 4, characterized in that: In the mean transfer diffusion forward process, the distribution z at each time step t t satisfy: Among them, z hr is the high-resolution image potential feature, z lr is the potential feature of the low-resolution image, α t and β t is the scale parameter in the SD diffusion process, satisfying α t ,β t >0 and w t Represents a time-dependent weight curve.

6. The real scene image super-resolution method based on mean shift diffusion according to claim 4, characterized in that: The reverse process of mean shift diffusion is as follows: p θ (With t-1 |from t ,With lr )=α t-1 With θ +(in t-1 β t-1 -In t η t )(With lr -With θ )+η t ∈′ θ +σ t ∈1; in, And t is a variable parameter, z lr is the distribution z of the latent features of the low-resolution image at the tth time step t , the distribution z at the t-1th time step t-1 , α t-1 and β t-1 is the scale parameter in the SD diffusion process, satisfying α t-1 ,β t-1 >0 and w t represents a time-dependent weight curve; noise prediction network model ∈′ θ (z t ,z lr ,t),z′ θ (z t ,z lr ,t) as the high-resolution image potential feature z hr Estimate of ∈′ θ is the target noise, ∈1 is the randomly sampled standard Gaussian noise, and θ is the model parameter.

7. The real scene image super-resolution method based on mean shift diffusion according to claim 6, characterized in that: A time-dependent weight curve w t , specifically: Among them, γ is a hyperparameter that controls the speed of the distribution mean shift, and T is the time step.

8. Real scene image super-resolution system based on mean-shift diffusion, characterized in that: include: A latent feature acquisition module is configured to acquire a low-resolution image in a real scene, and use a pre-trained variational autoencoder to project the low-resolution image into a latent space to obtain a latent feature of the low-resolution image; An initial diffusion state determination module is configured to add noise to the potential features of the low-resolution image to obtain an initial diffusion state; The image super-resolution module is configured to iteratively use the pre-trained stable diffusion model for denoising based on the starting diffusion state, and use the potential features of the low-resolution image as a control signal to guide each round of iteration to obtain a super-resolution reconstructed image; Among them, the mean shift diffusion process is used to determine the training strategy and the stable diffusion model of the stable diffusion model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the real scene image super-resolution method based on mean-shift diffusion as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the real scene image super-resolution method based on mean shift diffusion as described in any one of claims 1 to 7 are implemented.

Images