Image super-resolution generation model construction method and image super-resolution generation method and system

Abstract

The invention discloses a method for constructing an image super-resolution generation model. The method comprises the following steps: constructing a mixed data set containing various types of images; preprocessing the images in the mixed data set to obtain initial low-quality images and corresponding image text descriptions; determining a first initial learning framework, and generating a corresponding high-quality reference image according to the preprocessed initial low-quality image; training the first initial learning framework to obtain a first-stage repair model; determining a second initial learning framework, and generating a high-resolution image according to the random noise-added original image, a high-quality reference image generated by the first-stage repair model and image text description prediction; and training the second initial learning framework to obtain a target second-stage image super-resolution generation model. According to the method, a two-stage network framework model is designed and developed based on a potential diffusion model, rich prior of the diffusion model is utilized, and the generalization ability of the model is greatly improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a real image super-resolution method based on diffusion prior potential feature alignment. Background Art

[0002] Image restoration tasks (such as super-resolution, deblurring, etc.) are important tasks in the field of computer vision, which aim to restore the missing parts from damaged or missing image data through algorithmic means, so as to predict images with higher quality and resolution. Image restoration has a wide range of applications and research significance in many fields, such as image compression, video perception processing, medicine and remote sensing imaging, etc.

[0003] Traditional image restoration methods are based on linear interpolation, sparse coding, and texture synthesis. These algorithms require iteration, are slow, and are not particularly effective. They can produce noticeable blurring effects in larger areas. In recent years, deep learning-based image restoration methods have made significant progress. These methods are trained using large-scale image datasets to learn high-level features and contextual information of images, thereby achieving more accurate and natural image restoration.

[0004] Among them, for image super-resolution tasks, diffusion-based models have achieved impressive results in terms of perception. However, according to existing observations, diffusion models tend to reconstruct images with richer visual details, but with lower fidelity to the original images. The main reasons are as follows: First, since the diffusion model usually uses random Gaussian noise as a starting point, although the input image is used as the control signal of the ControlNet control network, the control signal features and the denoised output cannot be well aligned in the latent space condition, which introduces unnecessary randomness into the generated output image. In addition, during the denoising process, high-frequency information such as texture of the input image has less energy and is therefore more easily distorted than low-frequency information such as structure. Secondly, the typical image super-resolution problem is usually defined under a constrained setting where the degradation process is simple and known. This has successfully promoted a large number of excellent super-resolution reconstruction algorithms, but the generalization ability of the algorithm is limited.

[0005] Therefore, in the task of generative image super-resolution, enhancing the reconstruction of high-frequency details such as edges and textures is an important task. Summary of the invention

[0006] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0007] One object of the present invention is to provide a method for constructing a real image super-resolution generation model based on diffusion prior latent feature alignment, by constructing a two-stage network framework and training a model that reconstructs and generates high-quality images under the guidance of a given low-quality image.

[0008] Another object of the present invention is to provide an image super-resolution method, which utilizes the above-constructed model to reconstruct a high-quality image under the guidance of a given low-quality image.

[0009] Another object of the present invention is to provide an image super-resolution system.

[0010] In order to achieve the above-mentioned object, the present invention provides, on one hand, a method for constructing an image super-resolution generation model, comprising: Construct a mixed dataset containing multiple types of images; and preprocess the original images in the mixed dataset to obtain initial low-quality images and corresponding image text descriptions; Determine a first initial learning framework, generate a corresponding high-quality reference image according to the preprocessed initial low-quality image; determine a loss function according to the original image, the initial low-quality image and the generated high-quality reference image, train the first initial learning framework, and obtain a first-stage restoration model; A second initial learning framework is determined to generate a high-resolution image based on the randomly noisy original image, the high-quality reference image generated by the first-stage restoration model, and the image text description prediction; a second loss function is determined based on the randomly added noise and the predicted removed noise, the second initial learning framework is trained, and the target second-stage image super-resolution generation model is obtained.

[0011] A further preferred technical solution of the present invention is that the method of constructing a mixed data set containing multiple types of images; and preprocessing the images in the mixed data set to obtain an initial low-quality image and a corresponding image text description; comprising: Construct a mixed dataset containing at least synthetic images, natural scene images, wild images, face images, and real images; Perform cropping, normalization and data augmentation operations on the original image to generate the corresponding initial low-quality image; The BLIP-2 model is used to generate text descriptions for each image in the mixed dataset.

[0012] Preferably, the first initial learning framework adopts a SwinIR network model architecture, including a shallow feature extraction module, a deep feature extraction module and a high-quality image reconstruction module; The shallow feature extraction module and the high-quality image reconstruction module adopt an encoder-decoder architecture; The deep feature extraction module consists of multiple residual Transformer blocks, MLPs and convolutional layers.

[0013] Preferably, a first initial learning framework is determined, and a corresponding high-quality reference image is generated according to the preprocessed initial low-quality image; a loss function is determined according to the original image, the initial low-quality image and the generated high-quality reference image, and the first initial learning framework is trained to obtain a first-stage restoration model; including: The initial low-quality image is added to the first initial learning framework. The initial low-quality image passes through the shallow feature extraction module and the deep feature extraction module successively to extract the shallow features and deep features of the image. Then, the shallow features and the deep features are added as the input of the high-quality image reconstruction module to obtain a high-quality reference image. The loss value of the initial low-quality image and the corresponding high-quality reference image is calculated to determine the first loss function, which is expressed as:

[0014]

[0015] in and represent the original image and the initial low-quality image respectively, It is the high-quality reference image predicted by the restoration module, obtained through regression learning, and used to fine-tune the second-stage image super-resolution generation model; Perform iterative optimization of network hyperparameters to obtain a trained first-stage repair model.

[0016] Preferably, the second initial learning framework includes a VAE autoencoder, a ControlNet control network and a diffusion denoising U-Net network; Among them, the VAE autoencoder includes an encoder and a decoder; The ControlNet control network includes an encoder and an intermediate block; The diffusion denoising U-Net network consists of an encoder, an intermediate block and a decoder, where both the intermediate block and the decoder contain multiple feature alignment zero-attention modules.

[0017] Preferably, a second initial learning framework is determined, and a high-resolution image is generated according to the randomly noisy original image, the high-quality image generated by the first-stage restoration model, and the image text description prediction; a second loss function is determined according to the randomly added noise and the predicted removed noise, and the second initial learning framework is trained to obtain a target second-stage image super-resolution generation model; including: Freeze the weights of the first-stage restoration model, input the initial low-quality image into the first-stage restoration model, and generate a high-quality reference image corresponding to the initial low-quality image; The original image is encoded using the VAE autoencoder encoder and randomly noisy to convert it into a noisy latent code. ; Input the corresponding image text description into the pre-trained CLIP encoder to obtain text image features; The obtained corresponding high-quality reference image is converted into a tensor and then combined with the noise latent code Adding, taking text image features as additional semantic signals, inputting into the ControlNet control network to obtain conditional control signals of different scales; Noise Latent Code The text image features are input into a diffusion denoising U-Net network for denoising. The intermediate block and decoder of the diffusion denoising U-Net network obtain potential features of different scales. The potential features of different scales and the obtained conditional control signals of different scales are input into the feature alignment zero attention module. Each feature alignment zero attention module processes potential features and conditional control signals of the same scale. The potential features are passed through a multi-head self-attention layer and normalized by residual addition. The conditional control signal and the potential features are passed through a cross-attention network layer to obtain a fused feature. The fused feature and the potential feature are then added and passed through a feedforward network and normalized. The obtained features are passed through a zero linear layer to obtain the final feature map. Repeat the denoising operation, iterate T steps, and obtain the image latent code by the diffusion denoising U-Net network prediction. ; Then the image latent code Input into the decoder of the VAE autoencoder to generate a high-resolution image corresponding to the original image; The loss value is calculated by calculating the noise added each time in the random noise addition operation and the noise removed by each iterative prediction in the iterative denoising operation, and the second loss function is determined, which is expressed as:

[0018] in t is the diffusion step length, is the high-quality reference image predicted by the inpainting module, c is the text image feature, represents the diffusion denoising U-Net network; Iterative optimization of network hyperparameters is performed to obtain a fine-tuned and trained target second-stage image super-resolution generation model.

[0019] Another aspect of the present invention provides a method for generating an image super-resolution, comprising: Obtaining a low-resolution or blurred image to be generated; According to the above construction method, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; input the low-resolution or blurred image to be generated into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0020] Another aspect of the present invention provides an image super-resolution generation system, comprising: An image acquisition module, used to acquire a low-resolution or blurred image to be generated; A model building module, used to build a first-stage restoration model and a second-stage image super-resolution generation model according to the above-mentioned building method; The image generation module is used to freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; the low-resolution or blurred image to be generated is input into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0021] Another aspect of the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions enabling a computer to execute the above-mentioned image super-resolution generation method.

[0022] Yet another aspect of the present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls logic instructions in the memory to execute the above-mentioned image super-resolution generation method.

[0023] On the other hand, the present invention provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above-mentioned image super-resolution generation method.

[0024] Beneficial effects: Based on the potential diffusion model, the present invention designs and develops a two-stage network framework model for image super-resolution and other restoration tasks, which not only utilizes the rich priors of the diffusion model, but also greatly improves the generalization ability of the model. First, the random degradation and restoration module has a better and more accurate prediction effect on various complex degradations in the real world, improves the accuracy of image super-resolution, enables the model to better learn the information and features of the real image, and can also reduce the burden of the second-stage diffusion model to process real low-quality images; secondly, the diffusion model performs well in image generation tasks, and it has a strong ability in approximating diversity and complex distributions, and the pre-trained StableDiffusion has rich priors, and can perform various samplings in the feature space. The generated images are very realistic, and are more refined and clear in close objects and surface details. This two-stage network structure makes full use of the advantages of each network to generate higher quality images.

[0025] The present invention uses the ControlNet control network to fine-tune the pre-trained Stable Diffusion, controls the diffusion model to generate the desired image or output instead of random generation; and uses the feature alignment zero attention module to align the image pixel features and the noise input denoising output, which can well preserve the image high-frequency information, thereby generating an image with richer and clearer texture details, and the image can also achieve good structural alignment with the input low-quality image.

[0026] The present invention uses multiple data sets to mix and train real image super-resolution networks, making full use of information from different data sets to improve the performance of the model. This method helps the network to generalize better in various practical applications because it can adapt to diverse training data.

[0027] The real image super-resolution network proposed in the present invention not only achieves better results in image super-resolution tasks, but also shows good performance advantages in other image restoration tasks such as deblurring and denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for constructing an image super-resolution generation model according to the present invention; Figure 2 A schematic diagram of the framework of the image super-resolution generation model construction method of the present invention; Figure 3 This is the SwinIR network model architecture diagram for the repair module; Figure 4 A data processing diagram of the feature alignment zero-attention module of the present invention; Figure 5 Schematic diagram of the framework of the image super-resolution generation method of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0030] Combine the following Figure 1-Figure 5 The present invention describes an image super-resolution generation model construction method and an image super-resolution generation method and system.

[0031] Regarding the problems to be solved by the present invention, firstly, since the diffusion model usually takes random Gaussian noise as the starting point, although the input image is used as the control signal of the ControlNet control network, the conditional control signal features and the denoised output in the latent space cannot be well aligned, which leads to the introduction of unnecessary randomness in the generated output image, and in the denoising process, the high-frequency information such as the texture of the input image has less energy, so it is more easily distorted than the low-frequency information such as the structure. In order to solve these problems, the present invention focuses on proposing a feature alignment zero attention module (FAZA), which performs cross-attention operations on the conditional control signal and the denoised output output of the ControlNet control network to achieve feature alignment.

[0032] Secondly, the typical image super-resolution problem is usually defined under a constrained setting, where the degradation process is simple and known. This has successfully promoted a large number of excellent super-resolution reconstruction algorithms, but the generalization ability of the algorithms is limited. In order to deal with degraded images in the real world, the present invention pre-trains an image restoration module in multiple degradations to improve the generalization ability in real scenes. In addition, the present invention uses a mixed data set to train the network, which further improves the generalization ability of the model.

[0033] Therefore, the present invention designs and develops an image super-resolution deep learning algorithm based on deep learning algorithms and image processing technology. The algorithm can compress and encode the images uploaded by users, and then denoise them using the powerful prior of the diffusion model. Finally, it can reconstruct high-quality images under the guidance of given low-quality images.

[0034] In view of the technical problems to be solved and the concept of the present invention, the following embodiments are proposed. First, an image super-resolution generation model and a corresponding model construction method and training method need to be designed. Then, the constructed model is used to generate image super-resolution, and then a corresponding image super-resolution generation system, a non-transitory computer-readable storage medium, an electronic device, and a computer program product are provided.

[0035] Embodiment 1: This embodiment provides a method for constructing an image super-resolution generation model. Generally speaking, it includes: Construct a mixed dataset containing multiple types of images; and preprocess the original images in the mixed dataset to obtain initial low-quality images and corresponding image text descriptions; Determine a first initial learning framework, generate a corresponding high-quality reference image according to the preprocessed initial low-quality image; determine a loss function according to the original image, the initial low-quality image and the generated high-quality reference image, train the first initial learning framework, and obtain a first-stage restoration model; A second initial learning framework is determined to generate a high-resolution image based on the randomly noisy original image, the high-quality reference image generated by the first-stage restoration model, and the image text description prediction; a second loss function is determined based on the randomly added noise and the predicted removed noise, the second initial learning framework is trained, and the target second-stage image super-resolution generation model is obtained.

[0036] The focus of this embodiment is the construction and training of the two-stage model. The following is a detailed description of each step. Figure 1 , Figure 2 As shown, including: 1. Construct a mixed dataset containing multiple types of images; and preprocess the original images in the mixed dataset to obtain initial low-quality images and corresponding image text descriptions; including: S1. Construct a mixed dataset for training, and perform cropping, normalization, and data augmentation operations on the original images to generate corresponding initial low-quality images; at the same time, use the BLIP-2 model to generate text descriptions for each image in the mixed dataset.

[0037] Furthermore, the mixed dataset consists of DIV2K (synthetic images), Flickr2K (natural scenes), OST (wild images), the first 6000 images of FFHQ (face images), and the RealSR (real image) dataset. Each dataset consists of RGB images of different resolutions as original images, and only the highest resolution images are used during training.

[0038] Furthermore, in order to achieve mixed training of multiple datasets, the size of the RGB images is cropped and standardized to the same resolution size of 512×512×3, and linear normalization is used to make the pixel value range between [0,1]. The original image and the initial low-quality image obtained in step S1 should be preprocessed as above.

[0039] 2. Determine the first initial learning framework, generate the corresponding high-quality reference image according to the preprocessed initial low-quality image; determine the loss function according to the original image, the initial low-quality image and the corresponding high-quality reference image, train the first initial learning framework, and obtain the first-stage restoration model; Figure 3 As shown, including: S2, inputting the initial low-quality image obtained in step S1 into the first initial learning framework, predicting and generating a high-quality reference image of the same size, rich details and high accuracy as the initial low-quality image; The first initial learning framework adopts the SwinIR network model architecture, including a shallow feature extraction module, a deep feature extraction module and a high-quality image reconstruction module; the shallow feature extraction module and the high-quality image reconstruction module adopt an encoder-decoder architecture; the deep feature extraction module consists of multiple residual Transformer blocks, MLPs and convolutional layers.

[0040] The initial low-quality image passes through the shallow feature extraction module and the deep feature extraction module successively to extract the shallow features and deep features of the image, and then the shallow features and the deep features are added as the input of the high-quality image reconstruction module to obtain a high-quality reference image.

[0041] S3, calculate the loss value of the original image, the initial low-quality image and the corresponding high-quality reference image, determine the first loss function, perform iterative optimization of the network hyperparameters, and obtain the trained first-stage restoration model. The first loss function is expressed as:

[0042]

[0043] in and represent the original image and the initial low-quality image respectively, It is a high-quality reference image predicted by the restoration module, obtained through regression learning, and used to fine-tune the second-stage image super-resolution generation model.

[0044] 3. Determine the second initial learning framework, generate a high-resolution image based on the randomly noisy original image, the high-quality reference image generated by the first-stage restoration model, and the image text description prediction; determine the second loss function based on the randomly added noise and the predicted removed noise, train the second initial learning framework, and obtain the target second-stage image super-resolution generation model.

[0045] The second initial learning framework includes VAE autoencoder, ControlNet control network and diffusion denoising U-Net network; Among them, the VAE autoencoder includes an encoder and a decoder; the ControlNet control network includes an encoder and an intermediate block; the diffusion denoising U-Net network includes an encoder, an intermediate block and a decoder, and both the intermediate block and the decoder contain multiple feature alignment zero-attention modules.

[0046] S4, freezing the weights of the first-stage restoration model, inputting the initial low-quality image into the first-stage restoration model, and generating a high-quality reference image with the same size, rich details, and high accuracy as the initial low-quality image.

[0047] S5. Add the high-quality reference image obtained in step S4 to the second initial learning framework.

[0048] In step S5, the second initial learning framework uses the pre-trained text-to-image latent diffusion model StableDiffusion and fine-tunes it. During the fine-tuning process, the VAE autoencoder has frozen parameters and is not trained.

[0049] S6. Use the VAE autoencoder to encode the original image and perform random noise operation to convert it into a noise latent code. ; Input the image text description obtained in step S1 into the pre-trained CLIP encoder to obtain text image features; In step S6, the parameters of the CLIP encoder are also frozen when fine-tuning Stable Diffusion, and no training is performed, while the noise latent code It follows a Gaussian distribution.

[0050] S7, the noise latent code obtained in step S6 Input the ControlNet control network with text image features; convert the high-quality reference image obtained in step S4 into a tensor and add it to the noise latent code The text image features are added as additional semantic signals and input into the ControlNet control network to obtain conditional control signals of different scales.

[0051] When fine-tuning the diffusion model, the ControlNet control network needs to be trained, and its initial weight parameters are initialized from the pre-trained diffusion denoising U-Net network checkpoint.

[0052] S8, the noise latent code obtained in step S6 The text image features are input into a diffusion denoising U-Net network for denoising. The intermediate block and decoder of the diffusion denoising U-Net network obtain potential features of different scales. The potential features of different scales and the obtained conditional control signals of different scales are input into the feature alignment zero attention module. Each feature alignment zero attention module processes potential features and conditional control signals of the same scale; the potential features pass through a multi-head self-attention layer and are normalized by residual addition; the conditional control signal and the potential features pass through a cross-attention network layer to obtain a fused feature, and then the fused feature and the potential feature are added and passed through a feedforward network and normalized, and the obtained features pass through a zero linear layer to obtain the final feature map.

[0053] In step 8, when fine-tuning the diffusion denoising U-Net network, its encoder, intermediate block, and decoder are all imported into the U-Net model weights of Stable Diffusion, and are frozen and not involved in training. Only the feature alignment zero attention module is trained, and its initialization weight is zero. Due to the misalignment of the conditional control signal with the noise input, it is not enough to retain the original image input details. Therefore, the feature alignment zero attention module is a flexible operation that is adjusted by applying the attention mechanism. Specifically, Figure 4 As shown in the figure, the denoised output of the U-Net decoder is input into the self-attention layer, followed by the cross-attention layer, where the query (Q) comes from the previous self-attention layer and the conditional control signal output by the ControlNet control network is used as the key (K) and value (V). In order to eliminate harmful noise, a linear layer with zero initial weight is introduced after the feedforward network.

[0054] Because all network layers in the feature alignment zero-attention module do not change the size of the feature map dimension, the final output potential feature has the same size as the input potential feature dimension. Due to the global information processing advantage of Transformer, it achieves good global retention of features, and the conditional control signal aligns and controls the denoising output well through the cross-attention network layer.

[0055] S9, repeat the denoising operation in step S8, iterate T steps, and obtain a clean, noise-free image latent code predicted by the diffusion denoising U-Net network ; S10, the clean, noise-free image latent code obtained in step S9 Input into the decoder of the VAE autoencoder to generate a high-resolution image that is the same as the original image but with richer details and higher quality; S11, calculate the loss value of the noise added each time in the random noise addition operation of step S6 and the noise removed by each iterative prediction in the iterative denoising operation of step S9, determine the second loss function, perform iterative optimization of the network hyperparameters, and obtain the fine-tuned and trained target second-stage image super-resolution generation model. The second loss function is expressed as:

[0056] in t is the diffusion step length, is the high-quality reference image predicted by the inpainting module, c is the text image feature, Represents the diffusion denoising U-Net network.

[0057] The constructed model is then used to generate image super-resolution, and a corresponding image super-resolution generation system, a non-transitory computer-readable storage medium, an electronic device and a computer program product are provided.

[0058] Based on the latent diffusion model, this embodiment designs an image super-resolution generation model for restoration tasks such as image super-resolution, as well as corresponding model construction methods and training methods. In the first stage, this embodiment pre-trains an image restoration module in multiple degradations, extracts degradation-insensitive features, and guides the diffusion process in combination with high-level image information to improve the generalization ability in real scenes. In the second stage, the generation ability of the latent diffusion model is used to achieve the restoration and high-quality reconstruction of real images. In this embodiment, a conditional control network ControlNet is introduced to control the embedding from the original image to achieve fine-tuning of the pre-trained diffusion model, and a feature alignment zero attention module (FAZA) is introduced in the denoising U-Net network of the diffusion model of this embodiment, so that the diffusion model can perceive the conditional control signal in the latent space and achieve better alignment with the denoising output. It not only better retains the low-frequency structural information of the image, but also can achieve better enhanced reconstruction of the high-frequency texture features of the image.

[0059] Finally, since the pre-trained latent diffusion model is a text-based diffusion model in which text is used as input, while image restoration usually uses LQ images as input. Although some other diffusion model-based methods adopt empty text prompts, it has been shown that content-related titles can improve the generation results. In this embodiment, a pre-trained BLIP-2 network is used to extract image caption information from the input image, and a CLIP encoder is used to convert the text information into image-level features, providing additional semantic signals to control the diffusion process.

[0060] Using a single dataset to train an image super-resolution network relies too much on synthetic data, resulting in limited generalization of the model. A network trained with a mixed dataset uses datasets from multiple different source scenes for training. This training method aims to increase the model's ability to understand different scenes and features, and improve the generalization and robustness of the model. In this embodiment, a mixed dataset of five datasets, including synthetic images, real images, field images, and face images, is collected to train the proposed image super-resolution network, which improves the robustness and generalization of the network.

[0061] Embodiment 2: This embodiment provides a method for generating image super-resolution, such as Figure 5 As shown, including: Obtaining a low-resolution or blurred image to be generated; According to the above construction method, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; input the low-resolution or blurred image to be generated into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0062] Embodiment 3: This embodiment provides an image super-resolution generation system, comprising: An image acquisition module, used to acquire a low-resolution or blurred image to be generated; A model building module, used to build a first-stage restoration model and a second-stage image super-resolution generation model according to the above-mentioned building method; The image generation module is used to freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; the low-resolution or blurred image to be generated is input into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0063] Embodiment 4: This embodiment provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to execute an image super-resolution generation method, such as Figure 5 As shown, including: Obtaining a low-resolution or blurred image to be generated; According to the above construction method, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; input the low-resolution or blurred image to be generated into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0064] Embodiment 5: This embodiment provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call logic instructions in the memory to execute an image super-resolution generation method, including: The method comprises the following steps: Take the low-resolution or blurred image to be generated; According to the above construction method, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; input the low-resolution or blurred image to be generated into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0065] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0066] Embodiment 6: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute an image super-resolution generation method, including: The method comprises the following steps: Take the low-resolution or blurred image to be generated; According to the above construction method, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; input the low-resolution or blurred image to be generated into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

[0067] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an image super-resolution generation model, characterized in that: include: Construct a mixed dataset containing multiple types of images; And preprocess the original images in the mixed data set to obtain initial low-quality images and corresponding image text descriptions; Determine a first initial learning framework, generate a corresponding high-quality reference image according to the preprocessed initial low-quality image; determine a loss function according to the original image, the initial low-quality image and the generated high-quality reference image, train the first initial learning framework, and obtain a first-stage restoration model; A second initial learning framework is determined to generate a high-resolution image based on the randomly noisy original image, the high-quality reference image generated by the first-stage restoration model, and the image text description prediction; a second loss function is determined based on the randomly added noise and the predicted removed noise, the second initial learning framework is trained, and the target second-stage image super-resolution generation model is obtained.

2. The method for constructing an image super-resolution generation model according to claim 1, characterized in that: The method comprises: constructing a mixed data set including multiple types of images; and preprocessing the images in the mixed data set to obtain initial low-quality images and corresponding image text descriptions; including: Construct a mixed dataset containing at least synthetic images, natural scene images, wild images, face images, and real images; Perform cropping, normalization and data augmentation operations on the original image to generate the corresponding initial low-quality image; The BLIP-2 model is used to generate text descriptions for each image in the mixed dataset.

3. The method for constructing an image super-resolution generation model according to claim 1, characterized in that: The first initial learning framework adopts the SwinIR network model architecture, including a shallow feature extraction module, a deep feature extraction module and a high-quality image reconstruction module; The shallow feature extraction module and the high-quality image reconstruction module adopt an encoder-decoder architecture; The deep feature extraction module consists of multiple residual Transformer blocks, MLPs and convolutional layers.

4. The method for constructing an image super-resolution generation model according to claim 3, characterized in that: Determine a first initial learning framework, and generate a corresponding high-quality reference image according to the preprocessed initial low-quality image; Determine the loss function based on the original image, the initial low-quality image and the generated high-quality reference image, train the first initial learning framework, and obtain the first-stage restoration model; including: The initial low-quality image is input into the first initial learning framework. The initial low-quality image passes through the shallow feature extraction module and the deep feature extraction module successively to extract the shallow features and deep features of the image. Then, the shallow features and the deep features are added as the input of the high-quality image reconstruction module to obtain a high-quality reference image. The loss value of the initial low-quality image and the corresponding high-quality reference image is calculated to determine the first loss function, which is expressed as: ； ； in and represent the original image and the initial low-quality image respectively, It is the high-quality reference image predicted by the restoration module, obtained through regression learning, and used to fine-tune the second-stage image super-resolution generation model; Perform iterative optimization of network hyperparameters to obtain a trained first-stage repair model.

5. The method for constructing an image super-resolution generation model according to claim 1, characterized in that: The second initial learning framework includes a VAE autoencoder, a ControlNet control network, and a diffusion denoising U-Net network; Among them, the VAE autoencoder includes an encoder and a decoder; The ControlNet control network includes an encoder and an intermediate block; The diffusion denoising U-Net network consists of an encoder, an intermediate block and a decoder, where both the intermediate block and the decoder contain multiple feature alignment zero-attention modules.

6. The method for constructing an image super-resolution generation model according to claim 5, characterized in that: Determine a second initial learning framework to generate a high-resolution image based on the randomly noisy original image, the high-quality reference image generated by the first-stage restoration model, and the image text description prediction; Determine the second loss function according to the randomly added noise and the predicted removed noise, train the second initial learning framework, and obtain the target second stage image super-resolution generation model; including: Freeze the weights of the first-stage restoration model, input the initial low-quality image into the first-stage restoration model, and generate a high-quality reference image corresponding to the initial low-quality image; The original image is encoded using the VAE autoencoder encoder and randomly noisy to convert it into a noisy latent code. ; Input the corresponding image text description into the pre-trained CLIP encoder to obtain text image features; The obtained corresponding high-quality reference image is converted into a tensor and then combined with the noise latent code Adding, taking text image features as additional semantic signals, inputting into the ControlNet control network to obtain conditional control signals of different scales; Noise Latent Code The text image features are input into a diffusion denoising U-Net network for denoising. The intermediate block and decoder of the diffusion denoising U-Net network obtain potential features of different scales. The potential features of different scales and the obtained conditional control signals of different scales are input into the feature alignment zero attention module. Each feature alignment zero attention module processes potential features and conditional control signals of the same scale. The potential features are passed through a multi-head self-attention layer and normalized by residual addition. The conditional control signal and the potential features are passed through a cross-attention network layer to obtain a fused feature. The fused feature and the potential feature are then added and passed through a feedforward network and normalized. The obtained features are passed through a zero linear layer to obtain the final feature map. Repeat the denoising operation, iterate T steps, and obtain the image latent code by the diffusion denoising U-Net network prediction. ; Then the image latent code Input into the decoder of the VAE autoencoder to generate a high-resolution image corresponding to the original image; The loss value is calculated by calculating the noise added each time in the random noise addition operation and the noise removed by each iterative prediction in the iterative denoising operation, and the second loss function is determined, which is expressed as: ； in t is the diffusion step length, is the high-quality reference image predicted by the inpainting module, c is the text image feature, represents the diffusion denoising U-Net network; Iterative optimization of network hyperparameters is performed to obtain a fine-tuned and trained target second-stage image super-resolution generation model.

7. A method for generating image super-resolution, characterized in that: include: Obtaining a low-resolution or blurred image to be generated; According to the construction method according to any one of claims 1 to 6, a first-stage restoration model and a second-stage image super-resolution generation model are constructed; Freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; The low-resolution or blurred image to be generated is input into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

8. An image super-resolution generation system, characterized in that: include: An image acquisition module, used to acquire a low-resolution or blurred image to be generated; A model construction module, used for constructing a first-stage restoration model and a second-stage image super-resolution generation model according to the construction method according to any one of claims 1 to 6; An image generation module, used to freeze the model weights of the first-stage restoration model and the second-stage image super-resolution generation model; The low-resolution or blurred image to be generated is input into the first-stage restoration model and the second-stage image super-resolution generation model to generate a super-resolution image.

9. A non-transitory computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and the computer instructions enable the computer to execute the image super-resolution generation method described in claim 7.

10. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls the logic instructions in the memory to execute the image super-resolution generation method described in claim 7.

Images