Real image arbitrary scale super-resolution system and method based on unpaired data

Abstract

The application provides a real image arbitrary scale super-resolution system and method based on unpaired data, which comprises the following steps: firstly, providing an unpaired high-resolution image as a reference image; performing bicubic down-sampling on the reference image according to a target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics of the target scale; performing adaptive fusion of content and degradation characteristics by using a real scene low-resolution image to be processed and the low-resolution reference image to obtain a target low-resolution image; and performing reconstruction on the target low-resolution image according to the target super-resolution scale to obtain a super-resolution image. The application is based on a phased training process, learns content representation first, and then decomposes scale-related degradation representation into complementary components to solve the challenge of mutual entanglement of degradation modes and scale factors. Therefore, only synthetic data training is required, without real paired data, which significantly improves the practicability and reconstruction quality of the scheme.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to a system and method for arbitrary-scale super-resolution of real images based on unpaired data. Background Technology

[0002] Single-image super-resolution refers to reconstructing a high-resolution image from a low-resolution observation. Early research was typically based on simplified models, employing synthetic bicubic downsampling to simulate image degradation and relying on hand-designed image priors to reconstruct high-frequency details. Thanks to the hierarchical feature representation capabilities of deep neural networks, subsequent research shifted to end-to-end learning to directly establish a mapping relationship from low-resolution to high-resolution images, achieving significant results. However, such supervised learning methods based on synthetic paired data typically rely on simple pixel-level loss functions (such as L1 / L2 loss) for optimization, easily converging to the average of pixel statistics, resulting in output images lacking texture details and exhibiting blurry visual effects. More importantly, these methods face two major drawbacks in real-world scenarios: first, synthetic bicubic degradation differs significantly from the complex degradation (such as blur and noise) of real-world scenes, leading to a significant performance degradation in practical applications; second, most models are trained only at a fixed scale, requiring retraining or maintaining multiple independent models for different target scales, resulting in poor flexibility and low efficiency. These shortcomings make it difficult for existing methods to meet the dual requirements of complex degradation and flexible scale in real-world scenarios. Therefore, exploring learning paradigms that do not rely on pixel-level paired data to address the challenges of both real-world complex degradation and arbitrary scale has become a clear direction and core difficulty in this field.

[0003] To address the challenges of complex degradation in real-world images, unpaired image super-resolution methods have emerged. These methods aim to learn a mapping from low-quality observations with unknown and complex degradation to high-quality images using only unpaired real-world images, while strictly maintaining content consistency between the output and input. The core challenge lies in establishing an effective mapping between the real low-resolution and high-resolution domains in the absence of pixel-level paired supervision. Generative Adversarial Networks (GANs) and their variants are key generative frameworks in this field, driving the generator to output results that conform to the target domain distribution through adversarial training, effectively improving the model's efficiency and generalization ability. In contrast, methods such as VQ-VAE decouple the content and degradation representation of unpaired images in a discrete latent space, establishing discrete encodings through a learnable codebook, thereby achieving cross-domain transformation. In recent years, diffusion models, with their superior training stability and generative diversity, have provided a new and powerful paradigm for capturing and reconstructing richer, finer-grained image details.

[0004] On the other hand, arbitrary-scale super-resolution aims to achieve continuous-scale upsampling. Since training is typically based on synthetic degradation assumptions (such as bicubic downsampling), the performance of such mainstream models in real-world scenarios is limited. For example, methods based on meta-learning strategies aim to learn shared representations across scales, thereby generalizing to unseen scales during the inference phase. Inspired by Neural Radiation Fields (NeRF), implicit neural representation methods, such as LIIF and HIIF, treat images as continuous functions defined in a latent space, allowing pixel values ​​to be queried at arbitrary continuous coordinates. To overcome the limitations of synthetic data, the COZ scheme based on continuous optical zoom attempts to directly acquire real-world continuous-scale paired data by constructing a precise imaging system, but this introduces new problems such as high cost and susceptibility to hardware bias.

[0005] Analysis of the two aforementioned technical approaches reveals a complementary yet mutually exclusive approach to real-world scenarios: super-resolution methods based on unpaired data aim to handle real-world degradation but are typically limited to fixed upsampling scales; while arbitrary-scale super-resolution methods, though capable of flexible upsampling, suffer from performance limitations in handling complex real-world degradation due to reliance on synthetic degradation assumptions or constraints imposed by costly and difficult-to-scale real-world data acquisition schemes. Therefore, constructing a unified model capable of simultaneously handling real-world degradation and arbitrary scales is urgently needed. However, the core challenge in representing degradation in real-world scenarios lies in the entanglement between degradation patterns and scale factors, requiring model training to cover a high-dimensional joint space comprised of complex real-world degradation types and infinitely continuous scales. Systematically constructing such high-quality paired datasets is extremely costly; while real-time generation of training data pairs incurs an unbearable computational burden due to the need to model this complex space. Therefore, exploring effective decoupling and representation of this entanglement without relying on expensive real-world paired data to establish efficient cross-domain, cross-scale mappings has become a key driver for the practical application of this technology. Summary of the Invention

[0006] In view of the above, the main objective of this invention is to propose a real-scale super-resolution system and method for unpaired data to solve the aforementioned technical problems.

[0007] This invention proposes an arbitrary-scale super-resolution system for real images based on unpaired data, the system comprising:

[0008] The reference image preprocessing module is used for:

[0009] An unpaired high-resolution image is provided as a reference image. The reference image is then bicubic downsampled according to the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale.

[0010] The degradation domain adaptive module is used for:

[0011] By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image.

[0012] The arbitrary-scale super-resolution module, connected to the degradation domain adaptive module, is used for:

[0013] Based on the target's super-resolution scale, perform image super-resolution reconstruction on the target's low-resolution image to obtain a super-resolution image.

[0014] This invention also proposes a method for arbitrary-scale super-resolution of real images based on unpaired data, wherein the method applies the aforementioned arbitrary-scale super-resolution system for real images based on unpaired data, and includes the following steps:

[0015] Provide an unpaired high-resolution image as a reference image;

[0016] The reference image is downsampled bicubic based on the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale.

[0017] By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image.

[0018] Based on the target's super-resolution scale, perform image super-resolution reconstruction on the target's low-resolution image to obtain a super-resolution image.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] 1. By transforming the real-world super-resolution problem at any scale into a feature adaptation task from real-world degradation to bicubic degradation at the target scale, training only needs to rely on controllable synthetic data. This effectively avoids the computational bottleneck of generating massive amounts of paired data to cover continuous scales and real-world complex degradation, greatly reducing data preparation costs and computational burden.

[0021] 2. During inference, by adapting low-resolution images of real-world scenes to low-resolution versions with bicubic degradation characteristics at the target scale, direct compatibility with arbitrary-scale super-resolution models pre-trained on synthetic bicubic degradation data is achieved, significantly improving the practicality and ease of deployment of the solution in real-world scenarios.

[0022] 3. Through phased training, based on the decoupling of content and scale-related degradation representations, the scale-related degradation representations are further decomposed into two complementary components: scale-independent degradation and scale modulation representations. This effectively solves the challenge of the entanglement between degradation patterns and scale factors in real-world super-resolution at any scale, and achieves the separation of scale influences and precise control over degradation characteristics.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of the real image arbitrary-scale super-resolution system based on unpaired data proposed in this invention;

[0025] Figure 2 This is a schematic diagram of the kernel modulation convolution of the condition characterization generation unit of the present invention;

[0026] Figure 3 This is a schematic diagram of the phased training process used in the training module of this invention;

[0027] Figure 4 This is a flowchart illustrating the learning phase of the content representation of this invention.

[0028] Figure 5 This is a schematic diagram of the scale-independent degradation representation learning stage of the present invention;

[0029] Figure 6 This is a schematic diagram of the scale modulation characterization learning stage of the present invention;

[0030] Figure 7 This is a flowchart illustrating the training phase of the conditional diffusion model of the present invention.

[0031] Figure 8 This is a flowchart of the arbitrary-scale super-resolution method for real images based on unpaired data proposed in this invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0033] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0034] Please see Figure 1 This embodiment provides a real-image super-resolution system at arbitrary scales based on unpaired data, the system comprising:

[0035] The reference image preprocessing module is used for:

[0036] An unpaired high-resolution image is provided as a reference image. The reference image is then bicubic downsampled according to the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale.

[0037] The degradation domain adaptive module is used for:

[0038] By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image.

[0039] In a preferred embodiment of the present invention, the degradation domain adaptive module includes:

[0040] The content encoder is used to encode low-resolution images of the real scene to be processed, thereby obtaining a content representation.

[0041] A degradation encoder is used to encode a low-resolution reference image with bicubic degradation characteristics at the target scale to obtain a scale-independent degradation representation.

[0042] A scale encoder is used to encode a low-resolution reference image with target-scale bicubic degradation characteristics to obtain a scale modulation characterization.

[0043] The conditional representation generation unit, connected to the content encoder, degradation encoder, and scale encoder, is used to dynamically generate a degradation kernel adapted to the input degradation state based on scale-independent degradation representation and scale modulation representation, and to convolve the content representation using the degradation kernel to generate the conditional representation.

[0044] The conditional diffusion model, connected to the conditional representation generation unit, is used to concatenate the conditional representation with the noisy image in the channel dimension. Through an iterative denoising process controlled by time steps, it generates a target low-resolution image whose content is consistent with the real scene low-resolution image and has the target scale bicubic degradation characteristics.

[0045] In a preferred embodiment of the present invention, a content encoder is used to encode a low-resolution image of a real scene to be processed to obtain a content representation. The content encoder includes a convolutional module, a residual convolutional network, a linear layer, and a vector quantization layer connected in sequence; wherein, the convolutional module consists of three... Convolutional layers are connected in series, and residual convolutional networks are composed of multiple residual blocks connected in sequence.

[0046] In a preferred embodiment of the present invention, encoding a low-resolution reference image with target-scale bicubic degradation characteristics to obtain a scale-independent degradation characterization specifically includes the following steps:

[0047] A low-resolution reference image with bicubic degradation characteristics at the target scale is input into a convolutional layer for processing to extract the first shallow layer features;

[0048] The first shallow features are input into a residual convolutional network, and deep features are extracted through multiple sequentially connected residual blocks to obtain the extracted deep features.

[0049] The extracted deep features are sequentially processed through two subsequent activation functions. A convolutional layer, a pooling layer, and two subsequent activation functions. The convolutional layer process outputs a scale-independent degradation representation.

[0050] In a preferred embodiment of the present invention, encoding a low-resolution reference image with target-scale bicubic degradation characteristics to obtain a scale modulation characterization specifically includes the following steps:

[0051] A low-resolution reference image with bicubic degradation characteristics at the target scale is input into a convolutional layer for processing to extract the second shallow layer features;

[0052] The second shallow features are input into a network consisting of three sequentially connected scale feature processing units and a convolutional layer to obtain scale-related deep features. Each scale feature processing unit includes two sequentially connected Transformer blocks that employ a sliding window attention mechanism. Scale-aware location encoding modulated by the target super-resolution scale is introduced into the sliding window attention mechanism of the Transformer blocks.

[0053] The scale-correlated deep features are residually concatenated with the second shallow features, and then subjected to average pooling and linear transformation in sequence to output the scale-modulated representation.

[0054] In a preferred embodiment of the present invention, in the Transformer block, a scale-aware position encoding modulated by the target super-resolution scale is introduced to encode the target super-resolution scale information into the feature space. The scale-aware position encoding is generated and embedded into the self-attention computation through the following process. Specifically, introducing the scale-aware position encoding modulated by the target super-resolution scale into the sliding window attention mechanism of the Transformer block includes the following steps:

[0055] ① Divide the input feature map into multiple parts of size . The non-overlapping windows are used, and the features within each window are linearly projected to obtain the query vectors. Key vector Sum value vector ;

[0056] ② Obtain the relative coordinates between each token within the window, concatenate the relative coordinates with the preset target super-resolution scale information in the height and width directions, input the data into a lightweight network consisting of two linear layers, and generate scale-aware position codes.

[0057] ③ The scale-aware position code is embedded as a bias term in the window self-attention calculation. The corresponding process has the following relationship:

[0058] ;

[0059] in, Calculate window self-attention; , , These are the query vector, key vector, and value vector, respectively. For query vector and key vector Dimensions Static position encoding for embedding scale-free information. Dynamic position encoding for embedding scale information, For normalized exponential functions, This is a transpose operation.

[0060] Please see Figure 2 In a preferred embodiment of the present invention, a degradation kernel adapted to the input degradation state is dynamically generated based on scale-independent degradation representation and scale-modulated representation, and the content representation is convolved using the degradation kernel to generate a conditional representation. Specifically, the following steps are included:

[0061] Scale-independent degeneracy representation and scale modulation characterization The representations are stitched together along the channel dimension. The stitched representations are then first fed into a fusion network for fusion, and then into a kernel generation network to generate the basic degenerate kernels. ;

[0062] Characterizing scale modulation Input the affine transform coefficient generation network to generate scaling coefficients. and offset coefficient ;

[0063] Using scaling factor and offset coefficient For basic degenerate nuclei Perform channel-level affine modulation to obtain the modulation degenerate kernel. The affine modulation process has the following relationship:

[0064] ;

[0065] in, To multiply each channel, To add channel by channel, Based on the degenerate nucleus, This is the scaling factor. This is the offset coefficient. To modulate the degenerate nucleus;

[0066] Modulation degenerate nucleus As a convolution kernel, it represents the content. Perform depthwise convolution operations to generate conditional representations. The depthwise convolution operation has the following relationship:

[0067] ;

[0068] in, For depthwise convolution operations, For content representation, This is a conditional characterization.

[0069] In a preferred embodiment of the present invention, the conditional representation and the noisy image are concatenated along the channel dimension, and an iterative denoising process based on time step control is used to generate a target low-resolution image whose content is consistent with the real scene low-resolution image and has target-scale bicubic degradation characteristics. Specifically, the following steps are included:

[0070] A partial noise initialization strategy based on the input image is adopted, and the low-resolution image of the real scene is denoted as... Set the start time step And sample random Gaussian noise The initial noise image is calculated according to the noise addition formula, and the corresponding process has the following relationship:

[0071] ;

[0072] in, For the initial noisy image, For the noise scheduler at time step Defined cumulative noise figure, Low-resolution images of real-world scenes. The starting time step, It is positive random Gaussian noise. , This represents a standard normal distribution, where 0 indicates a vector with zero mean. The unit covariance matrix; the total time steps of the conditional diffusion model in this embodiment. The set inference start time step The dimension of the time step embedding vector is 256, and the channel dimension of the conditional representation is 64.

[0073] Noise image As the initial state for the reverse denoising process, it is obtained from time step 1 through the conditional diffusion model. Start gradually reducing noise to For each time step The operation to be performed is as follows:

[0074] Add noise to the image With conditional representation Channels are concatenated along the channel dimension and input together with the current time step t into the conditional diffusion model to obtain the predicted noise. and variance prediction parameters ;

[0075] Based on the predicted noise, the predicted target image is calculated, and the corresponding process has the following relationship:

[0076] ;

[0077] in, For the predicted target image, For the noise scheduler at time step The defined cumulative noise figure, For a noisy image;

[0078] Based on the predicted target image, the mean of the predicted posterior distribution is calculated, and the corresponding process has the following relationship:

[0079] ;

[0080] in, To predict the mean of the posterior distribution, The coefficients defined for the noise dispatcher, For the noise scheduler at time step The cumulative noise figure as defined;

[0081] Based on variance prediction parameters Coefficients defined by the noise scheduler The theoretical baseline variance of the coefficients defined by the noise dispatcher. The calculation of the predicted posterior distribution variance follows the following relationship:

[0082] ;

[0083] in, To predict the variance of the posterior distribution, For variance prediction parameters, The theoretical baseline variance calculated for the coefficients defined for the noise dispatcher. ;

[0084] Using the predicted posterior distribution mean and predicted posterior distribution variance The image at the next time step is obtained by sampling the square root of the image, and the corresponding process has the following relationship:

[0085] ;

[0086] in, Inverse random Gaussian noise is used to introduce randomness during the backsampling process. ;

[0087] When noise reduction reaches the time step At that time, a target low-resolution image is obtained. The target low-resolution image is consistent with the input real scene low-resolution image in terms of content, and at the same time has the target scale bicubic degradation characteristics.

[0088] The arbitrary-scale super-resolution module, connected to the degradation domain adaptive module, is used for:

[0089] Based on the target's super-resolution scale, perform image super-resolution reconstruction on the target's low-resolution image to obtain a super-resolution image.

[0090] Please see Figure 3As a preferred embodiment of the present invention, it further includes a training module. The training module trains the arbitrary-scale super-resolution module and the degradation domain adaptive module in a phased training manner. The core design of this process is as follows: firstly, the content representation and scale-related degradation representation of the image are decoupled, and then the scale-related degradation representation is further decomposed into scale-independent degradation representation and scale-related modulation representation. The specific training strategy is as follows: content representation learning trains the content encoder and its corresponding content decoder through reconstruction loss; scale-independent degradation representation and scale modulation representation learning train the degradation encoder and its degradation decoder, and the scale encoder and its scale decoder, respectively, by combining reconstruction constraints and contrastive learning strategies; the conditional diffusion model training freezes the aforementioned encoders, enabling the model to learn to fuse the representations obtained by the three encoders into conditions, and thereby learn to approximate the distribution of the low-resolution image after bicubic degradation at the target scale.

[0091] In this process, the parameter freezing strategy ensures the stability of the representations during the learning process, while the contrastive learning strategy enhances the ability to distinguish between scale-independent degradation and scale-modulated representations. The content decoder, degradation decoder, and scale decoder have identical structures and are all auxiliary training components, consisting of a convolutional layer, a residual convolutional network containing five cascaded residual blocks, and two convolutional layers with ReLU activation functions. These three decoders are only used during their respective training phases and are removed during inference. Randomized synthetic degradation refers to the degradation process implemented using the Real-ESRGAN method. The specific training process is as follows:

[0092] Phase 1: Pre-training of arbitrary scale super-resolution module: Obtain training samples of low-resolution-high-resolution image pairs with bicubic degradation characteristics, and pre-train the arbitrary scale super-resolution module using a large number of training samples to obtain the pre-trained arbitrary scale super-resolution module.

[0093] Phase 2, Content Representation Learning: High-resolution images from the training samples are acquired, and low-resolution images obtained by randomly synthesizing and degrading the high-resolution images and randomly varying the downsampling scale are used as content representation learning data; low-resolution images obtained by bicubic downsampling the same high-resolution image at the same random scale are used as supervised target images.

[0094] The content representation learning data is processed sequentially through the content encoder and content decoder, and the output of the content decoder is constrained by the supervised target image. In this way, the content encoder is trained to extract content representation by minimizing the reconstruction error and the representation discretization constraint.

[0095] Please see Figure 4 This phase of training is implemented within the VQ-VAE self-supervised framework, aiming to learn discretized content representations. The content encoder processes the content representation learning data. The vector is mapped to a feature vector, and then quantized using a vector quantization process within a predefined codebook. The nearest neighbor embedding vector is found to form a discretized content representation, where, The embedding vector is from a pre-defined codebook. For codebook size, For each embedding vector The training design employs a content encoder to process the learning data into content representations, which are then reconstructed into images by the content decoder. This design uses inputs with randomly varying scales and degradation levels to encourage the model to focus on learning the image content itself, avoiding overfitting to specific degradation patterns. Furthermore, vector quantization discretizes continuous features, effectively isolating the content from complex real-world degradation patterns to achieve content representation learning.

[0096] The total loss function in the content representation learning phase has the following relationship:

[0097] ;

[0098] in, For content representation learning data, Let the total loss function be the content representation learning phase. Supervised target images are bicubic downsampled at the same random scale. It consists of a convolutional module, a residual convolutional network, and linear layers in the content encoder; For content decoder functions, The distance is the squared Euclidean distance. To stop the gradient operator, Hyperparameters for controlling the weights of commitment loss;

[0099] Phase 3, Scale-Independent Degradation Representation Learning: Freeze the trained content encoder; use a low-resolution image, which is randomly degraded from a high-resolution image at a predefined baseline scale, as the first anchor image;

[0100] The first anchor image is simultaneously input into the content encoder and the degradation encoder to be trained to obtain content representation and scale-independent degradation representation, respectively.

[0101] A degradation kernel is generated from the scale-independent degradation representation and used as a convolution kernel. After performing a convolution operation on the content representation, it is input into the degradation decoder to obtain the reconstructed image. A first contrastive learning sample set is constructed based on degradation consistency. The image is reconstructed with the constraints of the first anchor image itself, and the degradation encoder is jointly trained by the first positive and negative sample pairs in the first contrastive learning sample set through contrastive learning loss to learn the essential differences between different degradation types and form a scale-independent degradation representation.

[0102] Please see Figure 5In the scale-independent degradation representation learning stage, joint training is performed through contrastive learning and anchor image reconstruction; a first contrastive learning sample set based on degradation consistency is constructed, in which the first anchor image... Let be a low-resolution image that has been randomly synthesized and degraded at a predefined baseline scale; assuming there are a total The first positive and negative sample pair, for the first positive and negative sample pair The first sample pair The first positive sample For the first anchor point image The first negative sample is a low-resolution image subjected to the same randomized degradation but with different content. For the first anchor point image Low-resolution images with identical content but subjected to different random synthetic degradation;

[0103] The total loss function for the scale-independent degenerate representation learning stage has the following relationship:

[0104] ;

[0105] in, Let the weighting coefficient corresponding to the first reconstruction loss be denoted as . , Let be the total loss function for the scale-independent degenerate representation learning phase. The first reconstruction loss, The first contrastive learning loss;

[0106] The first contrastive learning loss has the following relationship:

[0107] ;

[0108] in, For a degraded encoder, In this embodiment, the preset threshold is used. ; As the first positive sample, For the first anchor point image, This is the first negative sample;

[0109] The first reconstruction loss has the following relationship:

[0110] ;

[0111] in, The first anchor point image predicted by the degradation decoder. The total number of image samples. This is the sample index.

[0112] Phase 4, Scale Modulation Representation Learning: Freeze the trained content encoder and degenerate encoder; use the low-resolution image obtained by bicubic downsampling of the high-resolution image and random scaling of the downsampling as the second anchor image;

[0113] The second anchor image is simultaneously input into the content encoder, the degradation encoder, and the scale encoder to be trained to obtain content representation, scale-independent degradation representation, and scale modulation representation, respectively.

[0114] The three representations are fused by kernel modulation convolution operation, and the fusion result is input into the scale decoder to obtain the reconstructed image. At the same time, a second contrastive learning sample set is constructed based on scale consistency. The reconstructed image is constrained by the second anchor image itself, and the scale encoder is jointly trained by the second positive and negative sample pairs in the second contrastive learning sample set through contrastive learning loss, so that it learns to extract scale modulation representations.

[0115] Please see Figure 6 In the scale modulation representation learning stage, joint training is performed through contrastive learning and anchor image reconstruction; a second contrastive learning sample set based on scale consistency is constructed, in which the second anchor image... Let be a low-resolution image obtained through bicubic downsampling with randomly varying downsampling scales; assuming a total of The second positive and negative sample pair, for the first A second sample pair The second positive sample For the second anchor point image Low-resolution images with the same bicubic downsampling scale but different content, the second negative sample For the second anchor point image Low-resolution images with the same content but different bicubic downsampling scales;

[0116] The total loss function in the scale-modulated representation learning stage has the following relationship:

[0117] ;

[0118] in, The weighting coefficients corresponding to the second reconstruction loss. For the second reconstruction loss, For the second contrastive learning loss, The total loss function for the scale-modulated representation learning stage

[0119] The second contrastive learning loss has the following relationship:

[0120] ;

[0121] in, For scale encoders, In this embodiment, the preset threshold is used. ; This is the second positive sample. For the second anchor point image, This is the second negative sample;

[0122] The second reconstruction loss has the following relationship:

[0123] ;

[0124] in, This is the second anchor point image predicted by the scale decoder.

[0125] Phase 5, Conditional Diffusion Model Training: Freeze the trained content encoder, degradation encoder, and scale encoder; perform random scale bicubic downsampling on the high-resolution image to obtain a low-resolution image for training.

[0126] The low-resolution images used for training are input into the content encoder, the degradation encoder, and the scale encoder, respectively, to obtain content representation, scale-independent degradation representation, and scale modulation representation.

[0127] The content representation, scale-independent degradation representation, and scale-modulated representation are generated as conditional representations through kernel-modulated convolution operations of the conditional representation generation unit.

[0128] Starting with the low-resolution image used for training, its value at time step is calculated through a forward diffusion process. Noise version This yields a noisy image;

[0129] The conditional representation and the noisy image are stitched together along the channel dimension, and then along with the time step. Both are used as inputs to the conditional diffusion model. By minimizing the variational lower bound of the negative log-likelihood, the model learns to approximate the distribution of the low-resolution image after bicubic degradation at the target scale, thus enabling the training of the conditional diffusion model.

[0130] Please see Figure 7 During this training phase, the conditional diffusion model, guided by conditional representations, is trained using the denoising diffusion probability model (IDDPM) framework, ensuring that the image distribution it learns matches the expected input distribution of subsequent arbitrary-scale super-resolution modules.

[0131] ① The low-resolution image used for training each training sample Random sampling time step And sample random Gaussian noise The noisy image is calculated based on the forward diffusion formula. ,in The cumulative noise figure is defined during the forward diffusion process and is used to control the degree of noise addition. This represents the total time step;

[0132] ② Based on Bayes' theorem, using low-resolution images for training and noisy images Calculate the true posterior distribution true posterior distribution mean and the variance of the true posterior distribution (i.e., the theoretical baseline variance of the coefficients defined by the noise dispatcher).

[0133] ③ Add noise to the image Time step and conditional representation Input conditional diffusion model, output predicted noise and variance prediction parameters This leads to the posterior distribution predicted by the model. mean and variance .

[0134] ④ Training is achieved by minimizing the variational lower bound of the negative log-likelihood, and optimization is performed using a hybrid loss function: ,in Weighting coefficients; noise prediction loss For the simplified mean squared error loss, i.e. The aim is to enable the conditional diffusion model to work on a given noisy image. Time step and conditions Predicting the addition of random Gaussian noise under the premise of... Variational lower bound loss By calculating the true posterior distribution posterior distribution predicted by the model The KL divergence between them is obtained when time step The loss term is the negative log-likelihood of the decoder.

[0135] Once the arbitrary-scale super-resolution module pre-training, content representation learning, scale-independent degradation representation learning, scale modulation representation learning, and conditional diffusion model training are completed, the content decoder, degradation decoder, and scale decoder are removed to complete the overall system training.

[0136] Please refer to Figure 8 This embodiment also provides a method for arbitrary-scale super-resolution of real images based on unpaired data, wherein the method applies the aforementioned arbitrary-scale super-resolution system for real images based on unpaired data, and the method includes the following steps:

[0137] Provide an unpaired high-resolution image as a reference image;

[0138] The reference image is downsampled bicubic based on the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale.

[0139] By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image.

[0140] Based on the target's super-resolution scale, perform image super-resolution reconstruction on the target's low-resolution image to obtain a super-resolution image.

[0141] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A real-scale super-resolution system for unpaired data of real images, characterized in that, The system includes: The reference image preprocessing module is used for: An unpaired high-resolution image is provided as a reference image. The reference image is then bicubic downsampled according to the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale. The degradation domain adaptive module is used for: By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image. The arbitrary-scale super-resolution module, connected to the degradation domain adaptive module, is used for: Based on the target super-resolution scale, perform image super-resolution reconstruction on the target low-resolution image to obtain a super-resolution image; The degradation domain adaptive module includes: The content encoder is used to encode low-resolution images of the real scene to be processed, thereby obtaining a content representation. A degradation encoder is used to encode a low-resolution reference image with bicubic degradation characteristics at the target scale to obtain a scale-independent degradation representation. A scale encoder is used to encode a low-resolution reference image with target-scale bicubic degradation characteristics to obtain a scale modulation characterization. The conditional representation generation unit, connected to the content encoder, degradation encoder, and scale encoder, is used to dynamically generate a degradation kernel adapted to the input degradation state based on scale-independent degradation representation and scale modulation representation, and to convolve the content representation using the degradation kernel to generate conditional representation. The conditional diffusion model, connected to the conditional representation generation unit, is used to concatenate the conditional representation with the noisy image in the channel dimension. Through an iterative denoising process controlled by time steps, it generates a target low-resolution image whose content is consistent with the real scene low-resolution image and has the target scale bicubic degradation characteristics.

2. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 1, characterized in that, Encoding a low-resolution reference image with bicubic degradation characteristics at the target scale yields a scale-independent degradation representation, specifically including the following steps: A low-resolution reference image with bicubic degradation characteristics at the target scale is input into a convolutional layer for processing to extract the first shallow layer features; The first shallow features are input into a residual convolutional network, and deep features are extracted through multiple sequentially connected residual blocks to obtain the extracted deep features. The extracted deep features are sequentially processed through two subsequent activation functions. A convolutional layer, a pooling layer, and two subsequent activation functions. The convolutional layer process outputs a scale-independent degradation representation.

3. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 2, characterized in that, Encoding a low-resolution reference image with bicubic degradation characteristics at the target scale yields a scale modulation representation, specifically including the following steps: A low-resolution reference image with bicubic degradation characteristics at the target scale is input into a convolutional layer for processing to extract the second shallow layer features; The second shallow layer features are input into a network consisting of three sequentially connected scale feature processing units and a convolutional layer to obtain scale-related deep features. Each scale feature processing unit includes two sequentially connected Transformer blocks that employ a sliding window attention mechanism. Scale-aware location encoding modulated by the target super-resolution scale is introduced into the sliding window attention mechanism of the Transformer blocks. The scale-correlated deep features are residually concatenated with the second shallow features, and then subjected to average pooling and linear transformation in sequence to output the scale-modulated representation.

4. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 3, characterized in that, The sliding window attention mechanism of the Transformer block introduces scale-aware position encoding modulated by the target super-resolution scale, specifically including the following steps: The input feature map is divided into multiple parts of size [missing information]. The non-overlapping windows are used, and the features within each window are linearly projected to obtain the query vector, key vector, and value vector, respectively. Obtain the relative coordinates between each token within the window, concatenate the relative coordinates with the preset target super-resolution scale information in the height and width directions, input the data into a lightweight network consisting of two linear layers, and generate scale-aware position codes. The scale-aware position encoding is embedded as a bias term in the window self-attention computation, and the corresponding process has the following relationship: ； in, Calculate window self-attention; , , These are the query vector, key vector, and value vector, respectively. For query vector and key vector Dimensions Static position encoding for embedding scale-free information. Dynamic position encoding for embedding scale information, For normalized exponential functions, This is a transpose operation.

5. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 4, characterized in that, Based on scale-independent degradation representation and scale-modulated representation, a degradation kernel adapted to the input degradation state is dynamically generated, and the content representation is convolved using the degradation kernel to generate conditional representation. The specific steps include the following: Scale-independent degeneracy representation and scale modulation characterization The representations are stitched together along the channel dimension. The stitched representations are then first fed into a fusion network for fusion, and then into a kernel generation network to generate the basic degenerate kernels. ; Characterizing scale modulation Input the affine transform coefficient generation network to generate scaling coefficients. and offset coefficient ; Using scaling factor and offset coefficient For basic degenerate nuclei Perform channel-level affine modulation to obtain the modulation degenerate kernel. The affine modulation process has the following relationship: ； in, To multiply each channel, To add channel by channel, Based on the degenerate nucleus, This is the scaling factor. This is the offset coefficient. To modulate the degenerate nucleus; Modulation degenerate nucleus As a convolution kernel, it represents the content. Perform depthwise convolution operations to generate conditional representations. The depthwise convolution operation has the following relationship: ； in, For depthwise convolution operations, For content representation, This is a conditional characterization.

6. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 5, characterized in that, The conditional representation and the noisy image are concatenated along the channel dimension, and an iterative denoising process based on time step control is used to generate a target low-resolution image whose content is consistent with the real scene low-resolution image and has the target scale bicubic degradation characteristics. The specific steps include the following: A partial noise initialization strategy based on the input image is adopted, and the low-resolution image of the real scene is denoted as... Set the start time step And sample random Gaussian noise The initial noise image is calculated according to the noise addition formula, and the corresponding process has the following relationship: ； in, For the initial noisy image, For the noise scheduler at time step Defined cumulative noise figure, Low-resolution images of real-world scenes. The starting time step, It is positive random Gaussian noise. , This represents a standard normal distribution, where 0 indicates a vector with zero mean. The unit covariance matrix; Noise image As the initial state for the reverse denoising process, it is obtained from time step 1 through the conditional diffusion model. Start gradually reducing noise to For each time step The operation to be performed is as follows: Add noise to the image With conditional representation Channels are concatenated along the channel dimension and input together with the current time step t into the conditional diffusion model to obtain the predicted noise. and variance prediction parameters ; Based on the predicted noise, the predicted target image is calculated, and the corresponding process has the following relationship: ； in, For the predicted target image, For the noise scheduler at time step The defined cumulative noise figure, For a noisy image; Based on the predicted target image, the mean of the predicted posterior distribution is calculated, and the corresponding process has the following relationship: ； in, To predict the mean of the posterior distribution, The coefficients defined for the noise dispatcher, For the noise scheduler at time step The cumulative noise figure as defined; Based on variance prediction parameters Coefficients defined by the noise scheduler The theoretical baseline variance of the coefficients defined by the noise dispatcher. The calculation of the predicted posterior distribution variance follows the following relationship: ； in, To predict the variance of the posterior distribution, For variance prediction parameters, The theoretical baseline variance calculated for the coefficients defined for the noise dispatcher. ; Using the predicted posterior distribution mean and the variance of the predicted posterior distribution The image at the next time step is obtained by sampling the square root of the image, and the corresponding process has the following relationship: ； in, Inverse random Gaussian noise is used to introduce randomness during the backsampling process. ; When noise reduction reaches the time step At that time, a low-resolution image of the target is obtained.

7. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 6, characterized in that, It also includes a training module, which uses a phased training approach to train the arbitrary-scale super-resolution module, the degradation domain adaptive module, and the conditional diffusion model. The specific steps are as follows: Phase 1: Pre-training of arbitrary scale super-resolution module: Obtain training samples of low-resolution-high-resolution image pairs with bicubic degradation characteristics, and pre-train the arbitrary scale super-resolution module using a large number of training samples to obtain the pre-trained arbitrary scale super-resolution module. Phase 2, Content Representation Learning: High-resolution images from the training samples are acquired, and low-resolution images obtained by randomly synthesizing and degrading the high-resolution images and randomly changing the downsampling scale are used as content representation learning data. The low-resolution image obtained by bicubic downsampling the same high-resolution image at the same random scale is used as the supervision target image; The content representation learning data is processed sequentially through the content encoder and content decoder, and the output of the content decoder is constrained by the supervised target image. In this way, the content encoder is trained to extract content representation by minimizing the reconstruction error and the representation discretization constraint. Phase 3, Scale-Independent Degradation Representation Learning: Freeze the trained content encoder; use a low-resolution image, which is randomly degraded from a high-resolution image at a predefined baseline scale, as the first anchor image; The first anchor image is simultaneously input into the content encoder and the degradation encoder to be trained to obtain content representation and scale-independent degradation representation, respectively. Degeneracy kernels generated from scale-independent degradation representations are used as convolution kernels. After performing convolution operations on the content representations, the kernels are input into the degradation decoder to obtain the reconstructed image. At the same time, a first contrastive learning sample set is constructed based on degradation consistency. The image is reconstructed using the constraints of the first anchor point image itself, and the degradation encoder is jointly trained by the first positive and negative sample pairs of the first contrastive learning sample set through contrastive learning loss to learn the essential differences between different degradation types and form a scale-independent degradation representation. Phase 4, Scale Modulation Representation Learning: Freeze the completed content encoder and degenerate encoder; The low-resolution image obtained by bicubic downsampling of the high-resolution image, with the downsampling scale varying randomly, is used as the second anchor image. The second anchor image is simultaneously input into the content encoder, the degradation encoder, and the scale encoder to be trained to obtain content representation, scale-independent degradation representation, and scale modulation representation, respectively. The three representations are fused by kernel modulation convolution operation, and the fusion result is input into the scale decoder to obtain the reconstructed image; at the same time, a second contrastive learning sample set is constructed based on scale consistency. The image is reconstructed using the constraints of the second anchor point image itself, and the scale encoder is jointly trained by the second positive and negative sample pairs in the second contrastive learning sample set through contrastive learning loss, so that it learns to extract scale modulation representations. Phase 5, Conditional Diffusion Model Training: Freeze the trained content encoder, degradation encoder, and scale encoder; The high-resolution image is downsampled at random scales in a bicubic manner to obtain a low-resolution image for training. The low-resolution images used for training are input into the content encoder, the degradation encoder, and the scale encoder, respectively, to obtain content representation, scale-independent degradation representation, and scale modulation representation. The content representation, scale-independent degradation representation, and scale-modulated representation are generated as conditional representations through kernel-modulated convolution operations of the conditional representation generation unit. Starting with the low-resolution image used for training, its value at time step is calculated through a forward diffusion process. Noise version This yields a noisy image; The conditional representation and the noisy image are stitched together along the channel dimension, and then along with the time step. Both are used as inputs to the conditional diffusion model. By minimizing the variational lower bound of the negative log-likelihood, the model learns to approximate the distribution of the low-resolution image after bicubic degradation at the target scale, thus enabling the training of the conditional diffusion model. Once the arbitrary-scale super-resolution module pre-training, content representation learning, scale-independent degradation representation learning, scale modulation representation learning, and conditional diffusion model training are completed, the content decoder, degradation decoder, and scale decoder are removed to complete the overall system training.

8. The real-image arbitrary-scale super-resolution system based on unpaired data according to claim 7, characterized in that, The total loss function in the content representation learning phase has the following relationship: ； in, For content representation learning data, Let the total loss function be the content representation learning phase. Supervised target images are bicubic downsampled at the same random scale. It consists of a convolutional module, a residual convolutional network, and linear layers in the content encoder; For content decoder functions, The distance is the squared Euclidean distance. To stop the gradient operator, Hyperparameters for controlling the weights of commitment loss; The total loss function for the scale-independent degenerate representation learning stage has the following relationship: ； in, The weighting coefficients corresponding to the first reconstruction loss are: Let be the total loss function for the scale-independent degenerate representation learning phase. The first reconstruction loss, The first contrastive learning loss; The first contrastive learning loss has the following relationship: ； in, For a degraded encoder, For the preset threshold, As the first positive sample, For the first anchor point image, This is the first negative sample; The first reconstruction loss has the following relationship: ； in, The first anchor point image predicted by the degradation decoder. The total number of image samples. For sample index; The total loss function in the scale-modulated representation learning stage has the following relationship: ； in, The weighting coefficients corresponding to the second reconstruction loss. For the second reconstruction loss, For the second contrastive learning loss, For the scale-modulated representation learning stage, the total loss function is denoted as . The second contrastive learning loss has the following relationship: ； in, For scale encoders, For the preset threshold, This is the second positive sample. For the second anchor point image, This is the second negative sample; The second reconstruction loss has the following relationship: ； in, This is the second anchor point image predicted by the scale decoder.

9. A super-resolution method for real images at arbitrary scales based on unpaired data, characterized in that, The method applies the arbitrary-scale super-resolution system for real images based on unpaired data as described in any one of claims 1 to 8, and the method includes the following steps: Provide an unpaired high-resolution image as a reference image; The reference image is downsampled bicubic based on the target super-resolution scale to obtain a low-resolution reference image with bicubic degradation characteristics at the target scale. By using a low-resolution image of the real scene to be processed and a low-resolution reference image with bicubic degradation characteristics at the target scale, an adaptive fusion of content and degradation characteristics is performed to obtain the target low-resolution image. Based on the target's super-resolution scale, perform image super-resolution reconstruction on the target's low-resolution image to obtain a super-resolution image.

Images