Image processing method and device, program product and electronic equipment

Abstract

The embodiment of the invention relates to an image processing method and device, a computer program product and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: carrying out the degradation processing of a plurality of sample high-resolution images based on an under-trained image reconstruction model, and generating a plurality of sample low-resolution images, forming a training data set according to the plurality of sample high-resolution images and the plurality of sample low-resolution images; obtaining a super-resolution model, determining a training condition of the super-resolution model, and performing model processing on the super-resolution model in the training data set based on the training condition to obtain a target super-resolution model; and carrying out image super-resolution processing on a low-resolution image based on the target super-resolution model to generate a high-resolution image. According to the invention, the matching performance and the generation efficiency of the training data set can be improved, and the image super-resolution effect is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to an image processing method, an image processing apparatus, a computer program product, and an electronic device. Background Art

[0002] Image super-resolution technology aims to improve visual quality by using algorithms to upscale low-resolution images to high-resolution. Learning-based image super-resolution methods typically rely on constructing pairs of high- and low-resolution images to learn the mapping from low to high resolution. Real-world image degradation is complex, so blind super-resolution can be used to restore high-quality high-resolution images from low-resolution images without knowing the specific degradation model.

[0003] The aforementioned blind super-resolution process typically obtains LR-HR training image pairs through the following methods: expensive equipment and a large amount of manpower for data collection to capture paired LR-HR image pairs, which is inefficient and costly. This method, which employs more complex modeling of the degradation model, does not fundamentally address the complexity of real-world degradation. Using a more complex degradation model to create paired LR-HR training pairs only fits the super-resolution model to a more complex, pre-defined degradation model, without ensuring that this degradation model matches the real world. Using generative adversarial networks to learn degradation models for real-world LR images can lead to training instability and prone to crashes. Summary of the Invention

[0004] The purpose of the present disclosure is to provide an image processing method, an image processing device, a computer program product, and an electronic device, thereby overcoming, at least to a certain extent, the problem of being unable to maximize network bandwidth during image processing due to limitations and defects in related technologies.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: performing degradation processing on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and forming a training data set based on the plurality of sample high-resolution images and the plurality of sample low-resolution images; obtaining a super-resolution model and determining a training status of the super-resolution model, performing model processing on the super-resolution model in the training data set based on the training status to obtain a target super-resolution model; and performing image super-resolution processing on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

[0006] In an exemplary embodiment of the present disclosure, the degradation processing of multiple sample high-resolution images based on the under-trained image reconstruction model to generate multiple sample low-resolution images includes: downsampling the multiple high-resolution images to obtain multiple intermediate images; fitting the multiple intermediate images through at least one under-trained image reconstruction model to determine multiple low-resolution images; wherein the degradation degrees of the multiple low-resolution images are different.

[0007] In an exemplary embodiment of the present disclosure, fitting the multiple intermediate images through at least one under-trained image reconstruction model to determine a plurality of low-resolution images includes: encoding each intermediate image through an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and decoding the encoding result through a decoder to determine the low-resolution image.

[0008] In an exemplary embodiment of the present disclosure, encoding each intermediate image by an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and decoding the encoding result by a decoder to determine the low-resolution image, including: extracting feature vectors of the intermediate image through multi-layer convolution operations, and performing feature mapping on the extracted feature vectors to obtain a discrete latent space representation; and generating the low-resolution image by performing layer-by-layer downsampling and convolution operations on the discrete latent space representation.

[0009] In an exemplary embodiment of the present disclosure, the performing model processing on the super-resolution model in the training dataset based on the training situation to obtain a target super-resolution model includes: if the training situation is untrained, performing model training on the super-resolution model based on the training dataset to obtain the target super-resolution model; if the training situation is trained, performing model fine-tuning on the super-resolution model based on the training dataset to obtain the target super-resolution model.

[0010] In an exemplary embodiment of the present disclosure, fine-tuning the super-resolution model based on the training dataset to obtain the target super-resolution model includes: fixing the model parameters of a first part of the super-resolution model, and updating the model parameters of a second part of the super-resolution model based on the training dataset to determine the target super-resolution model.

[0011] In an exemplary embodiment of the present disclosure, super-resolution processing is performed on a low-resolution image based on the target super-resolution model to generate a high-resolution image, including: extracting features from the low-resolution image based on the target super-resolution model to obtain image features; and upsampling the image features to reconstruct the low-resolution image into a target high-resolution image.

[0012] According to one aspect of the present disclosure, there is provided an image processing device, comprising: a training data generation module, configured to perform degradation processing on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and to form a training data set based on the plurality of sample high-resolution images and the plurality of sample low-resolution images; a model processing module, configured to obtain a super-resolution model and determine a training status of the super-resolution model, and perform model processing on the super-resolution model in the training data set based on the training status to obtain a target super-resolution model; and an image reconstruction module, configured to perform image super-resolution processing on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

[0013] According to one aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned image processing methods.

[0014] According to one aspect of the present disclosure, there is provided an electronic device, including: a processor; and

[0015] A memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image processing methods by executing the executable instructions.

[0016] The technical solution provided in the embodiments of the present disclosure, on the one hand, can perform degradation processing on sample high-resolution images according to an under-trained image reconstruction model without relying on paired LR-HR training data, and generate rich LR images and training data sets that are more consistent with the degradation pattern of the real world. Since only high-resolution images need to be provided, the data acquisition cost is reduced, the efficiency of training data set generation is improved, and the diversity and comprehensiveness of training data are improved. On the other hand, since the low-resolution model is automatically generated by the under-trained image reconstruction model, LR images with different degrees of degradation can be obtained, and the matching and accuracy of the output low-resolution image with the degradation pattern of the real world are improved, the problems of unstable training and easy collapse are avoided, and the reliability of training data set generation is improved. On the other hand, the sample low-resolution model is automatically generated by the under-trained image reconstruction model, and the generated training data set is applicable to any super-resolution model, which improves versatility and increases the scope of application.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0019] Figure 1 The following is a flow chart schematically illustrating an image processing method in an embodiment of the present disclosure.

[0020] Figure 2 The overall framework diagram of the blind super-resolution method combined with an under-trained image reconstruction model in an embodiment of the present disclosure is schematically shown.

[0021] Figure 3 The flowchart of how an under-trained image reconstruction model acquires a training data set is schematically shown.

[0022] Figure 4 The specific flow chart of image super-resolution processing is schematically shown.

[0023] Figure 5 The following schematically shows a block diagram of an image processing device in an embodiment of the present disclosure.

[0024] Figure 6 The block diagram of the electronic device according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] In some embodiments, learning-based image super-resolution methods typically rely on the construction of pairs of high-resolution and low-resolution images to learn the mapping from low resolution to high resolution. A common method for generating paired datasets is to degrade high-resolution images to generate corresponding low-resolution images, where Gaussian blur is one of the most commonly used degradation kernels. However, the image degradation process in the real world is much more complicated than this idealized degradation model. Actual degradation may involve multiple factors, such as motion blur, noise interference, optical distortion, and complex downsampling processes, which work together to cause a decrease in image quality. In addition, the degradation patterns in different scenarios are also different, which makes it difficult for a single degradation model to accurately describe the image degradation process in the real world.

[0028] This complexity leads to the problem of blind super-resolution, which involves recovering a high-quality high-resolution image from a low-resolution image without knowing the specific degradation model. Blind super-resolution not only requires restoring image detail and clarity, but also requires accurately estimating and reversing the image degradation process in the absence of explicit degradation information.

[0029] In the aforementioned blind super-resolution process, LR-HR training image pairs are typically obtained through the following methods: expensive equipment and a large amount of manpower are used for data collection to capture paired LR-HR images, which is inefficient and costly. This method not only requires significant resources and manpower, but also primarily captures static scenes such as buildings. It is difficult to capture two consecutive, identically angled images of dynamic scenes such as animals, resulting in certain limitations. The degradation model is further complicated, such as by introducing noise and compression on top of Gaussian blur. This method does not fundamentally address the complexity of real-world degradation. Using a more complex degradation model to create paired LR-HR training images only allows the super-resolution model to fit a more pre-defined, complex degradation model, without guaranteeing that this degradation model matches the real world. Using generative adversarial networks to learn degradation models for real-world LR images can lead to training instability and prone to corruption.

[0030] In order to solve the above technical problems, an image processing method is provided in the embodiments of the present disclosure, which is used to perform degradation processing on a high-resolution image based on an under-trained image reconstruction model to generate a low-resolution image, thereby generating a training data set, and performing blind super-resolution processing on the low-resolution image based on the training data set to obtain a high-resolution image. The method can be applied to scenarios where blind super-resolution processing is performed on any type of low-resolution image. Figure 1 As shown in , the image processing method mainly includes the following steps:

[0031] In step S110, degradation processing is performed on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and a training data set is formed based on the plurality of sample high-resolution images and the plurality of sample low-resolution images;

[0032] In step S120, a super-resolution model is obtained and a training status of the super-resolution model is determined, and model processing is performed on the super-resolution model in the training dataset based on the training status to obtain a target super-resolution model;

[0033] In step S130, image super-resolution processing is performed on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

[0034] The technical solution in the disclosed embodiments, on the one hand, can perform degradation processing on sample high-resolution images according to an under-trained image reconstruction model without relying on paired LR-HR training data, and generate rich LR images and training data sets that are more consistent with the real-world degradation pattern. Since only high-resolution images need to be provided, the data acquisition cost is reduced, the efficiency of training data set generation is improved, and the diversity and comprehensiveness of training data are improved. On the other hand, since the low-resolution model is automatically generated by the under-trained image reconstruction model, LR images with different degrees of degradation can be obtained, and the matching and accuracy of the output low-resolution image with the real-world degradation pattern are improved, the problems of unstable training and easy collapse are avoided, and the reliability of training data set generation is improved. On the other hand, the sample low-resolution model is automatically generated by the under-trained image reconstruction model, and the generated training data set is applicable to any super-resolution model, which improves versatility and increases the scope of application.

[0035] Next, refer to Figure 1 The image processing method in the embodiment of the present disclosure is specifically described in detail.

[0036] In step S110, degradation processing is performed on multiple sample high-resolution images based on an under-trained image reconstruction model to generate multiple sample low-resolution images, and a training data set is composed based on the multiple sample high-resolution images and the multiple sample low-resolution images.

[0037] In the disclosed embodiments, low-resolution images refer to images with low resolution, missing details, and reduced visual quality. In super-resolution tasks, low-resolution images are usually obtained through degradation processes such as downsampling, blurring, noise pollution, or compression distortion. High-resolution images are images with richer details, clearer edges, and higher pixel density. High-resolution images are usually original, non-degraded images and are regarded as target images in super-resolution tasks. The goal of the image super-resolution task is to use an algorithm to upgrade low-resolution images to high-resolution images, thereby improving the visual effect. The sample high-resolution image can be a static image or a dynamic image, which is not specifically limited here.

[0038] An undertrained image reconstruction model refers to an image reconstruction model that has not been trained enough. The undertrained image reconstruction model has not yet fully converged and still retains high-frequency information. The image reconstruction model can be a model used to determine the training samples of the target super-resolution model. The image reconstruction model can still maintain certain structural information in an undertrained state, while introducing natural degradation characteristics in the low-resolution image generation process, thereby being closer to the degradation process of the real world. In some embodiments, the image reconstruction model can be a VQ-VAE (Vector Quantised Variational AutoEncoder) model, a VQ-VAE2 (Vector Quantised Variational AutoEncoder2, a second generation of vector quantised variational autoencoders) model, or a MAE model (Masked Autoencoders). For example, the undertrained image reconstruction model can be a VQ-VAE2 trained for 16 rounds, a VQ-VAE2 trained for 32 rounds, or a VQ-VAE trained for 64 rounds.

[0039] Figure 2 The overall framework of the blind super-resolution method combined with the under-trained image reconstruction model is schematically shown in Figure 2 As shown in , it can include a training data generation module, a training module, and an inference module. The training data generation module includes a downsampling component and an image reconstruction model calling component. The image reconstruction model calling component will pull the undertrained image reconstruction model from the multi-scale undertrained model warehouse and call it to generate paired LR-HR training data. The training module includes a training data loading component and a model training component. The model training component will pull the super-resolution model from the super-resolution model warehouse for model training or fine-tuning. The inference module includes an LR test image loading component and a model inference component.

[0040] In some embodiments, reference Figure 2As shown in , a plurality of sample high-resolution images can be degraded by a training data generation module to generate a plurality of sample low-resolution images. Degradation processing refers to the operation of simulating the degradation process of real-world images to generate corresponding low-resolution images. In the embodiment of the present disclosure, the specific process of degradation processing can be: downsampling a plurality of sample high-resolution images to obtain a plurality of intermediate images, and inputting the plurality of intermediate images into at least one under-trained image reconstruction model for fitting to determine a plurality of sample low-resolution images. Downsampling refers to simulating the resolution loss during image acquisition or transmission by reducing the resolution of the sample high-resolution image. The downsampling method may include bicubic interpolation or nearest neighbor interpolation. In addition, degradation processing of high-resolution images can also be achieved by blurring, adding noise or compression. Downsampling by bicubic interpolation is used as an example for explanation.

[0041] For example, for each pixel in the sample high-resolution image, its corresponding coordinate position in the original image is found. Sixteen pixels surrounding that coordinate position are selected, and weights for these 16 pixels are calculated using the bicubic interpolation basis function. The weights of these 16 pixels are summed to obtain the value of the corresponding pixel in the intermediate image. This value is then used to determine the intermediate image.

[0042] After downsampling the sample high-resolution image to obtain an intermediate image, the intermediate image can be input into an undertrained image reconstruction model to obtain a sample low-resolution image. For example, the undertrained image reconstruction model may include any one of the models such as VQ-VAE, VQ-VAE2, MAE, etc. The undertrained image reconstruction model may include an encoder and a decoder. Based on this, the intermediate image can be input into the encoder, and the encoder converts the input intermediate image into a discrete latent space representation. The encoder performs multi-layer convolution operations to extract the feature vector of the intermediate image, and maps the extracted feature vector to a discrete latent space to determine the discrete latent space representation. The decoder decodes the discrete latent space representation. For example, the discrete latent space representation is input into the decoder, and the convolutional neural network in the decoder generates the details of the image through layer-by-layer downsampling and convolution operations, thereby obtaining a sample low-resolution image. The downsampling operation is used to reduce the resolution of the intermediate image, while the convolution operation is used to extract and combine the features of the image.

[0043] It should be noted that for each sample high-resolution image, one or more undertrained image reconstruction models can be selected. Based on this, multiple sample high-resolution images can be downsampled to obtain multiple intermediate images, which can then be encoded and decoded using one or more undertrained image reconstruction models to obtain multiple sample low-resolution images with varying degrees of degradation. For the same sample high-resolution image, processing it with different undertrained image reconstruction models will yield different sample low-resolution images. For different sample high-resolution images, processing them with the same undertrained image reconstruction model will also yield different sample low-resolution images.

[0044] refer to Figure 2 As shown in , the training data generation module may include a downsampling component and an image reconstruction model calling component. Based on this, the downsampling component can downsample the sample high-resolution image y to obtain a low-resolution intermediate result as the intermediate image x'. Next, the image reconstruction model calling component is used to obtain an undertrained image reconstruction model from the multi-scale undertrained model warehouse. The multi-scale undertrained model warehouse may include multiple undertrained image reconstruction models, and the multiple image reconstruction models may include at least VQ-VAE, VQ-VAE2, and MAE, and each image reconstruction model has an undertrained version with a different number of training rounds. Each undertrained image reconstruction model has the same probability of being selected. Based on this, an undertrained image reconstruction model can be randomly selected. Here, the undertrained image reconstruction model is an undertrained VQ-VAE2 model as an example for explanation.

[0045] For example, the low-resolution intermediate image x' can be input into the under-trained image reconstruction model for fitting to obtain the final low-resolution image x, which can be specifically referred to as shown in formula (1):

[0046] G θ :x'→x Formula (1)

[0047] Among them, G θ represents the under-trained image reconstruction model obtained from the multi-scale under-trained model warehouse, and θ represents the parameters in the model.

[0048] It should be noted that the resolution of the intermediate image is greater than the resolution of the low-resolution image and smaller than the resolution of the sample high-resolution image.

[0049] Furthermore, the sample high-resolution image and the generated sample low-resolution image can be combined into a training image pair, thereby obtaining a training dataset D that is closer to the real world. The training dataset D may include multiple training image pairs. The training dataset D can be expressed as formula (2):

[0050]

[0051] Where i represents the subscript of each training image pair in the training dataset D, and there are a total of N pairs of LR-HR training image pairs.

[0052] Figure 3 The flowchart of obtaining training data set for under-trained image reconstruction model is shown schematically in Figure 3 As shown in , the HR image can be downsampled to obtain an intermediate image, which can be a LR image with a lower resolution than the HR image. Furthermore, an undertrained image reconstruction model can be selected from a multi-scale undertrained model repository. The encoder and decoder in the undertrained image reconstruction model can be used to perform image fitting on the input intermediate image to obtain the LR image. The LR image can have a lower resolution than the intermediate image. Based on this, the HR image and the LR image can be combined into training image pairs, and a training dataset can be generated based on multiple training image pairs.

[0053] In the disclosed embodiment, unlike the traditional training method based on a fixed degradation model, the image reconstruction model calling component is based on a multi-scale generation strategy. It uses at least one under-trained image reconstruction model to extract features from multiple sample high-resolution images to obtain a discrete latent space representation, and then downsamples and convolves the discrete latent space representation. Since there are under-trained models with different training levels, it is possible to generate sample low-resolution images that are more consistent with the real-world degradation process based on multiple high-resolution images. Pull the under-trained VQ-VAE2 from the model warehouse, load the models of different stages according to the set number of training rounds (such as 16, 32 epochs) to generate LR images with different degrees of degradation, thereby increasing the diversity and richness of low-resolution images. The rich and diverse LR-HR training image pairs help improve the robustness of the super-resolution model on real low-resolution images. The high-resolution image is encoded and decoded, and the high-frequency errors retained by VQ-VAE2 in the under-trained state are used to make the generated low-resolution image closer to the degradation in the real world.

[0054] Without relying on paired LR-HR training data, a training data set that is more consistent with the real-world degradation pattern can be automatically generated, thereby improving the compatibility with the real world. This training data set can be applied to any super-resolution model. The method proposed in the present disclosure is applicable to a variety of scenarios such as static images, videos, and remote sensing images, which increases the application scenarios and improves data diversity. A multi-scale generation strategy is used to select multiple under-trained image reconstruction models of different training rounds to generate LR images with different degrees of degradation, thereby improving generalization ability. By inputting the sample high-resolution image into the under-trained image reconstruction model, it is possible to output LR-HR training image pairs that are more consistent with the real world, generate high-quality training data for any super-resolution model, improve the convenience of generating training data, and provide a low-cost blind super-resolution method that is not restricted by the model.

[0055] Next, in step S120, a super-resolution model is obtained and a training condition of the super-resolution model is determined. Based on the training condition, model processing is performed on the super-resolution model in the training data set to obtain a target super-resolution model.

[0056] In the embodiment of the present disclosure, the target super-resolution model can be obtained through the training module. Figure 2 As shown in , the training module may include a training data loading component and a model training component.

[0057] The training data loading component can obtain and parse data from the training data set generated by the training data generation module, and provide the training image pairs obtained from the training data set to the model training component for model processing.

[0058] The model training component can obtain a super-resolution model from the super-resolution model warehouse and perform training or fine-tuning to complete the model processing. For example, the training status of the super-resolution model can be determined, and the training status can be untrained or trained. Different training statuses will result in different model processing methods for the super-resolution model. Model processing may include model training or model fine-tuning. For example, if the training status of the super-resolution model is untrained, the super-resolution model can be trained on the training dataset D. If the training status of the super-resolution model is trained, the super-resolution model can be fine-tuned on the training dataset D. For example, if an untrained super-resolution model without model weights is pulled from the super-resolution model warehouse, the model training component will train the model on the training dataset. If the model pulled from the super-resolution model repository is a trained super-resolution model, such as EDSR (Enhanced Deep Residual Networks for Single Image Super-Resolution), RCAN (Residual Channel Attention Network), SwinIR (Image Restoration Using Swin Transformer), or other model files with model weights, the model training component will fine-tune the training dataset to further improve the super-resolution effect of the super-resolution model on low-resolution images in the real world.

[0059] In some embodiments, if the model processing is model training, the model parameters of the super-resolution model can be updated with the minimum loss function as the optimization objective, and the learning rate can be dynamically adjusted during the training process to train the super-resolution model. The loss function can be L1 loss or L2 loss, which is not specifically limited here. For example, a larger learning rate can be used in the early stages of training to quickly approach the optimal solution; the learning rate can be reduced in the later stages of training to enable more precise adjustments to the model as it approaches the optimal solution to avoid oscillation, thereby achieving dynamic adjustment of the learning rate.

[0060] If the model processing involves fine-tuning, a smaller learning rate can be used to fine-tune the super-resolution model on the training dataset generated in the previous stage. For example, the model parameters of the first part of the super-resolution model can be frozen, while only the model parameters of the second part can be fine-tuned. The model parameters of the first part can be the model parameters of the bottom layer, and the model parameters of the second part can be the model parameters of the top layer or a specific layer. The choice here can be determined based on actual needs, as long as the first and second parts do not overlap.

[0061] After obtaining the target super-resolution model based on model training or model fine-tuning, the model weights of the target super-resolution model can be stored locally for subsequent inference module calls.

[0062] In the embodiments of this disclosure, the proposed model training component is compatible with a variety of super-resolution models, including but not limited to EDSR, SwinIR, RCAN, and other different types of super-resolution models. Therefore, the technical solution provided by this disclosure is not limited by model type, can improve the robustness of super-resolution models for real low-resolution images, and provide flexible and efficient super-resolution processing for a variety of tasks.

[0063] In step S130, image super-resolution processing is performed on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

[0064] In the embodiments of the present disclosure, the low-resolution image refers to an unprocessed low-resolution image. When the target super-resolution model is obtained, the low-resolution image can be subjected to image super-resolution processing. The goal of image super-resolution processing is to use an algorithm to upgrade the low-resolution image to a high-resolution image, thereby improving the visual effect. Since the specific degradation model is unknown, the super-resolution task in the embodiments of the present disclosure can be a blind super-resolution task. The blind super-resolution task not only needs to restore the details and clarity of the low-resolution image, but also needs to accurately estimate and reversely restore the image degradation process in the absence of clear degradation information.

[0065] refer to Figure 2 As shown in , the above-mentioned image super-resolution processing can be achieved through an inference module. The inference module may include a test image loading component and a model inference component. The test image loading component is used to standardize and preprocess the input low-resolution image to ensure that it meets the input requirements of the super-resolution model. The test image loading component is not only suitable for single-frame static images, but also supports frame-by-frame extraction of low-resolution videos, which is convenient for subsequent model inference. The model inference component can use the trained target super-resolution model or the fine-tuned target super-resolution model weights to perform super-resolution processing on low-resolution images. It is worth mentioning that in the embodiment of the present disclosure, reasoning can also be performed on low-resolution videos without true values. During the low-resolution video processing process, a low-resolution video sequence can be input into the target super-resolution model, which is reconstructed frame by frame by the target super-resolution model, and finally a high-quality high-resolution video sequence is output.

[0066] In some embodiments, after a low-resolution image is input into a target super-resolution model, the target super-resolution model may perform feature extraction on the low-resolution image to obtain image features; further, the extracted image features may be upsampled to expand the image features to a high resolution to obtain a reconstructed high-resolution image. Upsampling may be performed by a sub-pixel convolution (PixelShuffle) operation. For example, sub-pixel convolution may be a pixel reshuffle, which directly generates a high-resolution image by rearranging the pixels in the feature map represented by the image features of the low-resolution image without the need for additional interpolation operations.

[0067] Exemplarily, the target super-resolution model can be any one of EDSR, RCAN, and SwinIR. In the embodiment of the present disclosure, the target super-resolution model is taken as the SwinIR model as an example for illustration. First, the low-resolution image is input into the SwinIR model. The SwinIR model extracts shallow features of the low-resolution image through the convolution layer in the Swin Transformer architecture, and further extracts high-frequency features as deep features through the window shifting mechanism and self-attention mechanism. The shallow features and the deep features are fused to obtain image features. Through sub-pixel convolution and convolution layers, the fused features are upsampled to the high-resolution space to reconstruct a high-quality high-resolution image from the low-resolution image, thereby realizing super-resolution image processing.

[0068] Figure 4 The flowchart of image super-resolution processing is schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0069] Step S402: Build a multi-scale under-trained model warehouse.

[0070] Among them, models such as VQ-VAE, VQ-VAE2, and MAE can be trained on smaller datasets, with only 16-64 epochs trained to form a warehouse of under-trained image reconstruction models.

[0071] Step S404: Build a super-resolution model repository, which contains some super-resolution algorithm architectures or pre-trained models with model weights.

[0072] Step S406: construct LR-HR training image pairs.

[0073] At least one undertrained image reconstruction model can be pulled from a multi-scale undertrained model repository to downsample multiple sample high-resolution images to obtain multiple intermediate images. The multiple intermediate images are then encoded and decoded based on the at least one undertrained image reconstruction model to generate multiple sample low-resolution images with varying degrees of degradation. Furthermore, the sample high-resolution images and sample low-resolution images can be combined into training image pairs, and a training dataset can be generated based on the multiple training image pairs.

[0074] Step S408: Pull the super-resolution model from the super-resolution model warehouse.

[0075] Step S410, determine whether the training status of the pulled super-resolution model is trained; if so, go to step S412; if not, go to step S414.

[0076] Step S412: fine-tune the super-resolution model to obtain a target super-resolution model.

[0077] Step S414: train the super-resolution model to obtain a target super-resolution model.

[0078] Step S416: performing image super-resolution processing on the low-resolution model based on the target super-resolution model to generate a high-resolution image.

[0079] In the disclosed embodiment, without relying on paired LR-HR training data, there is no need to collect real LR-HR training image pairs. The sample high-resolution images can be degraded according to the under-trained image reconstruction model, and rich LR images and training data sets that are more consistent with the real-world degradation pattern can be automatically generated, thereby improving the diversity and comprehensiveness of the training data. In addition, the generated training data set is applicable to any super-resolution model, which improves versatility. Since the low-resolution model is automatically generated by the under-trained image reconstruction model, LR images with different degrees of degradation can be obtained, so that the output training data is more consistent with the degradation mode of the real world, the generalization ability of the model is improved, and the matching and accuracy of the output low-resolution images with the degradation mode of the real world are improved. Since only HR images need to be provided, the training data is more consistent with the real world, manual collection is avoided, the data collection cost is reduced, and the efficiency of training data set generation is improved. The generation of sample low-resolution images by the under-trained image reconstruction model avoids the problems of unstable training and easy collapse, and improves the reliability of training data set generation.

[0080] The low-resolution image generation strategy provided in the disclosed embodiments can be seamlessly integrated with the existing super-resolution model training process, eliminating the need for additional modifications to the super-resolution model's network structure, further improving training efficiency and model versatility. The target super-resolution model derived from image processing is not only suitable for static image super-resolution tasks, but can also be applied to multiple fields, including video super-resolution (video enhancement) and remote sensing image super-resolution (satellite image resolution enhancement), expanding its scope of application and versatility.

[0081] The present disclosure also provides an image processing device. Figure 5 As shown, the image processing device 500 mainly includes the following modules:

[0082] A training data generation module 501 is configured to perform degradation processing on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and to form a training data set based on the plurality of sample high-resolution images and the plurality of sample low-resolution images;

[0083] A model processing module 502 is configured to obtain a super-resolution model and determine a training status of the super-resolution model, and perform model processing on the super-resolution model in the training dataset based on the training status to obtain a target super-resolution model;

[0084] The image reconstruction module 503 is configured to perform image super-resolution processing on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

[0085] In an exemplary embodiment of the present disclosure, the degradation processing of multiple sample high-resolution images based on the under-trained image reconstruction model to generate multiple sample low-resolution images includes: downsampling the multiple high-resolution images to obtain multiple intermediate images; fitting the multiple intermediate images through at least one under-trained image reconstruction model to determine multiple low-resolution images; wherein the degradation degrees of the multiple low-resolution images are different.

[0086] In an exemplary embodiment of the present disclosure, fitting the multiple intermediate images through at least one under-trained image reconstruction model to determine a plurality of low-resolution images includes: encoding each intermediate image through an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and decoding the encoding result through a decoder to determine the low-resolution image.

[0087] In an exemplary embodiment of the present disclosure, encoding each intermediate image by an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and decoding the encoding result by a decoder to determine the low-resolution image, including: extracting feature vectors of the intermediate image through multi-layer convolution operations, and performing feature mapping on the extracted feature vectors to obtain a discrete latent space representation; and generating the low-resolution image by performing layer-by-layer downsampling and convolution operations on the discrete latent space representation.

[0088] In an exemplary embodiment of the present disclosure, the performing model processing on the super-resolution model in the training dataset based on the training situation to obtain a target super-resolution model includes: if the training situation is untrained, performing model training on the super-resolution model based on the training dataset to obtain the target super-resolution model; if the training situation is trained, performing model fine-tuning on the super-resolution model based on the training dataset to obtain the target super-resolution model.

[0089] In an exemplary embodiment of the present disclosure, fine-tuning the super-resolution model based on the training dataset to obtain the target super-resolution model includes: fixing the model parameters of a first part of the super-resolution model, and updating the model parameters of a second part of the super-resolution model based on the training dataset to determine the target super-resolution model.

[0090] In an exemplary embodiment of the present disclosure, super-resolution processing is performed on a low-resolution image based on the target super-resolution model to generate a high-resolution image, including: extracting features from the low-resolution image based on the target super-resolution model to obtain image features; and upsampling the image features to reconstruct the low-resolution image into a target high-resolution image.

[0091] It should be noted that the specific details of each module in the above-mentioned image processing device have been described in detail in the corresponding image processing method, and therefore will not be repeated here.

[0092] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0093] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0094] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0095] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0096] Refer to the following Figure 6 hereinafter, an electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 6 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0098] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in the "Exemplary Method" section of the present disclosure according to various exemplary embodiments. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0099] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0100] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0101] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0102] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0103] It should be noted that some embodiments of the present disclosure further provide a computer program product, which includes a computer program, and the computer program implements the above method when executed by a processor.

[0104] In one embodiment, a computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, a computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), and the like. In one embodiment, a computer program product may be an intangible product containing a computer program. Exemplarily, a computer program product may be implemented as a virtual digital product, such as a digital file such as an executable file or installation package storing a computer program.

[0105] The code of the computer program can be written in one or more programming languages. Programming languages include C, Java, C++, etc. The program code can be executed entirely on the user computing device, partially on the user computing device, or as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device via any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., via an Internet connection provided by a carrier).

[0106] Computer programs can be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. Electronic devices can convert signals carrying computer programs into digital signals to run the computer programs. When the computer program is run on an electronic device, its code causes the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present disclosure.

[0107] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0108] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0109] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0110] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Performing degradation processing on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and forming a training data set based on the plurality of sample high-resolution images and the plurality of sample low-resolution images; Obtaining a super-resolution model and determining a training status of the super-resolution model, and performing model processing on the super-resolution model in the training dataset based on the training status to obtain a target super-resolution model; Performing image super-resolution processing on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

2. The image processing method according to claim 1, wherein: The method of performing degradation processing on a plurality of sample high-resolution images based on the under-trained image reconstruction model to generate a plurality of sample low-resolution images includes: downsampling the multiple high-resolution images to obtain multiple intermediate images; The multiple intermediate images are fitted using at least one under-trained image reconstruction model to determine multiple low-resolution images; wherein the multiple low-resolution images have different degrees of degradation.

3. The image processing method according to claim 2, wherein: The step of fitting the plurality of intermediate images by using at least one under-trained image reconstruction model to determine a plurality of low-resolution images comprises: Each intermediate image is encoded by an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and the encoding result is decoded by a decoder to determine the low-resolution image.

4. The image processing method according to claim 3, wherein: The step of encoding each intermediate image by an encoder in at least one under-trained image reconstruction model to obtain an encoding result, and decoding the encoding result by a decoder to determine the low-resolution image includes: The feature vector of the intermediate image is extracted through multi-layer convolution operations, and the extracted feature vector is feature mapped to obtain a discrete latent space representation; The low-resolution image is generated by performing layer-by-layer downsampling and convolution operations on the discrete latent space representation.

5. The image processing method according to claim 1, wherein: The performing model processing on the super-resolution model in the training data set based on the training situation to obtain a target super-resolution model includes: If the training state is untrained, performing model training on the super-resolution model based on the training dataset to obtain the target super-resolution model; If the training status is already trained, the super-resolution model is fine-tuned based on the training data set to obtain the target super-resolution model.

6. The image processing method according to claim 5, characterized in that Fine-tuning the super-resolution model based on the training dataset to obtain the target super-resolution model includes: The model parameters of the first part of the super-resolution model are fixed, and the model parameters of the second part of the super-resolution model are updated based on the training dataset to determine the target super-resolution model.

7. The image processing method according to claim 1, wherein: The super-resolution processing of the low-resolution image based on the target super-resolution model to generate a high-resolution image includes: Performing feature extraction on the low-resolution image based on the target super-resolution model to obtain image features; The image features are up-sampled to reconstruct the low-resolution image into a target high-resolution image.

8. An image processing device, characterized in that: include: a training data generation module, configured to perform degradation processing on a plurality of sample high-resolution images based on an under-trained image reconstruction model to generate a plurality of sample low-resolution images, and to form a training data set based on the plurality of sample high-resolution images and the plurality of sample low-resolution images; a model processing module, configured to obtain a super-resolution model and determine a training status of the super-resolution model, and perform model processing on the super-resolution model in the training dataset based on the training status to obtain a target super-resolution model; The image reconstruction module is used to perform image super-resolution processing on the low-resolution image based on the target super-resolution model to generate a high-resolution image.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1 to 7 by executing the executable instructions.

Images