An image super-resolution reconstruction method, system, device and storage medium
Patent Information
- Application Number
- CN202311253547.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2043-09-26
AI Technical Summary
[0003]本发明要解决的技术问题是现有干净数据集上训练的模型通常在实际测试数据上表现不佳的问题
[0025]本发明有效地利用了预训练超分辨率模型的先验知识，根据随机给定的退化模型参数，如模糊核和高斯白噪声等，帮助预训练超分辨率模型更好的挖掘测试图片本身的退化特征信息，从而使用自监督的方式将预训练的超分辨率模型快速适应于真实的测试数据，最终重构出更清晰的高分辨率图片。
Smart Images
Figure CN117372256B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image reconstruction and image enhancement, and in particular to a method, system, apparatus and storage medium for super-resolution reconstruction of images. Background Technology
[0002] Deep learning is widely used in image reconstruction, including tasks such as super-resolution, denoising, dehazing, and low-light enhancement, demonstrating significantly better performance than traditional vision techniques. However, while deep learning algorithms benefit from large-scale training data, they are also easily limited by the distribution of the training dataset, leading to performance limitations on test data with a distribution inconsistent with the training data. This is particularly true in super-resolution image reconstruction. Existing deep learning-based super-resolution models are typically trained on clean datasets (manufactured by fixed downsampling of high-resolution images). However, real-world test images often contain degradation types such as blur, noise, and compression artifacts, with distributions significantly different from the training dataset. Therefore, models trained on clean datasets usually perform poorly on real-world test data. Furthermore, constructing actual training data and training super-resolution models is time-consuming and labor-intensive, requiring substantial human, computational, and time costs. Summary of the Invention
[0003] The technical problem that this invention aims to solve is that models trained on existing clean datasets often perform poorly on real test data.
[0004] This invention proposes a method, system, apparatus, and storage medium for super-resolution reconstruction of images.
[0005] A super-resolution reconstruction method for images includes the following steps:
[0006] Step 1: Train a super-resolution model using clean data as the initial super-resolution model;
[0007] Step 2: Use clean data to perform degradation to obtain degraded images and corresponding degradation category labels. Input the degraded images and degradation category labels into the classifier, train the classifier by calculating the loss between the classifier and the real labels, and input the test images into the trained classifier to construct second-order degraded images.
[0008] Step 3: Input the second-order degraded image and the test image into the initial super-resolution model. Output the feature map of the second-order degraded image as the reconstructed image through the self-supervised reconstruction branch. Calculate the degradation loss between the reconstructed image and the test image to update the initial super-resolution model.
[0009] Step 4: Generate images using the updated super-resolution model.
[0010] Preferably, the method of using clean data for degradation is as follows: randomly generating different Gaussian blur kernels and convolving them on the clean image to obtain a blurred image, or randomly adding different Gaussian noise images to the clean image to obtain a noisy image, or using the JPEG compression algorithm to compress the clean image with a random compression ratio to obtain a compressed image.
[0011] Preferably, the method for constructing the second-order degenerate image is as follows: the classification probability of the test image is input into the classifier, and the image is constructed according to the following formula: Where c'1, c'2, c'3, and c'4 represent the probabilities that the test image is a clean image, a blurry image, a noisy image, and a compressed image, respectively. Indicates the original input image, I B ,I N Ic and Ic represent second-order degraded images processed using three degradation types: blurring, noise, and compression, respectively. D The constructed second-order degenerate image.
[0012] Preferably, the self-supervised reconstruction branch includes a PixelUnshuffle layer, a ResBlock layer, and a 3x3 convolutional layer. The PixelUnshuffle layer is used to downsample the feature map, and the ResBlock layer and the 3x3 convolutional layer are used to reconstruct the downsampled feature map into a three-channel image, which is the reconstructed image.
[0013] Preferably, the function for calculating the degradation loss between the reconstructed image and the test image is: ,in, Let I'o represent the L1 regularization function, I'o represent the reconstructed image, and Io represent the test image.
[0014] Preferably, the degradation category label is a one-hot label. When the degradation image contains only blurry degradation, the one-hot label of the degradation image indicates that the blurry category is 1, and all other categories are 0.
[0015] Preferably, the classification probability c' of the test image is [c'1c'2c'3c'4].
[0016] A super-resolution reconstruction system for images, comprising:
[0017] The acquisition module is used to acquire low-resolution test images as actual test inputs;
[0018] The super-resolution model inference module is used to extract features from the input image and output the final reconstructed high-resolution image.
[0019] A degradation type classifier is used to determine the degradation type contained in the input test image;
[0020] The degraded image construction module performs secondary degradation on the test image by predicting the degradation type. For example, it uses a randomly generated blur kernel to blur the test image or adds a randomly generated noise map to obtain a second-order degraded image.
[0021] The self-supervised degradation update module is used to reconstruct the original test image from the feature map of the second-order degradation image, and backpropagates the gradient to the super-resolution model through the self-supervised degradation loss to update the model parameters so that it can automatically adapt to the new test data distribution.
[0022] An image super-resolution reconstruction apparatus includes a processor and a memory, wherein the memory is used to store a program, and the processor is used to execute the program to implement the image super-resolution reconstruction method.
[0023] A super-resolution reconstruction storage medium for an image, and an apparatus for storing the super-resolution reconstruction method of the image.
[0024] Compared with the prior art, the present invention has the following advantages and effects:
[0025] This invention effectively utilizes the prior knowledge of the pre-trained super-resolution model. Based on randomly given degradation model parameters, such as blur kernel and Gaussian white noise, it helps the pre-trained super-resolution model to better mine the degradation feature information of the test image itself. Thus, it uses a self-supervised approach to quickly adapt the pre-trained super-resolution model to real test data, and finally reconstructs a clearer high-resolution image. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a schematic diagram of training a classifier in an embodiment of the present invention;
[0028] Figure 2 This is a schematic diagram of reconstructing an image using self-supervised degenerate branches in an embodiment of the present invention;
[0029] Figure 3 This is a schematic diagram of the image generated by the updated super-resolution model in an embodiment of the present invention. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0031] Example 1:
[0032] A super-resolution reconstruction method for images includes the following steps:
[0033] Step 1: Train a super-resolution model using clean data as the initial super-resolution model;
[0034] Step 2: Use clean data to perform degradation to obtain degraded images and corresponding degradation category labels. Input the degraded images and degradation category labels into the classifier, train the classifier by calculating the loss between the classifier and the real labels, and input the test images into the trained classifier to construct second-order degraded images.
[0035] Step 3: Input the second-order degraded image and the test image into the initial super-resolution model. Output the feature map of the second-order degraded image as the reconstructed image through the self-supervised reconstruction branch. Calculate the degradation loss between the reconstructed image and the test image to update the initial super-resolution model.
[0036] Step 4: Generate images using the updated super-resolution model.
[0037] In a specific example, in step 1, preferably, a super-resolution model is trained using clean data to obtain an initial super-resolution model, which is then deployed to the system.
[0038] During the actual system operation phase, the parameters of the pre-trained super-resolution model are adaptively adjusted online. Therefore, in the first step of this embodiment, a publicly available clean training dataset, such as the DIV2K dataset, is used to train a super-resolution model, which is then deployed to the system as the initial model. At this point, the system can perform super-resolution reconstruction on clean test images and obtain clear, high-resolution images. It is important to note that at this stage, when the input test images contain degradations such as blurriness, noise, and compression artifacts, and their data distribution is inconsistent with the training data distribution, the super-resolution model pre-trained using clean data cannot reconstruct a clear, high-resolution image. The high-resolution image reconstructed directly using this model usually has problems such as excessive blurriness and noise, making it difficult to meet the needs of actual users.
[0039] In one specific implementation, in step 2, a classifier is trained and used to predict the degradation type in the actual image, thereby constructing the corresponding second-order degradation image.
[0040] To adaptively update model parameters based on test images, this embodiment requires a classifier to determine the degradation type contained in the actual test images, thereby providing support for constructing degradation images. The process of constructing degradation images and training the classifier is as follows: Figure 1 As shown. To train the classifier, this example uses the DIV2K training dataset. Clean images within the dataset undergo random degradation. This involves randomly generating different Gaussian blur kernels and convolving them with the clean images to obtain blurred images, or randomly adding different Gaussian noise maps to the clean images to obtain noisy images, or using the JPEG compression algorithm to compress the clean images at random compression ratios to obtain compressed images. After randomly blurring the clean images in the training data, adding noise and performing image compression at arbitrary ratios, degradation images and corresponding one-hot degradation type labels are obtained. When a degradation image contains only blurry degradation, the one-hot label of the degradation image indicates a blurry category as 1, and all other categories as 0.
[0041] After constructing the corresponding degradation classification dataset, a deep learning-based classifier, such as ResNet18, is used. The input is a degradation image, and the classifier is fed the predicted probability of different degradation types in the degradation image. The Cross-Entropy classification loss is calculated between the predicted probability and the true one-hot label, and then the classifier parameters are updated according to the backpropagation algorithm.
[0042] After the degradation classifier is trained, the test image is input into the degradation classifier. The classifier receives the final degradation classification probability c' = [c'1c'2c'3c'4], where c'1, c'2, c'3, and c'4 represent the probabilities that the test image is a clean image, a blurry image, a noisy image, and a compressed image, respectively. Let Io represent the original input image. B ,I N Let Ic and Ic represent second-order degraded images processed using three degradation types: blurring, noise, and compression, respectively. The degraded image is constructed using the following formula: , among which, I D The final constructed second-order degenerate model, along with the original input... Pairs are used to form a training sample and are then used in the subsequent adaptive update process of the super-resolution model.
[0043] In one specific implementation, in step 3, the parameters of the pre-trained super-resolution model are updated using self-supervised degradation recovery loss based on the second-order degraded image and the original test image.
[0044] During the online operation of the super-resolution system, for test images Io arriving at different times, this embodiment uses a pre-trained degradation model classifier to construct the corresponding second-order degradation image Io based on step 2. D, obtain {Io, I D} data pairs, then I D The input is fed into a pre-trained super-resolution network. Through a self-supervised reconstruction branch, it outputs a reconstructed image I'o based on the feature map of the second-order degraded image. The self-supervised degradation loss function between I'o and Io is calculated, and the super-resolution model parameters are updated using backpropagation. The formula for calculating the self-supervised degradation loss function is: ,in, This represents the L1 regularization function. The specific self-supervised reconstruction process is as follows: Figure 3 As shown, the self-supervised reconstruction branch is located after the upsampling layer and before the reconstruction convolutional layer in the super-resolution network. This branch contains a PixelUnshuffle layer to downsample the feature map, and uses a ResBlock and a 3x3 convolutional layer to reconstruct the downsampled feature map into a three-channel image I'o, which has the same image resolution as the input image Io.
[0045] In one specific implementation, in step 4, a test image is input, and the updated super-resolution model is used for inference to obtain the final clear, high-resolution image. Let θ represent the super-resolution model trained with clean data, and θ' represent the super-resolution model after the adaptive update step with the test image. Finally, the test image is input into the updated super-resolution model θ', and the final high-resolution image I is generated. HR The final model inference process is as follows: Figure 3 As shown.
[0046] In one specific implementation, step 1 involves adaptively adjusting the parameters of the pre-trained super-resolution model online during the actual system operation phase. Therefore, in this embodiment, a publicly available clean training dataset, such as the DIV2K dataset, is used in the first step to train a super-resolution model, which is then deployed to the system as the initial model. At this point, the system can perform super-resolution reconstruction on clean test images and obtain clear, high-resolution images.
[0047] Step 2: To train the classifier, this example uses the DIV2K training dataset. Clean images within the dataset undergo random degradation. This involves randomly generating different Gaussian blur kernels and convolving them with the clean images to obtain blurred images, or randomly adding different Gaussian noise maps to the clean images to obtain noisy images, or using the JPEG compression algorithm to compress the clean images at random compression ratios to obtain compressed images. After randomly blurring the clean images in the training data, adding noise and performing arbitrary image compression and other degradation operations, degraded images and their corresponding one-hot degradation type labels are obtained. When a degraded image contains only blurred degradation, the one-hot label of the degraded image indicates a blurred category as 1, and all other categories as 0. After constructing the corresponding degradation classification dataset, a deep learning-based classifier, such as ResNet18, is used. The input is a degraded image, and the classifier is fed the predicted probabilities of different degradation types in the degraded image. The Cross-Entropy classification loss is calculated between the predicted probabilities and the true one-hot labels, and then the classifier parameters are updated using the backpropagation algorithm. After the degradation classifier is trained, the test image is input into the degradation classifier. The classifier receives the final degradation classification probability c' = [c'1c'2c'3c'4], where c'1, c'2, c'3, and c'4 represent the probabilities that the test image is a clean image, a blurry image, a noisy image, and a compressed image, respectively. Let Io represent the original input image. B ,I N Let Ic and Ic represent second-order degraded images processed using three degradation types: blurring, noise, and compression, respectively. The degraded image is constructed using the following formula: , among which, I D The final constructed second-order degenerate model is paired with the original input Io to form a training sample, which is then used in the subsequent adaptive update process of the super-resolution model.
[0048] Step 3: Using the pre-trained degradation model classifier, construct the corresponding second-order degradation image I in Step 2. D , obtain {Io, I D} data pairs, then I D The input is fed into a pre-trained super-resolution network. Through a self-supervised reconstruction branch, it outputs a reconstructed image I'o based on the feature map of the second-order degraded image. The self-supervised degradation loss function between I'o and Io is calculated, and the super-resolution model parameters are updated using backpropagation. The formula for calculating the self-supervised degradation loss function is: ,in, This represents the L1 regularization function. The specific self-supervised reconstruction process is as follows: Figure 3As shown, the self-supervised reconstruction branch is located after the upsampling layer and before the reconstruction convolutional layer in the super-resolution network. This branch contains a PixelUnshuffle layer to downsample the feature map, and uses a ResBlock and a 3x3 convolutional layer to reconstruct the downsampled feature map into a three-channel image I'o, which has the same image resolution as the input image Io.
[0049] Step 4: Let θ represent the super-resolution model trained with clean data, and θ' represent the super-resolution model after the adaptive update step with the test image. Finally, input the test image into the updated super-resolution model θ' to generate the final high-resolution image I. HR The final model inference process is as follows: Figure 3 As shown.
[0050] A super-resolution reconstruction system for images, comprising:
[0051] The acquisition module is used to acquire low-resolution test images as actual test inputs;
[0052] The super-resolution model inference module is used to extract features from the input image and output the final reconstructed high-resolution image.
[0053] A degradation type classifier is used to determine the degradation type contained in the input test image;
[0054] The degraded image construction module performs secondary degradation on the test image by predicting the degradation type. For example, it uses a randomly generated blur kernel to blur the test image or adds a randomly generated noise map to obtain a second-order degraded image.
[0055] The self-supervised degradation update module is used to reconstruct the original test image from the feature map of the second-order degradation image, and backpropagates the gradient to the super-resolution model through the self-supervised degradation loss to update the model parameters so that it can automatically adapt to the new test data distribution.
[0056] An image super-resolution reconstruction apparatus includes: at least one processor; at least one memory for storing at least one program; when the program is executed by the processor, the processor implements the reconstruction method described above.
[0057] A super-resolution reconstruction storage medium for an image stores instructions or programs that can execute the reconstruction method described above.
[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for super-resolution reconstruction of images, characterized in that, Includes the following steps: Step 1: Train a super-resolution model using clean data as the initial super-resolution model; Step 2: Use clean data to perform degradation to obtain degraded images and corresponding degradation category labels. Input the degraded images and degradation category labels into the classifier, train the classifier by calculating the loss between the classifier and the real labels, and input the test images into the trained classifier to construct second-order degraded images. The method of using clean data for degradation is as follows: randomly generating different Gaussian blur kernels to convolve the clean image to obtain a blurred image, or randomly adding different Gaussian noise images to the clean image to obtain a noisy image, or using the JPEG compression algorithm to compress the clean image with a random compression ratio to obtain a compressed image; The method for constructing the second-order degenerate image is as follows: input the classification probability of the test image into the classifier, and construct it according to the following formula: Where c'1, c'2, c'3, and c'4 represent the probabilities that the test image is a clean image, a blurry image, a noisy image, and a compressed image, respectively. Indicates the original input image, I B ,I N Ic and Ic represent second-order degraded images processed using three degradation types: blurring, noise, and compression, respectively. D The constructed second-order degenerate image; Step 3: Input the second-order degraded image and the test image into the initial super-resolution model. Output the feature map of the second-order degraded image as the reconstructed image through the self-supervised reconstruction branch. Calculate the degradation loss between the reconstructed image and the test image to update the initial super-resolution model. The self-supervised reconstruction branch includes a PixelUnshuffle layer, a ResBlock layer, and a 3x3 convolutional layer. The PixelUnshuffle layer is used to downsample the feature map, and the ResBlock layer and the 3x3 convolutional layer are used to reconstruct the downsampled feature map into a three-channel image, which is the reconstructed image. The function for calculating the degradation loss between the reconstructed image and the test image is: ,in, Let I'o represent the L1 regularization function, I'o represent the reconstructed image, and Io represent the test image. Step 4: Generate images using the updated super-resolution model.
2. The image super-resolution reconstruction method according to claim 1, characterized in that, The degradation category label is a one-hot label. When the degradation image contains only blurry degradation, the one-hot label of the degradation image indicates that the blurry category is 1, and all other categories are 0.
3. The image super-resolution reconstruction method according to claim 1, characterized in that, The classification probability of the test image is c' = [c'1c'2c'3c'4].
4. A super-resolution reconstruction system for images, characterized in that, A method for super-resolution reconstruction of an image according to any one of claims 1-3, comprising: The acquisition module is used to acquire low-resolution test images as actual test inputs; The super-resolution model inference module is used to extract features from the input image and output the final reconstructed high-resolution image. A degradation type classifier is used to determine the degradation type contained in the input test image; The degraded image construction module performs secondary degradation on the test image by predicting the degradation type. For example, it uses a randomly generated blur kernel to blur the test image or adds a randomly generated noise map to obtain a second-order degraded image. The self-supervised degradation update module is used to reconstruct the original test image from the feature map of the second-order degradation image, and backpropagates the gradient to the super-resolution model through the self-supervised degradation loss to update the model parameters so that it can automatically adapt to the new test data distribution.
5. An image super-resolution reconstruction apparatus, characterized in that, It includes a processor and a memory, the memory being used to store a program, and the processor being used to execute the program to implement the super-resolution reconstruction method for images according to any one of claims 1-3.
6. A super-resolution reconstruction storage medium for images, characterized in that, A device for storing the super-resolution reconstruction method of the image according to any one of claims 1-3.