A lightweight super-resolution reconstruction model and system for compressing images

Abstract

The present application provides a kind of lightweight super-resolution reconstruction model and system for compressing image, and the lightweight super-resolution reconstruction model includes: the feature representation of HR image is trained with C-LR image in compression artifact removal submodule;Wherein, the feature representation of HR image is extracted from HR image using VGG pre-training model;Image super-resolution submodule recovers the LR image that compression artifact removal submodule is output as input, and carries out image super-resolution by residual information distillation network.The present application constructs lightweight super-resolution reconstruction network, designs objective function and loss function, adds feature representation to improve the details of reconstructed image, improves the quality of reconstructed image, and finally high-definition high-resolution image.

Description

Technical Field

[0001] This invention relates to super-resolution reconstruction technology, and more specifically, to a lightweight super-resolution reconstruction model and system for compressed images. Background Technology

[0002] With the continuous development of information technology, human demand for image information is gradually increasing, making image dissemination and storage extremely important. Super-resolution reconstruction technology can enhance the resolution of acquired degraded images and videos, offering advantages such as low cost and high practicality, thus possessing significant theoretical and practical value. However, in practical applications, due to limitations in storage capacity and transmission bandwidth, images and videos are generally downsampled and compressed to reduce data volume, as seen in mobile communications and the internet. However, downsampling results in information loss in high-resolution (HR) images, and subsequent compression can lead to undesirable visual effects, such as common compression artifacts and blockiness. These adverse effects not only weaken the quality of visual perception but also increase the difficulty of computer vision applications. For example, edge detection, face recognition, object segmentation, and ultra-high definition all require high-quality images, necessitating methods to reconstruct high-resolution, clear images from given low-resolution (LR) compressed images.

[0003] Current super-resolution reconstruction algorithms are designed for uncompressed images, meaning the low-resolution images they process only suffer from downsampling or blurring degradation. However, the compression effect introduced during the quantization stage of compressed images is strongly correlated with the image's inherent information, especially at high compression ratios. This means that conventional super-resolution reconstruction algorithms for uncompressed images cannot effectively handle compressed images. Therefore, directly using traditional super-resolution methods to reconstruct compressed images from real life rarely yields satisfactory results.

[0004] A direct solution to the compressed image super-resolution task is to perform artifact removal and super-resolution processing separately on the compressed low-resolution image. Although this method achieves good results by cascading existing artifact removal and super-resolution algorithms, the error accumulation in the artifact removal stage will be amplified in the super-resolution stage because the cascaded network cannot be trained end-to-end, and the correlation between the two tasks cannot be fully utilized.

[0005] Currently, the main algorithms specifically designed for super-resolution reconstruction of compressed images include:

[0006] 1. Kang et al. proposed the LJSRDB algorithm, which utilizes the ideas of learning-based sparse representation and morphological component analysis to perform compression artifact removal and super-resolution reconstruction operations sequentially. Xiong et al. combined PED regularization with a learning-based block matching method, which effectively removed block artifacts and improved the quality of the reconstructed image.

[0007] 2. Lee et al. used a dual-learning-based method to solve the problem of super-resolution of compressed images. By adding a denoising training set, they extended the existing denoising algorithm to dual learning.

[0008] 3. Zhao et al. proposed performing three operations on the compressed image: noise removal, classification, and mapping-based reconstruction.

[0009] 4. Sun et al. proposed a JPEG compressed image super-resolution network based on a joint deep network, which consists of a deblocking sub-network and a super-resolution sub-network cascaded together. Feature fusion, residual image fusion, and skip connections are used to compensate for the loss of high-frequency information.

[0010] 5. Chen et al. considered the removal of compression artifacts and the super-resolution problem as two related sub-problems and proposed the CISRDCNN algorithm based on CNN. This algorithm consists of a deblocking module, an upsampling module, and a quality enhancement module. To preserve the functionality of each module and the correlation between the two sub-problems, the algorithm first trains each module end-to-end separately, and finally trains the entire network through joint optimization.

[0011] Currently, most algorithms treat the removal of compressed shadows and upsampling as two separate stages, which can easily lead to overly smooth reconstructed images. At the same time, most existing algorithms still suffer from drawbacks such as high training time and high computational complexity.

[0012] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0013] The purpose of this invention is to address the shortcomings of existing technologies by providing a lightweight super-resolution reconstruction model and system for compressed images.

[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0015] The first aspect of this invention provides a lightweight super-resolution reconstruction model for compressed images, comprising:

[0016] The feature representations of the HR images are trained together with the C-LR images in the compression artifact removal submodule; wherein, the feature representations of the HR images are the feature representations extracted from the HR images using the VGG pre-trained model;

[0017] The image super-resolution submodule takes the LR image restored by the compression artifact removal submodule as input and performs image super-resolution reconstruction through the residual information distillation network to obtain the SR image.

[0018] A second aspect of the present invention provides a system for lightweight super-resolution reconstruction of compressed images, the system comprising:

[0019] Memory; and

[0020] A processor coupled to the memory is configured to perform super-resolution reconstruction using the lightweight super-resolution reconstruction model for compressed images, based on instructions stored in the memory, when performing super-resolution reconstruction of images or videos.

[0021] A third aspect of the present invention provides an end-to-end super-resolution image transmission system, wherein an image or video to be transmitted is compressed at the transmitting end and then transmitted to the receiving end;

[0022] The receiving end is configured as the lightweight super-resolution reconstruction system for compressed images, and reconstructs the received images or videos.

[0023] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0024] 1) This invention employs deep learning methods to perform super-resolution reconstruction of compressed images, resulting in high-resolution images;

[0025] 2) This invention achieves the design of a lightweight super-resolution network by introducing a residual information distillation network (RIDN).

[0026] 3) This invention improves the detail of the restored image by adding feature representations of high-resolution images;

[0027] 4) This invention introduces a comprehensive loss function: the L1 loss function of the compression artifact removal submodule and the L1 loss function of the image super-resolution submodule;

[0028] 5) This invention can be applied to mobile terminals to reconstruct compressed images and obtain clear, high-resolution images. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the design of the model of the present invention.

[0030] Figure 2 This is a structural block diagram of the compression artifact removal submodule of the present invention.

[0031] Figure 3This is a structural block diagram of the super-resolution submodule structure of the present invention.

[0032] Figure 4 This is a structural block diagram of the residual information distillation block RIDB of the present invention.

[0033] Figure 5 This is a structural block diagram of the residual block RB of the present invention.

[0034] Figure 6 The results of comparing the model of this invention with existing super-resolution models. Detailed Implementation

[0035] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment uses three data types: C-LR image (compressed LR image), LR (low resolution) image, and HR (high resolution) image. Assuming that the lost information in the compressed image can be jointly modeled using JPEG artifacts in the feature domain and high-frequency information from the clear HR and LR images, the model's final learning objective is set to minimize the residual between the ground truth high-resolution image and the upsampling result of the compressed image. This provides a lightweight super-resolution reconstruction model for compressed images, including:

[0038] The feature representations of the HR images are trained together with the C-LR images in the compression artifact removal submodule; wherein, the feature representations of the HR images are the feature representations extracted from the HR images using the VGG pre-trained model;

[0039] The image super-resolution submodule takes the LR image restored by the compression artifact removal submodule as input and performs image super-resolution reconstruction through the residual information distillation network to obtain the SR image.

[0040] Compression artifact removal submodule

[0041] The goal is to eliminate artifacts and blockiness caused by compression operations, such as Figure 2 As shown, the module includes a feature extraction convolutional layer, multiple residual information distillation blocks (RIDBs), a feature fusion block, and a reconstruction block connected in sequence. The output of each residual information distillation block (RIDB) is also used as the input of the feature fusion block. The module first performs initial feature extraction using a 3*3 convolutional kernel to generate coarse features from the input C-LR image. Then, multiple residual information distillation blocks (RIDBs) are stacked in a chain to gradually refine the extracted features.

[0042] Each first residual information distillation block, such as Figure 4As shown, the process includes multiple distillation steps. Each distillation step splits the input features into two paths through a channel splitting operation. The left path uses a 1×1 convolutional layer to preserve features, while the right path is fed into the residual block RB for feature distillation, effectively reducing the number of parameters. The split channel features from the right path are then fed into the residual block RB for finer-grained residual learning. Specifically, for the residual block RB, as shown... Figure 5 As shown, it consists of two 3×3 convolutions and an activation module. The RB can benefit from residual learning without introducing any additional parameters, and it is located in the body of RIDB to better refine features. The information distillation result of the previous distillation step is used as the input of the next distillation step, and the last distillation step only preserves information. The information-preserved results of each distillation step are aggregated and assembled by a 1×1 convolutional layer, and then a 3×3 convolutional layer is used to smooth the aggregated result as follows.

[0043] Image super-resolution submodule

[0044] like Figure 3 As shown, the image super-resolution submodule uses essentially the same network configuration as the compression artifact removal submodule, except for the final sub-pixel layer. Specifically, the compression artifact removal submodule is trained using the predicted sharp low-resolution image as input, primarily to predict the residual image between the LR image and the HR ground truth image. The process of upscaling the image from low to high resolution using the sub-pixel layer implicitly incorporates the interpolation process within the preceding convolution, improving model performance through automatic feature map learning. Unlike the deconvolution operation used for upsampling, the sub-pixel layer only changes the image size in the penultimate layer. Because the preceding convolution operations are performed on the low-resolution image, the time is shortened, and the final model outputs a high-resolution super-resolution image.

[0045] When training the model, the objective function of the model is designed as follows:

[0046]

[0047] in, Represents high-resolution images, This represents an LR image compressed from JPEG. Represents a sharp LR image output from the compression artifact removal submodule; matrix This indicates a degradation operation. and These represent the bicubic downsampling operation and the JPEG image compression operation, respectively.

[0048] When training the model, a loss function needs to be designed. Since the overall framework of the model includes two typical tasks—removing compression artifacts and image super-resolution—two loss functions are set. Let... For the training set, It is the i-th input C-LR image. It is the corresponding uncompressed LR image. It is the corresponding high-resolution ground truth (HR) image;

[0049] The compression artifact removal submodule's task phase involves restoring the compressed low-resolution image to its uncompressed state. Therefore, the corresponding set of images for this phase is ( , Since the corresponding pixels in the C-LR image and the LR image correspond, the goal of this stage is to recover the accurate pixel values. Therefore, the loss function L1 for this stage is designed as follows:

[0050]

[0051] Where W is the width of the image and H is the height of the image;

[0052] In the image super-resolution task stage of the image super-resolution submodule, based on the C-LR image processed in the previous task stage, a super-resolution operation is used to further scale the intermediate product to the size of the HR image, thus changing the image resolution. To obtain more accurate SR image generation, the loss function L2 in this stage is designed as follows:

[0053]

[0054] To balance the weights of these two losses, the L1 loss and L2 loss are combined with equal weight to obtain the model's total loss, Ltotal.

[0055] .

[0056] The specific training process of the model in this embodiment is as follows:

[0057] Step 1: Download the dataset and preprocess the dataset;

[0058] First, download the public dataset. We selected the DIV2K dataset, which contains 1000 high-quality RGB images. 800 images are used for training, 100 for validation, and the remaining 100 for testing. To obtain compressed low-resolution images, the input images are preprocessed: first, the high-resolution image is downsampled using bicubic interpolation to obtain an uncompressed low-resolution image; then, the downsampled image is compressed using the JPEG encoder in MATLAB to obtain a JPEG-compressed low-resolution image, which is then input into the model. This embodiment uses the standard JPEG compression method, with a compression quality factor (QF) of 20 and a magnification factor of 2.

[0059] Step 2: Use the VGG16 pre-trained model to extract the feature representation of the HR image for subsequent training.

[0060] Step 3: Construct a lightweight super-resolution reconstruction model network for image compression, design the objective function and loss function to eliminate compression artifacts or add more image details, and obtain high-resolution images.

[0061] Step 4: During the network training phase, the C-LR image, along with the feature representation extracted from the HR image, is used as the network input. In the compression artifact removal submodule, the LR image is used as the label image, and the submodule minimizes... The loss function continuously learns the mapping between C-LR images and LR images, enabling this module to generate clear LR images. The image super-resolution submodule takes the LR image generated by the previous submodule as input and performs image super-resolution reconstruction through a residual information distillation network. This module uses the HR image as the label image and minimizes... The loss function generates high-resolution SR images. By jointly optimizing the overall loss function, the entire network is trained to obtain the final model.

[0062] Step 5: During the training phase, the image size in the dataset is set to 256×256, and images with a compression quality of 10 are used for both training and testing. The experimental batch size is set to 64. For the first 100 epochs, the learning rate is 0.0002, and in the subsequent 100 epochs, the learning rate linearly decreases to 0 until the final model is obtained.

[0063] Step 6: During the testing phase, input the test data into the trained model to obtain a clear HR image after the JPEG compressed image is reconstructed.

[0064] Comparative experiment

[0065] To verify the performance of the model in this embodiment, this experiment compares the model with five other methods: Bicubic, ARCNN, VDSR, RCAN, and IMDN. This comparative experiment was conducted on two datasets, Set5 and Set14, containing 5 and 14 test samples respectively. Table 1 shows the experimental results of each model when the scaling factor is 2, using PSNR and SSIM as metrics to measure network performance.

[0066] Dataset Scale Set5 Set14 Bicubic ×2 27.434 / 0.806 26.037 / 0.685 ARCNN ×2 27.678 / 0.811 26.121 / 0.688 VDSR ×2 28.438 / 0.807 26.514 / 0.710 RCAN ×2 29.929 / 0.846 26.784 / 0.708 IMDN ×2 30.841 / 0.867 27.202 / 0.763 Ours ×2 30.953 / 0.872 27.269 / 0.774

[0067] Table 1. Comparison of each model on the Set5 and Set14 test sets.

[0068] In addition, two images were selected for this experiment. Figure 6 The figures show the visual comparison results of each model on these two images when the magnification is 2 and the compression quality factor (QF) is 20. The results show that the model in this embodiment performs better than Bicubic overall. Bicubic and ARCNN exhibit significant block artifacts, while VDSR and RCAN suffer from overly smoothed edges and loss of many image details. Compared to these models, the model in this embodiment demonstrates superior reconstruction performance.

[0069] Example 2

[0070] The difference between this embodiment and Embodiment 1 is that it provides a lightweight super-resolution reconstruction system for compressed images, the system comprising:

[0071] Memory; and

[0072] A processor coupled to the memory is configured to perform super-resolution reconstruction using the lightweight super-resolution reconstruction model for compressed images as described in Example 1, based on instructions stored in the memory, when performing super-resolution reconstruction of images or videos.

[0073] Memory may include, for example, system memory, fixed non-volatile storage media, etc. System memory may store, for example, the operating system, application programs, the boot loader, and other programs.

[0074] The system may also include input / output interfaces, network interfaces, and storage interfaces. These interfaces, as well as the memory and processor, can be connected via a bus, for example. The input / output interfaces provide connection interfaces for input / output devices such as monitors, mice, keyboards, and touchscreens. The network interfaces provide connection interfaces for various networked devices. The storage interfaces provide connection interfaces for external storage devices such as SD cards and USB flash drives.

[0075] Example 3

[0076] The difference between this embodiment and embodiment 2 is that: an end-to-end super-resolution image transmission system is provided, in which the image or video to be transmitted is compressed at the sending end and then transmitted to the receiving end;

[0077] The receiving end is configured as the lightweight super-resolution reconstruction system for compressed images described in Example 2, and reconstructs the received image or video.

[0078] In practical applications, the receiving end is typically a mobile terminal such as a smartphone or tablet. Due to limitations in network bandwidth and storage space, network transmission often uses low-resolution, lossy compressed images, resulting in low image quality received at the receiving end. Under low bandwidth conditions, high-compression-ratio compressed images are transmitted faster and occupy less storage space. Using the image transmission system of this embodiment, the received image can be reconstructed to obtain a high-resolution image, saving network bandwidth and storage space.

[0079] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-non-transitory readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0080] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A lightweight super-resolution reconstruction model for compressed images, characterized in that, include: The feature representations of the HR images are trained together with the C-LR images in the compression artifact removal submodule; wherein, the feature representations of the HR images are the feature representations extracted from the HR images using the VGG pre-trained model; The compression artifact removal submodule includes a first feature extraction convolutional layer, multiple first residual information distillation blocks, a first feature fusion block, and a first reconstruction block connected in sequence. The output of each first residual information distillation block is also used as the input of the first feature fusion block. Each first residual information distillation block includes multiple distillation steps. Each distillation step divides the input into two paths: one path uses a 1×1 convolutional layer to retain information, and the other path is fed into the first residual block for information distillation. The information distillation result of the previous distillation step is used as the input of the next distillation step, and the last distillation step only retains information. The information retention results of each distillation step are aggregated and assembled by a 1×1 convolutional layer, and then a 3×3 convolutional layer is used to smooth the aggregated and assembled result. The image super-resolution submodule takes the LR image restored by the compression artifact removal submodule as input and performs image super-resolution reconstruction through the residual information distillation network to obtain the SR image; When training the model, the objective function of the model is designed as follows: in, Represents high-resolution images, This represents an LR image compressed from JPEG. Represents a sharp LR image output from the compression artifact removal submodule; matrix This indicates a degradation operation. and These represent the bicubic downsampling operation and the JPEG image compression operation, respectively. When training a model, design the model's loss function, where is set For the training set, It is the i-th input C-LR image. It is the corresponding uncompressed LR image. It is the corresponding high-resolution ground truth (HR) image; The loss function L1 for the training compression artifact removal submodule is designed as follows: Where W is the width of the image and H is the height of the image; The loss function L2 for the super-resolution sub-module stage of training images is designed as follows: The L1 loss and L2 loss are combined with equal weight to obtain the model's total loss, Ltotal. 。 2. The lightweight super-resolution reconstruction model for compressed images according to claim 1, characterized in that: The image super-resolution submodule includes a second feature extraction convolutional layer, multiple second residual information distillation blocks, a second feature fusion block, a second reconstruction block, and a sub-pixel layer connected in sequence. The output of each second residual information distillation block is also used as the input of the second feature fusion block. Each second residual information distillation block includes multiple distillation steps. Each distillation step divides the input into two paths: one path uses a 1×1 convolutional layer to retain information, and the other path is fed into the second residual block for information distillation. The information distillation result of the previous distillation step is used as the input of the next distillation step, and the last distillation step only retains information. The information retention results of each distillation step are aggregated and assembled by a 1×1 convolutional layer, and then a 3×3 convolutional layer is used to smooth the aggregated and assembled result.

3. A system for lightweight super-resolution reconstruction of compressed images, characterized in that, The system includes: Memory; and A processor coupled to the memory is configured to perform super-resolution reconstruction using the lightweight super-resolution reconstruction model for compressed images as described in any one of claims 1-2, based on instructions stored in the memory.

4. An end-to-end super-resolution image transmission system, characterized in that: The image or video to be sent is compressed at the sending end and then transmitted to the receiving end; The receiving end is configured as the lightweight super-resolution reconstruction system for compressed images as described in claim 3, and reconstructs the received image.

5. The end-to-end super-resolution image transmission system according to claim 4, characterized in that: The receiving end is a mobile terminal.

Images