A sub-pixel sampling-based lightweight super-resolution network and application
A lightweight super-resolution network using subpixel sampling and attention mechanisms solves the problems of information loss and high computational cost in existing technologies, achieving high-quality image reconstruction and model optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2023-07-12
- Publication Date
- 2026-07-31
AI Technical Summary
Existing super-resolution networks suffer from information loss, high computational cost, and artifacts in image feature pooling and deconvolution operations, which affect image quality. At the same time, the model parameters and computational cost are relatively large.
By employing a subpixel sampling method and combining the Unet++ network structure, spatial attention, and channel attention mechanisms, feature mapping and fusion are performed through subpixel downsampling and upsampling, reducing computational load while preserving image information.
It improves image quality, reduces model parameters and computational cost, while preserving image details, resulting in faster training speeds and generating images that are closer to real high-resolution images.
Smart Images
Figure CN116862772B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a lightweight super-resolution network based on sub-pixel sampling. Background Technology
[0002] Super-resolution refers to the technique of converting low-resolution images into high-resolution images using algorithms. Super-resolution is commonly used in satellite imaging, biometrics, medical imaging, microscopic image processing, and multimedia and video enhancement.
[0003] Image super-resolution has evolved over several decades. Early methods included interpolation, such as bilinear and bicubic interpolation. These methods leveraged the relationships between neighboring pixels to increase image resolution through interpolation. While simple to implement, their resolution improvement was not ideal. Other methods are reconstruction-based, typically utilizing a low-resolution version of the image along with prior knowledge to reconstruct a high-resolution version. These methods are usually based on image restoration and reconstruction theory, utilizing pixel correlations, image texture, and edge features for reconstruction. Some commonly used methods include least-squares-based nearest neighbor methods and regularized projection inversion methods. Additionally, there are sparse representation-based methods. These methods obtain an initial set of estimated high-resolution images by sparsely representing the high-frequency components of the low-resolution image, and then iteratively optimize and solve for the high-resolution image.
[0004] Currently, convolutional neural networks (CNNs) are one of the most prevalent methods for super-resolution reconstruction. They learn the mapping between low-resolution and high-resolution images through various convolutional layers, pooling layers, and activation functions, thereby improving image resolution. The encoder-decoder architecture is the most commonly used feature extraction architecture in super-resolution networks. By training the encoder and decoder to learn image features, high-resolution reconstructed images can be generated from low-resolution input images, thus achieving super-resolution image reconstruction. Pooling layers and deconvolutions are often used for downsampling and upsampling during encoding and decoding. However, these two methods often suffer from the following problems:
[0005] (1) Pooling the image features will discard some information, resulting in the loss of image details, which may affect the performance of the model to some extent.
[0006] (2) Pooling layers reduce the size of feature maps, but they increase computation because pooling layers usually require a lot of convolution operations.
[0007] (3) Using deconvolution for upsampling, due to the overlapping and zero-padding of neighboring pixels, will cause obvious artifacts and checkerboard patterns in the image, affecting image quality and generating additional parameters.
[0008] Therefore, how to improve the quality of generated images and enhance the performance of super-resolution networks while reducing model parameters and computational load has become an important research direction. Summary of the Invention
[0009] To address the aforementioned problems, this invention proposes a novel image sampling method that performs scale transformation on feature maps through sub-pixel sampling. Inspired by camera imaging mechanisms, this invention leverages the sub-pixel principle to fully utilize complementary information between each image. Furthermore, this invention combines the sub-pixel convolution proposed by efficient sub-pixel convolutional neural networks, using this proposed sub-pixel convolution as a sub-pixel upsampling operation and its inverse process as a sub-pixel downsampling operation. This invention designs a lightweight super-resolution network based on sub-pixel sampling and fuses feature information at different scales through spatial attention and channel attention.
[0010] To achieve the objective of this invention, the following technical solution is proposed:
[0011] A lightweight super-resolution network based on sub-pixel sampling is constructed by the following steps:
[0012] Using a 4-layer Unet++ network as the basic network structure, subpixel downsampling is employed for downsampling, and subpixel upsampling for upsampling is used for upsampling. Specifically:
[0013] Based on Unet++, it also includes: a spatial attention mechanism module, a channel attention mechanism module, and multi-scale feature fusion;
[0014] Before entering the Unet++ network architecture, the number of image channels is expanded to 64 through a 3×3 convolutional layer; each unit of Unet++ includes two 3×3 convolutional layers in addition to the spatial attention mechanism module, and each convolutional layer is followed by a ReLU activation function;
[0015] The extracted features are input into the Unet++ unit, and feature mappings at different scales are learned through subpixel downsampling and subpixel upsampling.
[0016] Dense connections are used to connect non-adjacent units that extract the same feature scale together;
[0017] At the end of the Unet++ network, feature maps of different scales are upsampled to the same size to perform multi-scale feature fusion.
[0018] Then, the channel attention mechanism of the channel attention mechanism module and the 3×3 convolutional layer are used to better fuse the feature maps of different sampling levels together;
[0019] The image is then enlarged to ×2, ×3, and ×4 of the low-resolution image by another subpixel upsampling;
[0020] Finally, a residual structure is used to connect the network input and output.
[0021] Preferably, the sub-pixel upsampling specifically involves combining pixels at the same location from multiple feature maps into a new feature map, making full use of the complementary information between each feature map. This process is referred to as sub-pixel upsampling.
[0022] The subpixel downsampling is the inverse process of subpixel upsampling, specifically: the feature map is decomposed into multiple new feature maps based on the pixels of adjacent parts.
[0023] Preferably, the spatial attention mechanism module includes three convolutional layers, including two convolutional layers with kernel size of 1 and one convolutional layer with kernel size of 3. First, a 1×1 convolutional layer is used to integrate the number of channels, and then the weight of each pixel is obtained through the last convolutional layer and the Sigmoid gating function, giving more weight to important feature points.
[0024] Preferably, in the channel attention mechanism module, the features are first passed through two 3×3 convolutional layers, then globally pooled, and then subjected to dimensionality increase and decrease by two 1×1 convolutional layers. Finally, the weight of each feature channel is obtained using the Sigmoid gating function, and channels with rich features are given greater weight.
[0025] This invention also provides an application method for a lightweight super-resolution network based on sub-pixel sampling, the specific steps of which include:
[0026] S1: Prepare the training set and preprocess the training images;
[0027] S2: A lightweight super-resolution network based on sub-pixel sampling was constructed;
[0028] The steps for building a lightweight super-resolution network based on sub-pixel sampling include:
[0029] Using a 4-layer Unet++ network as the basic network structure, subpixel downsampling is employed for downsampling, and subpixel upsampling for upsampling is used for upsampling. Specifically:
[0030] Based on Unet++, it also includes: a spatial attention mechanism module, a channel attention mechanism module, and multi-scale feature fusion;
[0031] Before entering the Unet++ network architecture, the number of image channels is expanded to 64 through a 3×3 convolutional layer; each unit of Unet++ includes two 3×3 convolutional layers in addition to the spatial attention mechanism module, and each convolutional layer is followed by a ReLU activation function;
[0032] The extracted features are input into the Unet++ unit, and feature mappings at different scales are learned through subpixel downsampling and subpixel upsampling.
[0033] Dense connections are used to connect non-adjacent units that extract the same feature scale together;
[0034] At the end of the Unet++ network, feature maps of different scales are upsampled to the same size to perform multi-scale feature fusion.
[0035] Then, the channel attention mechanism of the channel attention mechanism module and the 3×3 convolutional layer are used to better fuse the feature maps of different sampling levels together;
[0036] The image is then enlarged to ×2, ×3, and ×4 of the low-resolution image by another subpixel upsampling;
[0037] Finally, a residual structure is used to connect the network input and output.
[0038] The subpixel upsampling process involves combining pixels at the same location from multiple feature maps into a new feature map, making full use of the complementary information between each feature map. This process is referred to as subpixel upsampling.
[0039] The subpixel downsampling is the inverse process of subpixel upsampling, specifically: the feature map is decomposed into multiple new feature maps based on the pixels of adjacent parts.
[0040] The spatial attention mechanism module includes three convolutional layers, including two convolutional layers with a kernel of 1 and one convolutional layer with a kernel of 3. First, a 1×1 convolutional layer is used to integrate the number of channels. Then, the weights of each pixel are obtained through the last convolutional layer and the Sigmoid gating function, and more weights are given to important feature points.
[0041] In the channel attention mechanism module, the features are first passed through two 3×3 convolutional layers, then globally pooled, and then increased and decreased in dimensionality by two 1×1 convolutional layers. Finally, the weight of each feature channel is obtained by using the Sigmoid gating function, and channels with rich features are given greater weight.
[0042] S3: Input the processed dataset into the lightweight super-resolution network based on subpixel sampling described in step S2 for training;
[0043] S4: Input the test image into the trained super-resolution network described in step S3 for verification.
[0044] Preferred,
[0045] In step S1, the training set dataset is the DIV2K dataset;
[0046] Data augmentation is achieved through horizontal flipping, vertical flipping, and image rotation at 90°, 180°, and 270°.
[0047] The high-resolution image was cropped to 192×192, 144×144 and 96×96, which correspond to upsampling factors of ×4, ×3 and ×2, respectively.
[0048] All upsampling factors correspond to an input low-resolution image size of 48×48.
[0049] Preferred,
[0050] During the training of S3, low-resolution images with a batch size of 16 are first randomly selected from the processed dataset and input into the subpixel sampling super-resolution network. Adam is used as the optimizer for model training, where β1 = 0.9, β2 = 0.999, and ∈ = 10. -8 ;
[0051] During training, the initial learning rate is set to lr = 10. -4 The number of training cycles is halved every 50 training cycles, for a total of 200 training cycles.
[0052] Advantages and beneficial effects of the present invention:
[0053] 1. The network model designed in this invention does not generate any parameters during subpixel upsampling or downsampling, which can reduce network complexity to a certain extent.
[0054] 2. The subpixel sampling method in this invention does not lose image information when performing feature transformation on the image.
[0055] 3. This invention designs a spatial attention mechanism module and a channel attention mechanism module suitable for lightweight networks, which can meet the needs of feature fusion and have fewer parameters. Attached Figure Description
[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 This is a flowchart illustrating specific embodiments of the present invention;
[0058] Figure 2This is a structural diagram of the lightweight super-resolution network based on sub-pixel sampling of the present invention;
[0059] Figure 3 This is a schematic diagram of sub-pixel sampling in this invention;
[0060] Figure 4 This is a schematic diagram of the spatial attention mechanism module in this invention;
[0061] Figure 5 This is a schematic diagram of the channel attention mechanism module in this invention;
[0062] Figure 6 This is a comparison image of the images generated by the present invention and other super-resolution networks in this embodiment. Detailed Implementation
[0063] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0064] The purpose of this invention is to improve the sampling methods proposed in existing super-resolution models. This invention proposes a sub-pixel sampling method and constructs a lightweight super-resolution network based on sub-pixel sampling. While improving the performance of the super-resolution network, this invention further optimizes the network structure, reducing computational load and model parameters.
[0065] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.
[0066] For details regarding the lightweight super-resolution network based on sub-pixel sampling constructed in this invention, please refer to [link / reference]. Figure 2 ,
[0067] Using a 4-layer Unet++ as the basic network structure, subpixel downsampling is employed for downsampling, and subpixel upsampling is employed for upsampling. Specifically, subpixel upsampling involves combining pixels at the same location from multiple feature maps to form a new feature map, fully utilizing the complementary information between each feature map. This process is called subpixel upsampling. Subpixel downsampling is the inverse process of subpixel upsampling, specifically: based on the pixels of adjacent parts, the feature map is decomposed into multiple new feature maps.
[0068] The specific construction process is as follows:
[0069] Based on Unet++, it also includes: a spatial attention mechanism module, a channel attention mechanism module, and multi-scale feature fusion;
[0070] Before entering the Unet++ network architecture, a 3×3 convolutional layer is used to expand the number of image channels to 64; each unit of Unet++
[0071] In addition to the spatial attention mechanism module, it also includes two 3×3 convolutional layers, each of which is followed by a ReLU activation function;
[0072] The extracted features are input into the Unet++ unit, and feature mappings at different scales are learned through subpixel downsampling and subpixel upsampling.
[0073] Dense connections are used to connect non-adjacent units that extract the same feature scale together;
[0074] At the end of the Unet++ network, feature maps of different scales are upsampled to the same size to perform multi-scale feature fusion.
[0075] Then, the channel attention mechanism of the channel attention mechanism module and the 3×3 convolutional layer are used to better fuse the feature maps of different sampling levels together;
[0076] The image is then enlarged to ×2, ×3, and ×4 of the low-resolution image by another subpixel upsampling;
[0077] Finally, a residual structure is used to connect the network input and output.
[0078] The sub-pixel sampling proposed in this invention is as follows: Figure 3 As shown, pixels at the same location in multiple feature maps are combined to form a new feature map, fully utilizing the complementary information between each feature map; this process is called subpixel upsampling. Simultaneously, its inverse process is called subpixel downsampling, decomposing the feature map into four new feature maps based on adjacent pixels. The values in the figure represent subpixel upsampling / downsampling at a scale of 2. Subpixel downsampling reduces the image size to half its original size, while increasing the number of channels from 64 to 256, converting spatial information to channel information and preserving image features to the greatest extent. Similarly, subpixel upsampling converts the original 64 channels into 16 channels, doubling the size of the feature map. Experiments using this invention demonstrate that subpixel sampling, compared to pooling downsampling and deconvolution upsampling, improves image quality to a certain extent and achieves faster convergence during training.
[0079] In addition, the spatial attention mechanism module and the channel attention mechanism module used in this invention are as follows: Figure 4 and Figure 5As shown, the spatial attention mechanism module contains only three convolutional layers: two with a kernel of 1 and one with a kernel of 3. First, a 1×1 convolutional layer is used to consolidate the number of channels. Then, the weights for each pixel are determined by the last convolutional layer and the Sigmoid gating function, giving more weight to important feature points. In the channel attention mechanism module, features are first passed through two 3×3 convolutional layers, then globally pooled, and then subjected to dimensionality upscaling and downscaling by two 1×1 convolutional layers. Finally, the Sigmoid gating function is used again to determine the weights for each feature channel, giving greater weight to channels with rich features.
[0080] like Figure 1 As shown, this invention also discloses an application method for a lightweight super-resolution network based on sub-pixel sampling, the specific implementation steps of which include:
[0081] S1: Prepare the training set and preprocess the training images;
[0082] In step S1, the training set dataset is the DIV2K dataset;
[0083] Data augmentation is achieved through horizontal flipping, vertical flipping, and image rotation at 90°, 180°, and 270°.
[0084] The high-resolution image was cropped to 192×192, 144×144 and 96×96, which correspond to upsampling factors of ×4, ×3 and ×2, respectively.
[0085] All upsampling factors correspond to an input low-resolution image size of 48×48.
[0086] S2: A lightweight super-resolution network based on sub-pixel sampling was constructed;
[0087] The steps for building a lightweight super-resolution network based on sub-pixel sampling include:
[0088] Using a 4-layer Unet++ network as the basic network structure, subpixel downsampling is employed for downsampling, and subpixel upsampling for upsampling is used for upsampling. Specifically:
[0089] Based on Unet++, it also includes: a spatial attention mechanism module, a channel attention mechanism module, and multi-scale feature fusion;
[0090] Before entering the Unet++ network architecture, the number of image channels is expanded to 64 through a 3×3 convolutional layer; each unit of Unet++ includes two 3×3 convolutional layers in addition to the spatial attention mechanism module, and each convolutional layer is followed by a ReLU activation function;
[0091] The extracted features are input into the Unet++ unit, and feature mappings at different scales are learned through subpixel downsampling and subpixel upsampling.
[0092] Dense connections are used to connect non-adjacent units that extract the same feature scale together;
[0093] At the end of the Unet++ network, feature maps of different scales are upsampled to the same size to perform multi-scale feature fusion.
[0094] Then, the channel attention mechanism of the channel attention mechanism module and the 3×3 convolutional layer are used to better fuse the feature maps of different sampling levels together;
[0095] The image is then enlarged to ×2, ×3, and ×4 of the low-resolution image by another subpixel upsampling;
[0096] Finally, a residual structure is used to connect the network input and output.
[0097] The subpixel upsampling process involves combining pixels at the same location from multiple feature maps into a new feature map, making full use of the complementary information between each feature map. This process is referred to as subpixel upsampling.
[0098] The subpixel downsampling is the inverse process of subpixel upsampling, specifically: the feature map is decomposed into multiple new feature maps based on the pixels of adjacent parts.
[0099] The spatial attention mechanism module includes three convolutional layers, including two convolutional layers with a kernel of 1 and one convolutional layer with a kernel of 3. First, a 1×1 convolutional layer is used to integrate the number of channels. Then, the weights of each pixel are obtained through the last convolutional layer and the Sigmoid gating function, and more weights are given to important feature points.
[0100] In the channel attention mechanism module, the features are first passed through two 3×3 convolutional layers, then globally pooled, and then increased and decreased in dimensionality by two 1×1 convolutional layers. Finally, the weight of each feature channel is obtained by using the Sigmoid gating function, and channels with rich features are given greater weight.
[0101] S3: Input the processed dataset into the lightweight super-resolution network based on subpixel sampling described in step S2 for training;
[0102] During the training of S3, low-resolution images with a batch size of 16 are first randomly selected from the processed dataset and input into the subpixel sampling super-resolution network. This invention uses Adam as the optimizer for model training, where β1 = 0.9, β2 = 0.999, and ∈ = 10. -8During training, we set the initial learning rate lr = 10. -4 The number of training cycles is halved every 50 cycles, for a total of 200 training cycles. The network designed in this invention is implemented based on the PyTorch architecture and trained using an NVIDIA RTX 3090 GPU. Furthermore, this invention uses L1 loss instead of L2 loss function to achieve better convergence and results.
[0103] S4: Input the test image into the trained super-resolution network described in step S3 for verification.
[0104] In step S4, in order to better verify the effect of the present invention, the present invention uses the publicly available Urban100 dataset for comparison. Figure 6 This paper compares the results of the Subpixel Sampling Super-Resolution Network (SSN) of this invention with those of Bicubic Interpolation, Deep Convolutional Network Image Super-Resolution (SRCNN), Fast Convolutional Neural Network Super-Resolution (FSRCNN), Fast Image Super-Resolution Based on Deep Convolutional Neural Networks (VDSR), Deep Recurrent Residual Network Image Super-Resolution (DRRN), and Multiple Degradation Image Super-Resolution (SRMDNF). To better observe the differences in high-resolution image prediction among each method, we locally magnify the super-resolution predicted image. Figure 6 As can be seen, this invention not only significantly improves upon specific data but also delivers better visual effects compared to other methods, more closely resembling realistic HR images. Secondly, this method also preserves texture to a greater extent, especially noticeable in building lines. For the reconstruction of Urban100 img098, SSN's texture reconstruction effect is particularly outstanding, while other methods exhibit distortion and blurring to some extent. Furthermore, SSN also has a significant advantage in prediction time on the same device compared to other methods.
[0105] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A lightweight super-resolution network based on sub-pixel sampling, characterized in that, The construction steps include: Using a 4-layer Unet++ network as the basic network structure, subpixel downsampling is employed for downsampling, and subpixel upsampling for upsampling is used for upsampling. Specifically: Based on Unet++, it also includes: a spatial attention mechanism module, a channel attention mechanism module, and multi-scale feature fusion; Before entering the Unet++ network architecture, the number of image channels is expanded to 64 through a 3×3 convolutional layer; each unit of Unet++ includes two 3×3 convolutional layers in addition to the spatial attention mechanism module, and each convolutional layer is followed by a ReLU activation function; The extracted features are input into the Unet++ unit, and feature mappings at different scales are learned through subpixel downsampling and subpixel upsampling. Dense connections are used to link non-adjacent units of the same feature scale together; At the end of the Unet++ network, feature maps of different scales are upsampled to the same size to perform multi-scale feature fusion. Then, the channel attention mechanism of the channel attention mechanism module and the 3×3 convolutional layer are used to better fuse the feature maps of different sampling levels together; Subpixel upsampling was used to enlarge the image to ×2, ×3, and ×4 of the low-resolution image, respectively. Finally, a residual structure is used to connect the network input and output. The subpixel upsampling process involves combining pixels at the same location from multiple feature maps to form a new feature map, making full use of the complementary information between each feature map. This process is referred to as subpixel upsampling. The subpixel downsampling is the inverse process of subpixel upsampling, specifically: the feature map is decomposed into multiple new feature maps based on the pixels of adjacent parts; The spatial attention mechanism module includes three convolutional layers, including two convolutional layers with a kernel of 1 and one convolutional layer with a kernel of 3. First, a 1×1 convolutional layer is used to integrate the number of channels, and then the weight of each pixel is obtained through the last convolutional layer and the Sigmoid gate function, giving more weight to important feature points. In the channel attention mechanism module, the features are first passed through two 3×3 convolutional layers, then globally pooled, and then increased and decreased in dimensionality by two 1×1 convolutional layers. Finally, the Sigmoid gating function is used to obtain the weight of each feature channel, and channels with rich features are given greater weight.
2. An application of the lightweight super-resolution network based on sub-pixel sampling as described in claim 1, characterized in that, The specific steps include: S1: Prepare the training set and preprocess the training images; S2: Construct the lightweight super-resolution network based on sub-pixel sampling as described in claim 1; S3: Input the processed dataset into the lightweight super-resolution network based on subpixel sampling described in step S2 for training; S4: Input the test image into the super-resolution network trained in step S3 for verification.
3. The application of the lightweight super-resolution network based on sub-pixel sampling according to claim 2, characterized in that, In step S1, the training set dataset is the DIV2K dataset; Data augmentation is achieved through horizontal flipping, vertical flipping, and image rotation at 90°, 180°, and 270°. The high-resolution image was cropped to 192×192, 144×144 and 96×96, which correspond to upsampling factors of ×4, ×3 and ×2, respectively. All upsampling factors correspond to an input low-resolution image size of 48×48.
4. The application of the lightweight super-resolution network based on sub-pixel sampling according to claim 2, characterized in that, During the training of S3, low-resolution images with a batch size of 16 are first randomly selected from the processed dataset and input into a lightweight super-resolution network based on subpixel sampling. Adam is used as the optimizer for model training. , , ; During training, set the initial learning rate. The number of training cycles is halved every 50 training cycles, for a total of 200 training cycles.