An image sampling method for image super-resolution

Abstract

This invention discloses an image sampling method for image super-resolution, relating to the field of image super-resolution reconstruction. It aims to solve the problems of low reconstruction detail and unsuitability for decoding in existing technologies. The technical solution involves lossless spatial downsampling through a BRB module combined with a pixel deshuffling operator, and quantizing the original image pixel information using a quantization operator. During reconstruction, the decoder decodes the quantized pixel information to obtain the complete pixel details of the original image, thereby continuously recovering lost high-frequency details during upsampling and achieving highly accurate super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, specifically to an image sampling method for image super-resolution. Background Technology

[0002] A typical image super-resolution process can be expressed by mathematical formulas. y = DHx + n Let represent the original high-resolution image to be estimated, H be the blurring operator, D be the downsampling operator, n be additive noise, and y be the observed low-resolution image. Based on different assumptions about D, H, and n in the degradation model, existing technologies can be mainly classified into the following categories: 1. Traditional interpolation-based methods: These methods assume that the downsampling D is uniform sampling, the blurring operator H is an identity matrix, and noise is ignored. Missing pixels are estimated using predefined kernel functions such as bilinear or bicubic interpolation. Although computationally complex, they often lead to blurred edges because they ignore structural correlations at the image level.

[0003] 2. Sparse Representation-Based Methods: These methods typically assume that the downsampling operator D is a predefined bicubic downsampling algorithm and the blurring operator H is a specific blur kernel when dealing with degradation models. The core logic involves mapping image patches to a high- and low-resolution coupled dictionary and reconstructing the image by solving the following optimization problem: in For low-resolution dictionaries The sparse coefficients are used. The limitations of this method are: operators D and H are treated as fixed prior conditions, making it difficult to handle complex unknown degradation, and the iterative computation of sparse coding is inefficient in large-scale applications. This type of method is based on the prior assumption that natural image patches are sparse under an overcomplete dictionary, and reconstructs the image by learning the mapping relationship between high- and low-resolution dictionaries. Although it has strong mathematical interpretability, it suffers from bottlenecks such as high computational cost in the sparse coding stage and limited expressive power of the fixed dictionary.

[0004] 3. Deep Learning-Based Approaches: Modern deep learning methods approximate the inverse process of a degenerate model by constructing an end-to-end mapping network G. In most supervised learning scenarios, this model implicitly assumes that D is a fixed downsampling operation (such as bicubic downsampling) and H is the identity matrix. Its objective function is typically defined as: in These are the parameters of the neural network to be learned. They are learned using an existing dataset, and the learning process ends when a high reconstruction effect is achieved. During the inference phase, the learned parameters are used for super-resolution reconstruction. ,in It is a low-resolution input image. It is a reconstructed high-resolution variable rate image.

[0005] All of the above super-resolution methods are based on a common premise: that the algorithm cannot actively intervene in the downsampling process at the imaging end. However, in practical applications (such as Scalable Video Coding, SVC), imaging systems often have a built-in fixed downsampling module to optimize bandwidth or storage.

[0006] Chinese patent CN114266699B discloses an image super-resolution method, electronic device, and chip. Based on a NAS network, it stacks downsampling and upsampling modules to form a network structure. After inputting a low-resolution image, it achieves super-resolution reconstruction through identity mapping during the downsampling process. This existing patent still relies on the decoder being able to obtain the encoder's original data. When the decoder cannot obtain the encoder's original data, the reconstruction detail will significantly decrease.

[0007] In summary, existing super-resolution image reconstruction methods, when bandwidth is limited or enhancement layer data is lost, can only acquire low-resolution information from the base layer at the decoding end, resulting in poor reconstruction details. Existing downsampling operators (such as D) are mostly fixed mathematical mappings and do not consider how to preserve information based on image content features. Therefore, given a high-resolution source image, designing a downsampling scheme that adaptively preserves key features to achieve higher-quality reconstruction at the decoding end has become a core challenge in improving the performance of image compression and transmission systems. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the existing defects and provide an image sampling method for image super-resolution, which can effectively solve the problems in the background technology.

[0009] To achieve the above objectives, this invention discloses an image sampling method for image super-resolution. The technical solution includes high-resolution image degradation and super-resolution reconstruction. The high-resolution image degradation includes downsampling by the encoder, and the super-resolution reconstruction includes upsampling by the decoder. The downsampling is based on basic residual units and compresses the original resolution according to the geometric scaling law. During compression, lossless downsampling is achieved by introducing a pixel deshuffling operator, ultimately obtaining a floating-point downsampled image. The above downsampling process ensures that all original pixel information is retained in the tensor as feature values ​​after spatial compression; and the structures of the decoder and encoder are highly symmetrical. The floating-point downsampled image obtained by downsampling is then converted into a quantized image conforming to the uint8 format standard by introducing a quantization operator. ; The super-resolution reconstruction utilizes the original pixel information retained during downsampling.

[0010] On the encoder side, the core task of the model is to project the input high-resolution image onto a low-dimensional manifold space. This process utilizes a deep convolutional neural network model to perform a nonlinear downsampling mapping. According to the geometric scaling law, for 2x and 3x super-resolution configurations, the encoder compresses the spatial resolution to 1 / 4 and 1 / 9 of the original size, respectively. This drastic spatial dimensionality reduction is not only to reduce data redundancy but also to force the deep convolutional neural network model to learn a task-driven, highly condensed latent representation. Before the downsampled representation is input to the decoder, it must be quantized using the Q-operator to ensure that the data storage format is uint8.

[0011] The strategic focus of the encoding phase lies in the nonlinear expansion of the channel dimension. The model projects the initial 3-channel RGB image into a high-dimensional latent space with 512 channels. Based on this high-dimensional representation, the model introduces a pixel unshuffle operator. Unlike traditional bicubic downsampling, which leads to irreversible entropy loss by discarding high-frequency components, pixel unshuffle exchanges spatial resolution for channel depth by a squared ratio through space-to-depth reconstruction. This lossless downsampling mechanism ensures that all original pixel information is retained in the tensor as feature values ​​after spatial compression, laying the information foundation for subsequent accurate reconstruction. At the boundary between the codec and the encoder, a quantization operator Q is introduced. In traditional continuous-valued neural networks, feature values ​​are usually in float32 format, but in actual video encoding, satellite data transmission, or terminal storage scenarios, data must follow discrete bit depth constraints (such as uint8 format).

[0012] The decoder side is responsible for performing the inverse mapping task. Another meticulously designed convolutional neural network model (highly symmetrical to the encoder side structure) serves as the reconstruction engine, performing in-depth analysis of the degraded representations from the encoder. Utilizing discriminative priors preserved in previous-level features, the decoder continuously recovers lost high-frequency details during layer-by-layer upsampling. Through this end-to-end joint modeling, the decoder can effectively compensate for information loss caused by downsampling and quantization, ultimately synthesizing a high-resolution reconstructed image that performs exceptionally well in both perceptual quality and objective metrics (such as PSNR).

[0013] The decoder and encoder in the reconstruction phase exhibit a high degree of structural symmetry, and their core task is to achieve the inverse mapping from the compressed latent space to the high-dimensional pixel space. Their upsampling mechanism employs the pixel shuffle operator, corresponding to pixel unshuffle. This operator reorganizes the channel-dimensional features, redistributing the condensed channel information back into the spatial coordinate system.

[0014] The present invention takes a super-resolution image as input and quantizes the data that needs to be retained during the image degradation process, so that the data is decoded by the decoder during the reconstruction process and thus high-resolution reconstruction is achieved.

[0015] As a preferred embodiment of the present invention, the basic residual unit includes a main path and a shortcut path; the shortcut path includes an identity mapping mode and a projection mapping mode; When the step size is 1 and the number of input channels equals the number of output channels, the identity mapping mode is used for processing; Otherwise, use the projection mapping mode.

[0016] By dynamically selecting processing logic, computational redundancy can be minimized, ensuring the universality of residual learning and training stability.

[0017] As a preferred embodiment of the present invention, the downsampling process is as follows: Step 1, hierarchical feature extraction and channel expansion, Using basic residual units to process the input high-resolution image The channel dimension expands significantly from 3 to 512, as shown in the following equation: , in, This refers to the basic residual unit; Step 2, Feature Refinement and Dimensional Compression The features extracted in step 1 are refined using basic residual units, and the channel dimension is compressed to 16 dimensions: , , , , ; Step 3, Multi-granularity collaborative fusion, Execute the channel-level stitching operator to stitch the original image The extracted features at each level are aggregated into a unified high-dimensional tensor, as shown in the following equation: , in, Feature aggregation; Step 4, lossless spatial downsampling. The spatial pixels are rearranged to the channel dimension using a pixel deshuffling operator, reducing the spatial resolution without discarding information. The downsampled features are then fused using basic residual units to generate a floating-point downsampled image. : , , , , , in, Rearrange the pixel deshuffler operator; Step 5, Differentiable quantization, The result obtained in step 4 Mapping to the [0, 255] interval, a quantization operator is introduced. That is, by performing rounding operations through a pass-through estimator, a quantized image conforming to the uint8 format standard is generated. : .

[0018] As a preferred embodiment of the present invention, the super-resolution reconstruction process is as follows: Step a, quantize the image Perform feature restoration and dimensionality reduction. quantized image through basic residual units Perform feature restoration: , in, This refers to the basic residual unit; This reduces the channel dimension from 512 to 16: , , , , ; Step b, collaboration between shallow and deep features. Connect and fuse deep features with shallow features during the decoding process: , in, Feature aggregation; The dimension of the fused feature channels is expanded to 256: ; Step c, lossless spatial upsampling, By using the inverse operation of the pixel shuffling operator as an anti-shuffling upsampling operator, information from the channel dimension is remapped back to the spatial dimension, thereby obtaining a higher resolution feature map. , , in, Rearrange the pixel shuffle operator; Step d, final image reconstruction, The concatenation decoder takes as input the upsampled features and outputs a reconstructed high-resolution image through basic residual units. : , .

[0019] As a preferred embodiment of the present invention, the basic residual unit receives the input tensor. Dimensions ,in, The input channel is represented by H and W, which represent the spatial resolution of the input, respectively. The output dimension is... ; Indicates the output channel. and These represent the output spatial resolution, expressed by the formula. express, This indicates network output.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: When high-resolution images degrade, the present invention introduces a pixel deshuffling operator to exchange the spatial resolution for channel depth by a square multiple. This lossless downsampling mechanism ensures that all original pixel information is still retained in the tensor in the form of feature values ​​after spatial compression. Then, the feature values ​​of all original pixel information are quantized by the quantization operator to meet the transmission and storage requirements. During the super-resolution reconstruction process, the decoder can obtain the original pixel information by decoding the quantized feature values, thereby continuously recovering the lost high-frequency details during the upsampling process and achieving high-accuracy super-resolution reconstruction. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the main framework of the image downsampling based on a convolutional neural network model for image super-resolution according to the present invention; Figure 2This is a comparison table of the super-resolution reconstruction performance (PSNR) of different methods under 2x downsampling according to the present invention; Figure 3 This is a comparison table of the reconstruction results of the present invention and Comparative Example 1 on different datasets; Figure 4 This is a table showing the environment packages and their corresponding versions that the neural network model program of this invention depends on. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 First, the basic residual block (BRB) involved in this embodiment will be described: Receive input tensor (i.e., the input high-resolution image), its dimensions are , The input channel is represented by H and W, which represent the spatial resolution of the input, respectively. The output dimension is... ; Indicates the output channel. and These represent the output spatial resolution, expressed by the formula. express, This represents the network output. A basic residual unit consists of a main path and a shortcut path. The main path contains a sequentially connected first convolutional layer, first normalization layer, first activation layer, second convolutional layer, and second normalization layer. The shortcut path contains a projection mapping unit for dimension alignment.

[0024] -Main path processing steps: Step 1, First-level feature extraction, Input tensor First, we enter the first convolutional layer, using... The convolution kernel performs the processing. The stride is configured to 1 (preserving size). This convolution operation does not use a bias term, and the number of output channels is [number missing]. ; Step 2, spatial dimension normalization, The output of the first convolutional layer is subjected to group normalization. In this embodiment, the number of groups is set to 1, that is, a channel-wise normalization operation equivalent to layer normalization is performed, followed by in-place nonlinear activation using the ReLU function; Step 3, Second-level feature extraction, The activated features enter the second convolutional layer and are then processed. The convolution kernel has a constant stride of 1 to ensure that the spatial resolution of the feature map is no longer changed. Step 4, Secondary normalization, The output of the second convolution is then subjected to group normalization with a group size of 1 to generate the main path output features. .

[0025] -Shortcut Path processing steps: Step 5: The system dynamically selects processing logic based on the matching of input and output dimensions. Identity mapping mode: when stride=1 and the number of input channels Equal to the number of output channels At this time, the shortcut path does not perform any transformations and directly converts the input tensor. Transmitted to the summing end; Projection mapping mode: When any of the conditions of the above identity mapping mode are not met (downsampling or channel transformation exists), a shortcut path is activated. Convolutional layers (with stride consistent with the main path) and corresponding group normalization layers will convert the input tensor... Project onto a dimension space consistent with the main path output to generate projection features. .

[0026] - Residual fusion and final output Step 6: Add the features element by element. Output the main path Output of shortcut path or Element-wise addition is performed at the corresponding positions, which can be expressed as follows: ; Step 7, final nonlinear activation, The ReLU activation function is called again on the summed tensor to output the final residual block features. .

[0027] like Figure 1As shown, based on the aforementioned BRB module, this invention discloses an image sampling method for image super-resolution. The technical solution adopted includes high-resolution image degradation and super-resolution reconstruction, wherein high-resolution image degradation is a downsampling process of the encoder, and the specific process is as follows: Step a, hierarchical feature extraction and channel expansion, Use the BRB module to process the input high-resolution image The channel dimension expands significantly from 3 to 512, as shown in the following equation: ; In this embodiment, In the expression, Indicates the calculation order of the BRB module; Step b, feature refinement and dimensionality compression. The BRB module is used to refine the features extracted in step a and compress the channel dimension to 16 dimensions: , , , , ; Step c, multi-granularity collaborative fusion, Execute the channel-level concatenation operator (Concat) to concatenate the input tensor The original image and the extracted features at each level are aggregated into a unified high-dimensional tensor, as shown in the following equation: , in, Feature aggregation; Step d: Lossless spatial downsampling (Unshuffle). Using the pixel_unshuffle operator (downsampling factor) The spatial pixels are rearranged to the channel dimension, reducing the spatial resolution from (H, W) to (H / 2, W / 2) without discarding information. The downsampled features are then fused using the BRB module to generate a floating-point downsampled image. : , , , , , in, Rearrange the pixel_unshuffle operator; Step e, differentiable quantization, The result obtained in step d Mapping to the [0, 255] interval, a quantization operator is introduced. That is, by performing rounding operations using a straight-through estimator (STE), a quantized image conforming to the uint8 format standard is generated. : .

[0028] The super-resolution reconstruction process is as follows: Step A, quantize the image Perform feature restoration and dimensionality reduction. quantized image through basic residual units Perform feature restoration: , This reduces the channel dimension from 512 to 16: , , , , ; Step B, Deep and shallow feature collaboration. By concatenating and fusing deep and shallow features during the decoding process, multi-scale contextual information is fully utilized. , in, Feature aggregation; The dimension of the fused feature channels is expanded to 256: ; Step C, lossless spatial upsampling (Shuffle). Using the inverse operation of the pixel_shuffle operator as the upsampling operator (upsampling factor) for unshuffle. This involves remapping channel-dimensional information back to the spatial dimension, thereby obtaining higher-resolution feature maps without introducing artificial interpolation artifacts. , , in, Rearrange the pixel shuffle operator; Step D, final image reconstruction, The concatenation decoder takes the upsampled features as input and outputs a reconstructed high-resolution image through basic residual units. : , .

[0029] To verify the effectiveness of Example 1, Example 1 was compared with common bilinear interpolation, bicubic interpolation and SwinFIR, using the public dataset Set14 as the test images.

[0030] The results are as follows Figure 2 As shown, Figure 2 The results show a comparison of PSNR values ​​achieved by different interpolation methods on the Set14 standard test image set. The results indicate that Example 1 achieves superior performance compared to existing state-of-the-art methods, with an average improvement of 1.95 dB.

[0031] Furthermore, Example 1 and Comparative Example 1 (Chinese Patent CN118644389A: Lightweight Encoder-Decoder Image Super-Resolution Reconstruction Method, Storage Medium, and Electronic Device Based on Bidirectional Feedback Mechanism) were compared on different datasets (publicly available standard datasets Set5, Set14, and BSD100). Since Comparative Example 1 focuses on the luminance recovery of the Y channel while neglecting detail compensation of the chroma component, for a fair comparison, the PSNR of the Y channel was used for both examples. The comparison results are as follows: Figure 3 As shown, by Figure 3 As can be seen, the reconstruction effect of Example 1 on different datasets is on average 13.18dB higher than that of the comparison example. This is because Example 1 proposes a full-channel collaborative reconstruction mechanism, which can more effectively capture complementary information between channels by simultaneously optimizing the three RGB channels. This significantly improves the reconstruction accuracy and visual consistency of colors while maintaining brightness fidelity.

[0032] The training process based on the convolutional neural network model involved in Example 1 will be described below: -Model Environment Configuration The program system environment uses Linux. The program's dependencies, packages, and their corresponding versions are as follows: Figure 4 As shown; -Parameter Settings In Example 1, the training process of the model is implemented in the PyTorch computing framework. In order to balance the consumption of computing resources and the smoothness of model gradient updates, the batch size of the training task is configured to 8. The L1 norm loss function is used as a supervision signal in the loss function (Loss) to guide the model to maintain more accurate edge response and stronger robustness in the image reconstruction process; Furthermore, the adaptive moment estimation (Adam) algorithm is used as the optimization operator, and the initial learning rate is configured for parameter iteration.

[0033] To optimize the model's convergence performance in the later stages of training, a stepped learning rate adjustment mechanism was introduced: the total number of iterations was set to 1000 full-sample epochs; the learning rate was reduced by a decay factor of 0.5 every 100 epochs. This dynamic adjustment strategy helps the model quickly approach the optimal solution in the early stages of training and escape local minima in the later stages through fine-grained parameter adjustments, thereby achieving a better peak signal-to-noise ratio (PSNR) gain.

[0034] -Training set The original training sample library selected in this embodiment is a publicly available benchmark image set with high spatial resolution features (such as DIV2K or the DF2K set extended from it). This sample library contains thousands of static images with high pixel density (2K and above), covering complex topological structures, repetitive textures, and rich color gradients. This massive and high-quality raw data input provides sufficient supervision signals for the neural network to learn deep nonlinear feature maps.

[0035] To optimize memory overhead and improve computational efficiency during model training, Example 1 performs non-overlapping spatial block processing on the original high-resolution image. Specifically, a single large-size original image is divided into multiple sub-patches of 512×512 pixels. In the image block preprocessing flow of Example 1, a non-overlapping priority partitioning criterion is adopted: First, the original high-resolution image is divided into equally spaced sub-patches with a preset step size (e.g., 512 pixels) to generate non-overlapping standardized sub-patches. For the special case of image edges, if the width or height of the original image cannot be divided evenly by the preset step size, backtracking alignment sampling is performed on the remaining area at the end. Specifically, the end image block will be cropped back by 512 pixels based on the image edge, resulting in local pixel overlap between the end image block and its adjacent previous image block.

[0036] The model training was terminated after 1000 training iterations.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image sampling method for image super-resolution, comprising high-resolution image degradation and super-resolution reconstruction, wherein the high-resolution image degradation includes downsampling by an encoder, and the super-resolution reconstruction includes upsampling by a decoder, characterized in that: The downsampling is based on basic residual units and compresses the original resolution according to the geometric scaling law. During compression, lossless downsampling is achieved by introducing a pixel deshuffling operator, and finally a floating-point downsampled image is obtained. The above downsampling process ensures that all original pixel information is still retained in the tensor in the form of feature values ​​after spatial compression. The floating-point downsampled image obtained by downsampling is then converted into a quantized image conforming to the uint8 format standard by introducing a quantization operator. ; The super-resolution reconstruction utilizes the original pixel information retained during downsampling.

2. The image sampling method for image super-resolution according to claim 1, characterized in that: The basic residual unit includes the main path and the shortcut path; the shortcut path includes the identity mapping mode and the projection mapping mode. When the step size is 1 and the number of input channels equals the number of output channels, the identity mapping mode is used for processing; Otherwise, use the projection mapping mode.

3. The image sampling method for image super-resolution according to claim 2, characterized in that: The downsampling process is as follows: Step 1, hierarchical feature extraction and channel expansion, Using basic residual units to process the input high-resolution image The channel dimension expands significantly from 3 to 512, as shown in the following equation: ， in, This refers to the basic residual unit; Step 2, Feature Refinement and Dimensional Compression The features extracted in step 1 are refined using basic residual units, and the channel dimension is compressed to 16 dimensions: ， ， ， ， ； Step 3, Multi-granularity collaborative fusion, Execute the channel-level stitching operator to stitch the original image The extracted features at each level are aggregated into a unified high-dimensional tensor, as shown in the following equation: ， in, Feature aggregation; Step 4, lossless spatial downsampling. The spatial pixels are rearranged to the channel dimension using a pixel deshuffling operator, reducing the spatial resolution without discarding information. The downsampled features are then fused using basic residual units to generate a floating-point downsampled image. : ， ， ， ， ， in, Rearrange the pixel deshuffler operator; Step 5, Differentiable quantization, The result obtained in step 4 Mapping to the [0, 255] interval, a quantization operator is introduced. That is, by performing rounding operations through a pass-through estimator, a quantized image conforming to the uint8 format standard is generated. : 。 4. The image sampling method for image super-resolution according to claim 1 or 3, characterized in that: The super-resolution reconstruction process is as follows: Step a, quantize the image Perform feature restoration and dimensionality reduction. quantized image through basic residual units Perform feature restoration: ， in, This refers to the basic residual unit; This reduces the channel dimension from 512 to 16: ， ， ， ， ； Step b, collaboration between shallow and deep features. Connect and fuse deep features with shallow features during the decoding process: ， in, Feature aggregation; The dimension of the fused feature channels is expanded to 256: ； Step c, lossless spatial upsampling, By using the inverse operation of the pixel shuffling operator as an anti-shuffling upsampling operator, information from the channel dimension is remapped back to the spatial dimension, thereby obtaining a higher resolution feature map. ， ， in, Rearrange the pixel shuffle operator; Step d, final image reconstruction, The concatenation decoder takes as input the upsampled features and outputs a reconstructed high-resolution image through basic residual units. : ， 。 5. The image sampling method for image super-resolution according to claim 2 or 3, characterized in that: The received input tensor of the basic residual unit Dimensions ,in, The input channel is represented by H and W, which represent the spatial resolution of the input, respectively. The output dimension is... ; Indicates the output channel. and These represent the output spatial resolution, expressed by the formula. express, This indicates network output.

Images