Image super-resolution method based on dynamic modulation of nonlinear quantization according to feature attributes

Abstract

This invention discloses an image super-resolution method based on feature attribute dynamic modulation nonlinear quantization. The steps are as follows: S1, acquire the super-resolution model and the image set for training; S2, construct a bit-width dynamic modulation module, using the frequency domain spectral entropy calculated through the forward input tensor as a feature attribute index to achieve dynamic bit-width modulation; S3, construct an activation value quantization module, performing nonlinear compression transformation on the forward input tensor and outputting the quantized activation value; S4, construct a weight value quantization module, outputting the quantized weight tensor through uniform quantization of the channel-level weight tensor; S5, perform convolution operation between the quantized activation value and the quantized weight tensor; S6, train the super-resolution model improved by step S5 using the image set. This method achieves a peak signal-to-noise ratio / structural similarity superior to existing mainstream training quantization methods at low bit depths, visually eliminates artifacts, restores a clear structure, and does not increase additional inference computation overhead.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to an image super-resolution method based on dynamic modulation of feature attribute nonlinear quantization. BACKGROUND

[0002] Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) observation image, and is a basic and key problem in the field of computer vision. In recent years, methods based on deep convolutional neural networks (CNN) and visual Transformers (ViT) (such as EDSR, RCAN, SwinIR, etc.) have made significant breakthroughs in reconstruction quality. However, these performance improvements are often accompanied by huge computational overhead and storage requirements. Modern super-resolution models usually have millions of parameters and extremely high floating-point operations (FLOPs), which leads to a huge memory bottleneck, severely limiting their deployment and real-time application on resource-constrained hardware such as mobile terminals and edge devices.

[0003] In order to solve the above deployment problems, model quantization has become a mainstream compression technology. Quantization can significantly reduce model size and utilize hardware acceleration inference by converting the original high-precision floating-point numbers (such as FP32) to low-bit integers (such as INT8 or INT4). Quantization methods are mainly divided into two categories: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). Although QAT can maintain good performance at very low bits, it requires complete original training data sets for high-cost retraining, which is not applicable in scenarios where data privacy is limited or computing power is limited. In contrast, PTQ only needs a small amount of calibration data to directly quantize the pre-trained model, has the characteristics of high efficiency and "plug and play" without retraining, and is therefore more suitable for rapid deployment.

[0004] Although PTQ has deployment advantages, existing PTQ methods for super-resolution models still face serious performance degradation problems at low bit settings, mainly in the following two aspects:

[0005] First, existing linear quantization strategies cannot effectively adapt to the non-uniform distribution characteristics of super-resolution model features. Most existing PTQ methods employ a linear uniform quantization grid, which assumes that the data is relatively uniformly distributed within the quantization range. However, the activation values ​​of feature maps in super-resolution networks (especially residual structures) typically exhibit significant non-uniform distribution characteristics, such as a "heavy-tailed" distribution or a "peak" distribution highly concentrated near the zero point. Under such distributions, linear quantization leads to severe resource misallocation: on the one hand, a large number of quantization levels are allocated to the low-probability, large-value tail region, resulting in a waste of bit width resources; on the other hand, the information-dense region near the zero point (containing crucial high-frequency residual information) is sparsely sampled due to the excessively large quantization step size, leading to severe quantization noise.

[0006] Second, existing bit-width allocation strategies rely on static or single amplitude metrics, making it difficult to achieve precise resource allocation based on image content. To balance compression ratio and performance, mixed-precision quantization is widely adopted. However, existing bit-width allocation strategies primarily rely on simple amplitude-based statistical metrics (such as local variance, gradient norm, or Hessian matrix) to measure feature importance. These metrics merely reflect the energy strength of the signal, exhibiting "energy-structure ambiguity": high-energy regions may be meaningless random noise, while low-energy regions may contain regular textures or edge structures that are sensitive to human perception. Existing methods often lack in-depth analysis of image feature attributes (such as structural regularity and texture complexity), leading to high-bit-width resources being wasted on noise, while crucial structural details are allocated low bit-widths due to lower amplitudes, resulting in loss of detail or artifacts in the reconstructed image.

[0007] In summary, the mismatch between the linear uniform quantization grid and the distribution characteristics of super-resolution data in existing technologies, as well as the lack of perception of image structural features by bit-width allocation strategies based on a single amplitude index, are the main reasons for the performance degradation of low-bit super-resolution models. Therefore, there is an urgent need to develop a nonlinear quantization method that can adaptively adapt to the data distribution characteristics, and an adaptive mechanism that can dynamically modulate the quantization strategy according to image feature attributes, in order to achieve high-fidelity image reconstruction with low computational cost. Summary of the Invention

[0008] The purpose of this invention is to provide an image super-resolution method based on feature attribute dynamic modulation nonlinear quantization that solves the problems of large quantization noise and loss of key details in the post-training quantization of existing super-resolution models.

[0009] Therefore, the technical solution of the present invention is as follows:

[0010] An image super-resolution method based on feature attribute dynamic modulation nonlinear quantization includes the following steps:

[0011] S1. Obtain the super-resolution model and the image set for training;

[0012] S2. Construct multiple bit-width dynamic modulation modules and establish target correspondences with each convolutional layer in the super-resolution model. Calculate the frequency domain spectral entropy based on the forward input tensor of the target convolutional layer and use the frequency domain spectral entropy as a feature attribute index. Perform dynamic bit-width modulation on the target convolutional layer with the goal of preserving key details.

[0013] S3. Construct multiple activation value quantization modules and establish target correspondences with each convolutional layer in the super-resolution model. Perform nonlinear compression transformation on the forward input tensor of the target convolutional layer based on the target quantization bit width to output the quantized activation value.

[0014] S4. Construct multiple weight quantization modules and establish target correspondences with each convolutional layer in the super-resolution model. With minimizing the mean square error of the tensors before and after quantization as the objective function, search for the optimal cutoff point of the weight tensor of each channel to complete the uniform quantization of the channel-level weight tensors and output the quantized weight tensors.

[0015] S5. Improve the super-resolution model as follows: replace the input tensor of the corresponding target convolutional layer with the quantized activation values ​​output by each activation value quantization module, and replace the weight tensor of the corresponding target convolutional layer with the quantized weight tensor output by each weight value quantization module.

[0016] S6. The improved super-resolution model is trained using the image set obtained in step S1, so that when a low-resolution image is input, the improved super-resolution model outputs a high-resolution super-resolution image.

[0017] Furthermore, the specific steps of step S2 are as follows:

[0018] S201. Obtain the forward input tensor of the target convolutional layer. It is divided into several local feature blocks of the same size using a sliding window;

[0019] S202. Perform a discrete cosine transform on each local feature block to calculate the DCT coefficients of each local feature block and generate a DCT coefficient tensor.

[0020] S203. Apply a high-frequency mask to calculate the AC power spectrum of the DCT coefficient tensor; calculate the energy probability distribution of each frequency component within the AC power spectrum to obtain the tensor. The frequency domain spectral entropy, and normalized to The range is used to obtain characteristic attribute indicators;

[0021] S204, Based on Feature Attribute Indicators The bit width of the target convolutional layer is dynamically modulated.

[0022] Furthermore, the specific operation steps of step S201 are as follows:

[0023] 1) Define the sliding window size as M×M, the step size as M, and the value of M is in the range of 5-10;

[0024] 2) Determine the tensor The fill amount in the height and width directions is then applied using a reflective fill method, along the spatial dimensions H and W of the tensor. Perform boundary expansion to obtain the expanded filling tensor. The reflection fill method achieves expanded filling by mirroring the edge pixel values.

[0025] 3) Using a sliding window, expand the filling tensor. Divide the data into several local feature blocks and output the set of local feature block tensors P={P i Its dimensions are , where i is the index of the local feature block. This represents the total number of local feature blocks.

[0026] Furthermore, the specific steps of step S203 are as follows:

[0027] 1) Calculate the AC power spectrum of the DCT coefficient tensor, its expression is:

[0028] ,

[0029] In the formula, For DCT coefficient tensors; It is an M×M binary mask, with the DC component in the upper left corner set to 0 and the other AC components set to 1.

[0030] 2) Calculate the energy probability distribution of each frequency component in the AC power spectrum. Its expression is:

[0031] ,

[0032] In the formula, Frequency index; It is a minimal constant, and is generally set to This is used to prevent division by zero errors.

[0033] 3) Calculate tensors spectral entropy in the frequency domain and normalized to Range, to obtain tensor Feature attribute indicators Its expression is:

[0034] ,

[0035] ,

[0036] In the formula, For the maximum possible entropy, .

[0037] Furthermore, the operation steps of step S204 are described as follows.

[0038] 1) Based on feature attribute indicators The new judgment threshold range is calculated, and its expression is:

[0039] ,

[0040] In the formula, This is the initial threshold range. A coefficient used to control the modulation intensity;

[0041] 2) Obtain the local feature block tensor Corresponding local importance index It fills the tensor by expanding the local feature block. Obtained by taking the absolute mean or local gradient norm;

[0042] 3) Assign local importance indices to each local feature block With the new judgment threshold The comparison is performed, and the bit width modulation operation is completed based on the comparison result to output the target quantization bit width. :

[0043] like If the upper limit of the value is used to determine that the local feature block contains structural features in the original image that need to be preserved, then the baseline quantization bit width of the current target convolutional layer is determined. Perform an operation with a width of +1 and output the target quantization width. for: ;

[0044] like The lower limit value is used to determine whether the local feature block corresponds to background or noise in the original image. Then, the baseline quantization bit width of the current target convolutional layer is determined. Perform a -1 bit width operation and output the target quantization bit width. for: .

[0045] 4) Use the target quantization bit width calculated in step 3) The baseline quantization bit width of the target convolutional layer Update.

[0046] Furthermore, in step S3, the specific operation steps of each activation value quantization module are as follows:

[0047] S301, Learnable lower bound parameter and upper bound parameters For the forward input tensor of the target convolutional layer Perform a truncation operation;

[0048] S302. The truncated tensor is divided and scaled using a scaling factor to obtain a normalized tensor. ;

[0049] S303. Obtain the current target quantization bit width of the target convolution, and adjust the hyperparameters controlling the curvature of the quantization grid according to the different bit width ranges it is in. Perform step-by-step mapping assignment: 1) When the target quantization bit width If so, the current state is determined to be extremely low bit quantization, and then set... ;2) When If so, the current state is determined to be low-to-medium bit quantization, and then set... 3) When the target quantization bit width If the current state is determined to be high-bit quantization, then set... ;

[0050] S304, based on hyperparameters Perform a forward compression nonlinear transformation;

[0051] S305. Calculate the quantization level based on the current target quantization bit width parameter of the target convolutional layer, in order to achieve the desired quantization level in the compressed domain. Perform uniform rounding quantization operation internally.

[0052] Furthermore, in step S3, based on hyperparameters The expression for the nonlinear transformation of forward compression is:

[0053] , ,

[0054] In the formula, For symbolic functions, but , but , but ;

[0055] Quantization series The calculation expression is:

[0056] ;

[0057] Based on quantization series In the compression domain The expression for performing uniform rounding quantization is:

[0058] ,

[0059] In the formula, This indicates that the calculation result is rounded down to the nearest integer.

[0060] Furthermore, in step S4, the specific operation steps of each weight value quantization module are as follows:

[0061] S401. Slice the weight tensor in the target convolutional layer along the output channel dimension to obtain multiple channel-level weight tensors;

[0062] S402. Determine the weight tensor for each channel based on the lower and upper bound parameters of the truncation interval. Initial value of the cutoff interval ( ;

[0063] S403. With the objective of minimizing the mean square error of the tensors before and after quantization, search for the weight tensor of each channel. Find the optimal truncation interval and perform uniform rounding quantization to output the quantization weight tensor for each channel;

[0064] S404. Re-stack the multiple channel-level weight tensors that have been uniformly quantized to obtain the quantized weight tensor.

[0065] Compared with existing technologies, this image super-resolution method based on feature attribute dynamic modulation nonlinear quantization innovatively introduces a feature attribute index, frequency domain spectral entropy, into a bit-width dynamic modulation module to dynamically modulate the bit-width allocation strategy. By effectively distinguishing between "high-energy noise" and "high-energy structural texture," not only can key textures obtain high bit-width protection, but bit resources are also avoided from being wasted on random noise. On the other hand, this method optimizes the activation value quantization module and the weight value quantization module to solve the problem of insufficient quantization accuracy in the effective information region, which leads to excessive error in traditional linear quantization. Furthermore, this method achieves a performance breakthrough under low bit conditions. Under extremely challenging low bit settings (such as 4-bit), the objective indicators (peak signal-to-noise ratio / structural similarity) of this invention are superior to existing mainstream training quantization methods on multiple benchmark datasets. It also visually eliminates artifacts, restores clear structures, and does not increase additional inference computation overhead. Attached Figure Description

[0066] Figure 1 This is a flowchart of the image super-resolution method based on feature attribute dynamic modulation nonlinear quantization of the present invention;

[0067] Figure 2 This is a schematic diagram comparing the adaptation characteristics of the dynamic bit-width modulation strategy in the method of this invention and the amplitude static strategy in the traditional method for non-uniformly distributed data.

[0068] Figure 3 In the method of this invention, A schematic diagram of the compression characteristic curves of the nonlinear transformation function under different parameters, taking -law as an example;

[0069] Figure 4 This diagram illustrates the mapping relationship between the input value and the quantized output value in the method of this invention, as well as the alignment between quantization and data distribution.

[0070] Figure 5 This is a comparison diagram of the super-resolution reconstruction visual effects of the method of the present invention and existing mainstream post-training quantization methods on a standard test set in this embodiment of the invention.

[0071] Figure 6 The image shown is a schematic diagram comparing the image processing of the original image (img090) in the ablation experiment in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0073] See Figure 1 The specific implementation steps of this image super-resolution method based on feature attribute dynamic modulation nonlinear quantization are described below.

[0074] S1. Obtain the super-resolution model and the image set for training.

[0075] In step S1, the super-resolution model can be, but is not limited to, the EDSR model or the RDN model; the image set used for training can directly use a large-scale public dataset, such as DIV2K or Flickr2K; the dataset contains low-resolution image data and corresponding high-resolution image data; the calibration set used subsequently is constructed by randomly sampling from the image set.

[0076] S2. Construct multiple bit-width dynamic modulation modules and establish target correspondences with each convolutional layer in the super-resolution model. Calculate the frequency domain spectral entropy based on the forward input tensor of the target convolutional layer and use the frequency domain spectral entropy as a feature attribute index. Perform dynamic bit-width modulation on the target convolutional layer with the goal of preserving key details.

[0077] In step S2, the construction of the bit-width dynamic modulation module is used to solve the energy-structure ambiguity problem caused by the amplitude-dependent static strategy in traditional methods, and to achieve accurate content-aware allocation. Based on this, the present invention constructs a bit-width dynamic modulation module, which obtains the forward input of each target convolutional layer to calculate the feature attribute index: frequency domain spectral entropy, and then uses the frequency domain spectral entropy to dynamically modulate the bit-width allocation strategy. This adaptive strategy can effectively distinguish between "low-energy noise" and "low-energy structural texture", thereby ensuring that key textures in the image can obtain high bit-width protection by dynamically reducing the bit width of low-energy noise and increasing the bit width of low-energy structural texture, while avoiding wasting bit resources used for image processing on random noise.

[0078] In step S2, the specific operation steps of each bit-width dynamic modulation module are described as follows.

[0079] S201. Obtain the forward input tensor of the target convolutional layer. It is divided into several local feature blocks of the same size using a sliding window.

[0080] In step S201, the forward input tensor of the target convolutional layer These are activation values ​​passed down from the previous convolutional layer or input layer. , Indicates batch size (BatchSize) Indicates the number of channels. Indicates height. Indicates the width.

[0081] In step S201, the tensor is used to... The specific steps for dividing the feature into several local feature blocks of the same size are described below.

[0082] 1) Define the sliding window size as Step size is , for tensor Perform non-overlapping block operations.

[0083] 2) To ensure that both H and W in the spatial dimension are divisible by 8, we first determine the tensor. The fill amount in the height and width directions is then applied using a reflective fill method, along the spatial dimensions H and W of the tensor. Perform boundary expansion to obtain the expanded filling tensor. The reflection filling method specifically achieves extended filling by mirroring the edge pixel values; this method fills in the spatial dimensions while avoiding high-frequency edge interference introduced by zero filling.

[0084] 3) Use a sliding window to divide the extended padded tensor into several local feature blocks and output the set of local feature block tensors P = {P i}, whose size is , where i is the serial number of the local feature block, is the total number of local feature blocks.

[0085] S202. Perform discrete cosine transform (DCT) on each local feature block to generate a DCT coefficient tensor by calculating the DCT coefficients of each local feature block.

[0086] In this step S202, the calculation formula for the DCT coefficient of each feature block is:

[0087] where

[0088] In the formula, is the pre-calculated transformation matrix of the 8×8 local feature block, and is the local feature block tensor;

[0089] Furthermore, a DCT coefficient tensor is generated from the DCT coefficients of all the calculated local feature blocks.

[0090] S203. Apply a high-frequency mask (Mask) to calculate the alternating current power spectrum of the DCT coefficient tensor , and then calculate the frequency domain spectral entropy of the tensor based on the energy probability distribution of each frequency component in the alternating current power spectrum, and normalize it to the range, that is, the characteristic attribute index is obtained.

[0091] Specifically, the operation steps of this step S203 are described as follows.

[0092] 1) Calculate the alternating current power spectrum of the DCT coefficient tensor to remove the direct current (DC) component and only retain the alternating current (AC) frequency energy, and its expression is:

[0093] where

[0094] In the formula, is the DCT coefficient tensor; is an 8×8 binary mask, and its DC component in the upper left corner is set to 0, and the remaining AC components are set to 1.

[0095] 2) Calculate the energy probability distribution of each frequency component in the alternating current power spectrum Its expression is:

[0096] ,

[0097] In the formula, Frequency index; It is a minimal constant, and is generally set to This is used to prevent division by zero errors.

[0098] 3) Calculate tensors The frequency domain spectral entropy, and normalized to The range, i.e., the tensor obtained. Feature attribute indicators Its calculation expression is:

[0099] ,

[0100] ,

[0101] In the formula, For the maximum possible entropy, .

[0102] For the final calculated feature attribute index The lower the value, the stronger the tensor. The stronger the regularity of structural features (such as edges and textures) in the corresponding original image region.

[0103] S204, Based on Feature Attribute Indicators The bit width of the target convolutional layer is dynamically modulated.

[0104] Specifically, the operation steps of step S204 are described as follows.

[0105] 1) Based on feature attribute indicators The new judgment threshold range is calculated, and its expression is:

[0106] ,

[0107] In the formula, This is the initial threshold range. A coefficient used to control the modulation intensity.

[0108] In practice, Human-set experience values; initial judgment threshold The threshold range is defined as follows: First, the mean of the absolute values ​​or the local gradient norm of the tensor of each image in the calibration set is calculated. The results are arranged in a list, and the results at the 10th percentile and 90th percentile are taken from the list to form the initial judgment threshold range. The calibration set consists of at least 100 low-resolution images randomly selected from the dataset obtained in step S1.

[0109] 2) Obtain the local feature block tensor Corresponding local importance index It fills the tensor by expanding the local feature block. It can be obtained by taking the absolute mean or the local gradient norm.

[0110] 3) Assign local importance indices to each local feature block With the new judgment threshold The comparison is performed, and the bit width modulation operation is completed based on the comparison result to output the target quantization bit width. :

[0111] like If the upper limit of the value is used to determine that the local feature block contains structural features in the original image that need to be preserved, then the baseline quantization bit width of the current target convolutional layer is determined. Perform an operation with a width of +1 and output the target quantization width. for: ;

[0112] like The lower limit value is used to determine whether the local feature block corresponds to background or noise in the original image. Then, the baseline quantization bit width of the current target convolutional layer is determined. Perform a -1 bit width operation and output the target quantization bit width. for: .

[0113] 4) Use the target quantization bit width calculated in step 3) The baseline quantization bit width of the target convolutional layer Update.

[0114] This dynamic bit-width modulation strategy ensures that even in certain image regions, although local energy... Lower, but structurally strong ( Low leads to (Lowered), but can still cross the threshold to obtain higher quantization bit width protection.

[0115] like Figure 2The diagram shows a comparison of the adaptability of the dynamic bit-width modulation strategy in the method of this invention and the amplitude static strategy in the traditional method for non-uniformly distributed data. In the diagram, the horizontal axis is the image gradient, which is used to represent energy, and the vertical axis is the frequency domain spectral entropy, which is used to characterize the structural strength of the image (0 represents strong structure, 1 represents noise).

[0116] from Figure 2 As can be seen, the traditional static amplitude strategy for determining bit width (as shown by the dotted line in the figure) relies solely on energy for partitioning, resulting in the low-energy, strong-structure region in the lower left corner, i.e., the position marked by the red five-pointed star, being incorrectly assigned a low bit width, causing loss of image details. In contrast, the dynamic bit width modulation strategy of this invention uses a leftward tilt to the boundary based on frequency domain spectral entropy partitioning, effectively correcting the low-energy, strong-structure region in the lower left corner to a higher bit width range. In practical applications, this dynamic bit width modulation strategy demonstrates its ability to preserve key details such as distant building textures in image processing.

[0117] S3. Construct multiple activation value quantization modules and establish target correspondences with each convolutional layer in the super-resolution model, so as to quantize based on the target bit width. Nonlinearly apply the forward input tensor of the target convolutional layer Compress the changes to output the quantized activation values.

[0118] In step S3, the construction of the activation value quantization module is used to solve the problems of distribution mismatch and bit width resource waste in traditional linear activation quantization methods, specifically leading to blurring in the image processing results. Based on this, and considering the non-uniform distribution (such as Laplacian distribution) of the super-resolution residual features, this invention introduces a nonlinear quantization mechanism, namely, nonlinear... Compression significantly increases the sampling density in data-dense regions (such as near zero) without increasing the total bit width budget. Compared to linear quantization, the method of this invention can significantly reduce the quantization error in the effective information region, effectively preserving the high-frequency details of the image, thereby improving the quantization signal-to-noise ratio.

[0119] The specific operation steps of each activation value quantization module in step S3 are described below.

[0120] S301, Learnable lower bound parameter and upper bound parameters For the forward input tensor of the target convolutional layer Perform a truncation operation.

[0121] In step S301, the truncation operation is implemented using a boundary truncation function to limit the range of tensor values, ultimately outputting a truncated tensor. .

[0122] S302. The truncated tensor is divided and scaled using a scaling factor to obtain a normalized tensor. .

[0123] In step S302, the scaling factor Defined as: This will result in a normalized tensor. Strictly mapped to the normalized numerical range Inside.

[0124] S303. Obtain the current target quantization bit width of the target convolution. To adjust the hyperparameters controlling the curvature of the quantized grid based on its different bit width ranges. The specific rules for performing step-by-step mapping assignment are as follows:

[0125] When the target quantization bit width If so, the current state is determined to be extremely low bit quantization, and then set... To perform strong nonlinear compression, so as to preserve residual texture in near-zero region with high accuracy;

[0126] when If so, the current state is determined to be low-to-medium bit quantization, and then set... This allows for the effective processing of images by performing moderate nonlinear compression, which takes into account both large and small amplitude feature distributions.

[0127] When the target quantization bit width If the current state is determined to be high-bit quantization, then set... By performing weak nonlinear compression, the compression characteristics can be made close to a linear mapping, thus achieving effective image processing.

[0128] like Figure 3 The figure shown is a method of the present invention using The diagram illustrates the compression characteristic curves of the nonlinear transformation function, exemplified by -law, under different parameters. As can be seen from the diagram, when... Larger values ​​(e.g.) When the curve has a very steep slope near zero, it means that strong nonlinear compression will be performed to preserve the residual texture of the near-zero region with high precision, which is very suitable for low-bit (such as 4-bit) quantization. However, traditional linear quantization methods do not take into account the different distribution of image features, resulting in quantization errors in the effective information region and low quantization signal-to-noise ratio.

[0129] like Figure 4The figure shows a comparison of the nonlinear quantization method of this invention and the traditional linear quantization method in terms of quantization mapping and data distribution. Part (a) of the figure is a comparison of the mapping relationship between the input value and the quantized output value. The ideal identity mapping (dashed line) represents the lossless original continuous value. The gray steps represent traditional linear quantization. The width and height of the steps are fixed (uniform step size), indicating that the accuracy is the same regardless of whether the input value is large or small. The orange steps represent the nonlinear quantization of this method. The steps are very small near the zero point, which means that the nonlinear quantization method of this invention is very sensitive to changes in small values ​​and retains more details. The steps become very large at the end, which means that the nonlinearity of this invention is significant. For larger values, quantization methods only record a general range, sacrificing some precision. Part (b) in the figure shows the alignment comparison between quantization and data distribution. The cyan shading represents the Laplace distribution in the deep learning model, where most values ​​are near 0, with only a few outliers distributed on both sides. The gray dashed line represents the traditional quantization level. Regardless of the amount of data, the dividing line (quantization level) is evenly distributed, indicating that the scale is insufficient in the densest central region, while the scale is wasted on the sparse sides. The orange solid line represents the quantization level of this method. In the area with the highest data frequency, the orange lines are stacked very densely, ensuring maximum bit width utilization.

[0130] S304, based on hyperparameters The nonlinear transformation for forward compression is expressed as follows:

[0131] , ,

[0132] In the formula, For symbolic functions, but , but , but .

[0133] After the nonlinear transformation in step S304, the output tensor is... The numerical range is in Interval.

[0134] S305, Based on the current target quantization bit width parameter of the target convolutional layer Calculate the quantization level In the compressed domain Perform uniform rounding quantization operation internally.

[0135] Quantization series The calculation expression is:

[0136] ,

[0137] In the compression domain Perform uniform rounding quantization operation internally:

[0138] ,

[0139] In the formula, This indicates that the calculation result is rounded down to the nearest integer.

[0140] S306. Using the inverse transform formula of forward compression, Map back to the linear domain, then multiply by the scaling factor. Perform inverse normalization and output the quantized activation value tensor. .

[0141] S4. Construct multiple weight quantization modules and establish target correspondences with each convolutional layer in the super-resolution model. With minimizing the mean square error of the tensors before and after quantization as the objective function, search for the optimal cutoff point of the weight tensor for each channel to complete the uniform quantization of the channel-level weight tensors and output the quantized weight tensors.

[0142] In step S4, the construction of the weight quantization module aims to address the heterogeneous dynamic range differences in features across different channels present in traditional weight quantization methods. Specifically, this leads to high-frequency texture blurring and artifacts in smooth regions in the image processing results. Based on this, considering that each convolutional kernel contains multiple different filter characteristics, this invention introduces a channel-level independent optimization mechanism, i.e., calculating the optimal cutoff value for each channel separately. Compared to traditional weight quantization, this invention significantly improves the perceptual quality of low-bit quantization images without increasing model inference overhead.

[0143] The specific operation steps of each weight value quantization module in step S4 are described below.

[0144] S401, For the target convolutional layer with size... The weight tensor along the output channel Dimensional slicing Channel-level weight tensor , These represent the height and width of the convolution kernel, respectively.

[0145] S402. Determine the weight tensor for each channel based on the lower and upper bound parameters of the truncation interval. Initial value of the cutoff interval ( .

[0146] S403. With the objective of minimizing the mean square error of the tensors before and after quantization, search for the weight tensor of each channel. Find the optimal truncation interval and perform uniform rounding quantization to output the quantization weight tensor for each channel.

[0147] Specifically, in step S403, the weight tensor of each channel... The specific method of obtaining it is: based on the channel weight tensor The initial values ​​of the truncation interval are used to obtain the truncation weight tensor. Then, the weight tensor is scaled using a scaling factor. Perform division scaling to obtain the normalized numerical range. Weight tensors within ;Based on the baseline quantization bit width of the target convolutional layer Calculate the quantization level And perform uniform rounding quantization operation to obtain Finally, multiply by the scaling factor. Quantization weight tensor Repeat the above steps to minimize the mean square error of the tensors before and after quantization, and obtain the lower bound parameter of the truncation interval. and upper bound parameters That is, as the channel weight tensor The initial value of the cutoff interval.

[0148] In step S402 above, each channel weight tensor Initial value of the cutoff interval ( It can also be obtained using the same method as step S403, the difference being the quantization series. Based on the initial baseline quantization bit width of the target convolutional layer Calculated.

[0149] S404, for uniformly quantized... The channel-level weight tensors are re-stacked to a size of [size missing]. The weight tensor, i.e. the quantized weight tensor.

[0150] S5. The super-resolution model is improved as follows: the quantized activation values ​​output by each activation value quantization module are used to replace the input tensor of the corresponding target convolutional layer, and the quantized weight tensors output by each weight value quantization module are used to replace the weight tensors of the corresponding target convolutional layer. In this way, the quantized activation values ​​and quantized weight tensors are used to perform standard two-dimensional convolution (Conv2d) operations in the target convolutional layer, and then the output features are passed to the next layer.

[0151] S6. The improved super-resolution model is trained using the image set obtained in step S1 so that when a low-resolution (SR) image is input, the improved super-resolution model outputs a high-resolution (HR) super-resolution image.

[0152] To verify the effectiveness of the method of this invention, the improved super-resolution model based on the method of this invention and existing super-resolution models were tested with W6A6 and W4A4 (4-bit weights / activations) ultra-low bit quantization on the standard datasets Set5, Set14, and Urban100, respectively. Table 1 shows the test results on the standard datasets Set5 and Set14, and Table 2 shows the test results on the standard datasets BSD100 and Urban100; relevant references for existing super-resolution models are attached.

[0153] Table 1:

[0154]

[0155] Table 2:

[0156] As can be seen from the test results in Tables 1 and 2, under the highly challenging W4A4 setting, the proposed methods (EDSR-Ours and RDN-Ours) achieved state-of-the-art performance on all test sets. Particularly noteworthy is the Urban100 dataset, which contains rich textures. The proposed method, based on the improved RDN model, achieved a peak signal-to-noise ratio (PSNR) of 25.98 dB, a significant improvement of 2.6 dB compared to the current state-of-the-art AdaBM method's 23.38 dB. Simultaneously, the feature average bit width (FAB) using the proposed method remained at a low level (3.6-4.1 bits), demonstrating that the performance improvement stemmed from precise bit width allocation rather than blindly increasing computational cost.

[0157] To visually demonstrate the invention's ability to restore structural details, such as Figure 5 This image shows a comparison of the super-resolution reconstruction visual effects of the method of this invention and existing mainstream post-training quantization methods on a standard test set. The image illustrates the reconstruction performance of different methods on the Urban100 dataset under W4A4 settings; specifically... Figure 5 Part (a) shows the image processing comparison results based on the original image (img043) containing a dense grid structure. Specifically, the linear quantization methods used are MinMax and Percentile, both of which have the defect of completely blurring or collapsing the grid. The lines reconstructed by the AdaBM method have obvious breaks and jagged edges. In contrast, the method of this invention (Ours) clearly restores the continuous grid structure and is closest to the high-resolution image (HR) given in the Urban100 dataset. Figure 5 Part (b) shows the image processing comparison results based on the original image (img055) containing low-contrast glass curtain wall lines. Figure 5Section (c) shows the image processing comparison results based on the original image (img081) containing high-contrast color stripes; from Figure 5 The comparison results of parts (b) and (c) show that the method of the present invention effectively eliminates artifacts and maintains the sharpness and smoothness of the lines.

[0158] In summary, the above visual effect reconstruction comparison results fully demonstrate that the method of the present invention effectively solves the distribution adaptation problem and achieves visual effects that are even better than those of the full-precision model in low-bit super-resolution tasks.

[0159] Furthermore, the ablation experiment of the method of the present invention was conducted in the manner shown in Table 3 below.

[0160] Table 3:

[0161]

[0162] In Table 3, "--" indicates that the corresponding steps in the test method use the original traditional processing method, and "√" indicates that the corresponding steps in the test method are replaced with the corresponding steps of the present invention. As can be seen from the test results in Table 3, the method of the present invention (Test 8) has the best performance results compared with other methods.

[0163] like Figure 6 The image shown is a schematic diagram comparing the image processing of the original image (img090) in the ablation experiment. Figure 6 In this context, GT represents a high-resolution image. (Referring to Table 3 and...) Figure 6 Compared with the method of this invention, linear quantization causes a significant drop in the peak signal-to-noise ratio on Urban100, resulting in severe structural collapse and blurring of high-frequency textures such as dense window grids in buildings, with the originally clear geometric structure being completely smoothed out. Removing the dynamic modulation logic based on frequency domain spectral entropy leads to a large number of artifacts in the model in low-contrast areas caused by the failure of high-frequency feature determination. This manifests as obvious breaks and jagged edges in originally continuous fine lines (such as the outline of distant glass curtain walls), as well as unnatural banded color blocks in smooth transition areas. The method of this invention can achieve the best high-fidelity reconstruction effect.

[0164] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0165] References:

[0166] [1] Lim, B., Son, S., Kim, H., Nah, S., and Lee, K. M. Enhanced deepresidual networks for single image super-resolution.In CVPRW, 2017.

[0167] [2] Choukroun, Y., Kravchik, E., Yang, F., and Kisilev, P. Low-bitquantization of neural networks for efficient inference.In ICCV Workshops,2019.

[0168] [3] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard,A. G.,Adam, H., and Kalenichenko, D. Quantization and training of neural networksfor efficient integer arithmetic-only inference. In CVPR, 2018.

[0169] [4] Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., and Fan, R.Fullyquantized network for object detection. In CVPR,2019.

[0170] [5] Tu, Z., Hu, J., Chen, H., and Wang, Y. Toward accurate post-training quantization for image super resolution. In CVPR, 2023.

[0171] [6] Hong, C. and Lee, K. M. Adabm: On-the-fly adaptive bit mappingfor image super-resolution. In CVPR, 2024a.

[0172] [7] Wang, H., Lu, J., Zhang, Y., and Fu, Y. Outlier-aware post-training quantization for image super-resolution. CoRR, 2025a.

[0173] [8] Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. Resid-ualdense network for image super-resolution. In CVPR, 2018b.

Claims

1. An image super-resolution method based on feature attribute dynamic modulation nonlinear quantization, characterized by the following steps: include: S1. Obtain the super-resolution model and the image set for training; S2. Construct multiple bit-width dynamic modulation modules and establish target correspondences with each convolutional layer in the super-resolution model. Calculate the frequency domain spectral entropy based on the forward input tensor of the target convolutional layer and use the frequency domain spectral entropy as a feature attribute index. Perform dynamic bit-width modulation on the target convolutional layer with the goal of preserving key details. S3. Construct multiple activation value quantization modules and establish target correspondences with each convolutional layer in the super-resolution model. Perform nonlinear compression transformation on the forward input tensor of the target convolutional layer based on the target quantization bit width to output the quantized activation value. S4. Construct multiple weight quantization modules and establish target correspondences with each convolutional layer in the super-resolution model. With minimizing the mean square error of the tensors before and after quantization as the objective function, search for the optimal cutoff point of the weight tensor of each channel to complete the uniform quantization of the channel-level weight tensors and output the quantized weight tensors. S5. Improve the super-resolution model as follows: replace the input tensor of the corresponding target convolutional layer with the quantized activation values ​​output by each activation value quantization module, and replace the weight tensor of the corresponding target convolutional layer with the quantized weight tensor output by each weight value quantization module. S6. The improved super-resolution model is trained using the image set obtained in step S1, so that when a low-resolution image is input, the improved super-resolution model outputs a high-resolution super-resolution image.

2. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 1, characterized in that, The steps for step S2 are as follows: S201. Obtain the forward input tensor of the target convolutional layer. It is divided into several local feature blocks of the same size using a sliding window; S202. Perform a discrete cosine transform on each local feature block to calculate the DCT coefficients of each local feature block and generate a DCT coefficient tensor. S203. Apply a high-frequency mask to calculate the AC power spectrum of the DCT coefficient tensor; calculate the energy probability distribution of each frequency component within the AC power spectrum to obtain the tensor. The frequency domain spectral entropy, and normalized to The range is used to obtain characteristic attribute indicators; S204, Based on Feature Attribute Indicators The bit width of the target convolutional layer is dynamically modulated.

3. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 2, characterized in that, The specific operation steps of step S201 are as follows: 1) Define the sliding window size as M×M, the step size as M, and the value of M is in the range of 5-10; 2) Determine the tensor The fill amount in the height and width directions is then applied using a reflective fill method, along the spatial dimensions H and W of the tensor. Perform boundary expansion to obtain the expanded filling tensor. The reflection fill method achieves expanded filling by mirroring the edge pixel values. 3) Using a sliding window, expand the filling tensor. Divide the data into several local feature blocks and output the set of local feature block tensors P={P i Its dimensions are , where i is the index of the local feature block. This represents the total number of local feature blocks.

4. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 2, characterized in that, The specific operation steps of step S203 are as follows: 1) Calculate the AC power spectrum of the DCT coefficient tensor, its expression is: ， In the formula, For DCT coefficient tensors; For an M×M binary mask, the DC component in the upper left corner is set to 0, and the other AC components are set to 1; 2) Calculate the energy probability distribution of each frequency component in the AC power spectrum. Its expression is: ， In the formula, Frequency index; It is a minimal constant, and is generally set to This is used to prevent division by zero errors; 3) Calculate tensors spectral entropy in the frequency domain and normalized to Range, to obtain tensor Feature attribute indicators Its expression is: ， ， In the formula, For the maximum possible entropy, .

5. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 2, characterized in that, The operation steps of step S204 are as follows: 1) Based on feature attribute indicators The new judgment threshold range is calculated, and its expression is: ， In the formula, This is the initial threshold range. A coefficient used to control the modulation intensity; 2) Obtain the local feature block tensor Corresponding local importance index It fills the tensor by expanding the local feature block. Obtained by taking the absolute mean or local gradient norm; 3) Assign local importance indices to each local feature block With the new judgment threshold The comparison is performed, and the bit width modulation operation is completed based on the comparison result to output the target quantization bit width. : like If the upper limit of the value is used to determine that the local feature block contains structural features in the original image that need to be preserved, then the baseline quantization bit width of the current target convolutional layer is determined. Perform an operation with a width of +1 and output the target quantization width. for: ; like The lower limit value is used to determine whether the local feature block corresponds to background or noise in the original image. Then, the baseline quantization bit width of the current target convolutional layer is determined. Perform a -1 bit width operation and output the target quantization bit width. for: ; 4) Use the target quantization bit width calculated in step 3) The baseline quantization bit width of the target convolutional layer Update.

6. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 1, characterized in that, In step S3, the specific operation steps of each activation value quantization module are as follows: S301, Learnable lower bound parameter and upper bound parameters For the forward input tensor of the target convolutional layer Perform a truncation operation; S302. The truncated tensor is divided and scaled using a scaling factor to obtain a normalized tensor. ; S303. Obtain the current target quantization bit width of the target convolution, and adjust the hyperparameters controlling the curvature of the quantization grid according to the different bit width ranges it is in. Perform step-by-step mapping assignment: 1) When the target quantization bit width If so, the current state is determined to be extremely low bit quantization, and then set... ;2) When If so, the current state is determined to be low-to-medium bit quantization, and then set... 3) When the target quantization bit width If the current state is determined to be high-bit quantization, then set... ; S304, based on hyperparameters Perform a forward compression nonlinear transformation; S305. Calculate the quantization level based on the current target quantization bit width parameter of the target convolutional layer, in order to achieve the desired quantization level in the compressed domain. Perform uniform rounding quantization operation internally.

7. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 6, characterized in that, In step S3, based on hyperparameters The expression for the nonlinear transformation of forward compression is: ， ， In the formula, For symbolic functions, but , but , but ; Quantization series The calculation expression is: ； Based on quantization series In the compression domain The expression for performing uniform rounding quantization is: ， In the formula, This indicates that the calculation result is rounded down to the nearest integer.

8. The image super-resolution method based on feature attribute dynamic modulation nonlinear quantization according to claim 1, characterized in that, In step S4, the specific operation steps of each weight value quantization module are as follows: S401. Slice the weight tensor in the target convolutional layer along the output channel dimension to obtain multiple channel-level weight tensors; S402. Determine the weight tensor for each channel based on the lower and upper bound parameters of the truncation interval. Initial value of the cutoff interval ( ; S403. With the objective of minimizing the mean square error of the tensors before and after quantization, search for the weight tensor of each channel. Find the optimal truncation interval and perform uniform rounding quantization to output the quantization weight tensor for each channel; S404. Re-stack the multiple channel-level weight tensors that have been uniformly quantized to obtain the quantized weight tensor.

Images