Dynamic quantification method for super-resolution model of integrated granularity information image

Abstract

The invention discloses an integrated granularity information image super-resolution model dynamic quantization method, which comprises the following steps of: 1, constructing an image super-resolution quantization network which comprises a granularity sensing bit controller and an image entropy generalized distribution statistical module, wherein the granularity sensing bit controller forms a preliminary bit width allocation scheme; in the image super-resolution quantization network, generalized distribution statistics of image entropy is obtained through a generalized distribution statistics module of the image entropy, and a final quantization bit width scheme is obtained based on an entropy-to-bit quantization bit width adjustment mechanism; step 2, training the image super-resolution quantization network by using the training set; and step 3, inputting a to-be-tested low-resolution image into the trained image super-resolution quantization network, and outputting a final super-resolution image. According to the method, the quantization performance and the quantization efficiency of the super-resolution model can be further improved.

Description

Technical Field

[0001] The invention relates to the field of image processing methods, and in particular to a dynamic quantization method for an image super-resolution model integrating granularity information. Background Art

[0002] Single image super-resolution is a classic task in the underlying vision of the computer field, which aims to restore a high-resolution image from a damaged low-resolution image. Benefiting from the rapid development of deep learning, single image super-resolution methods based on convolutional neural networks (CNNs) and transform neural networks (Transformers) have achieved excellent performance. Although subsequent methods have made continuous breakthroughs in super-resolution performance, the model complexity of their methods has also continued to increase, which has also limited their ability to process large-size images (such as 2K and 4K) in practice.

[0003] Model quantization is an effective model compression technique that converts the weights and activation values ​​in the model from high-precision (such as 32-bit floating point numbers) to low-precision (such as 8-bit integer) data representation. Appropriate model quantization methods can reduce computational costs and memory overhead with minimal performance sacrifice. Unlike high-level visual task quantization, image super-resolution tasks are extremely sensitive to quantization. Previous methods modeled by adjusting the quantized representation range or activation value feature distribution. However, these methods ignore that the accuracy degradation caused by quantization may be determined by the image content. Some content in the image is more sensitive to quantization, so the restoration results after quantization with lower bit width are poor. Recently, dynamic quantization has received more attention in the field of image super-resolution. Existing super-resolution model dynamic quantization methods mostly adaptively adjust the quantization bit width corresponding to the layer and image block according to the local content of the image. Although these methods can maintain competitive performance while extending the potential of heavy super-resolution models to edge and mobile devices, they still have shortcomings in the trade-off between super-resolution accuracy and quantization efficiency. In addition, the existing super-resolution model dynamic quantization method adjusts the quantization bit width of each layer individually, which disrupts the inter-layer relationship of the original model, thereby reducing the representation ability of the quantization model.

[0004] Therefore, providing a more effective dynamic quantization method for image super-resolution model to deal with the above problems is an urgent problem to be solved by technicians in this field. Summary of the invention

[0005] The present invention provides a dynamic quantization method for an image super-resolution model integrating granularity information, so as to solve the problem of balancing the accuracy and quantization efficiency of existing dynamic quantization methods for super-resolution models.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A dynamic quantization method for an image super-resolution model integrating granularity information comprises the following steps:

[0008] Step 1: construct an image super-resolution quantization network, wherein the image super-resolution quantization network includes a granularity-aware bit controller and a generalized distribution statistics module of image entropy, wherein:

[0009] The granularity-aware bit controller obtains a multi-level granularity feature representation of the input image through an encoder of multiple downsampling operations, and then obtains the contribution of each image block in the input image to the entire input image based on the multi-level granularity feature representation using a differentiable sampling scheme, and corresponds the contribution of each image block to the entire input image to a potential quantization bit width, thereby forming a preliminary bit width allocation scheme;

[0010] In the image super-resolution quantization network, the image entropy of each image block in the input image is calculated through the generalized distribution statistics module of the image entropy, and the image blocks are sorted from small to large according to the image entropy and then the quantized statistics are performed, thereby obtaining the generalized distribution statistics of the image entropy. Then, based on the entropy-to-bit quantization bit width adjustment mechanism, the larger bit width in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller is adjusted by the entropy-to-bit mechanism according to the obtained generalized distribution statistics of the image entropy, thereby obtaining the final quantization bit width scheme;

[0011] Step 2: Use the training set to train the image super-resolution quantization network constructed in step 1 to obtain a trained image super-resolution quantization network;

[0012] Step 3: Input the low-resolution image to be tested into the trained image super-resolution quantization network obtained in step 2, and output the final super-resolution image.

[0013] In a further step 1, the particle differentiable sampling scheme process is as follows:

[0014] First, all granular features are processed by group normalization and then average pooled to the coarsest granularity;

[0015] Then, all the average pooled granular features are concatenated along the channel dimension, and then global average pooling is used to compress the multi-granular information to obtain the channel statistics;

[0016] The channel statistics are then updated through a linear layer of learnable weights to obtain the gating index;

[0017] Next, based on the principle that unnormalized logarithmic probability can represent the classification distribution of features, the gating index is introduced into the standard Gumbel distribution noise, and the noise samples are integrated to obtain a discrete gating index;

[0018] Finally, a softmax operation is performed on the sum of the gated score and the corresponding random standard Gumbel distribution noise to obtain the contribution of each image block to the entire input image, thereby determining the granularity level and pointing to the corresponding bit width, thereby achieving the correspondence between the contribution of each image block to the entire input image and the potential quantization bit width.

[0019] In the further step 1, the entropy-to-bit quantization bit width adjustment mechanism process is:

[0020] The number and size of quantile thresholds are set, and the generalized distribution statistics of image entropy are divided into multiple intervals. The number of intervals matches the number of adjusted bit widths. The image entropy value of each image block in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller is compared with the entropy value corresponding to the quantile threshold. The corresponding adjusted bit width can be determined according to the divided intervals, thereby achieving entropy-to-bit mechanism bit width adjustment for larger bit widths, and obtaining the final quantization bit width scheme.

[0021] During the further training in step 2, the set quantile threshold is adaptively updated using the exponential moving average method;

[0022] When the exponential moving average method is used for threshold adaptive update, the information entropy of the new training image block is normalized in each iteration and used as the external signal of the exponential moving average.

[0023] During further training in step 2, the average error between the output image and the high-resolution image pixels is used as an optimization constraint.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] The present invention is a super-resolution quantization method with unchanged layer quantization bit width, which provides guidance for low-bit dynamic quantization of image super-resolution models by utilizing multi-granularity clues of different image contents. The method constructs a granularity representation hierarchy from coarse to fine for image blocks, and proposes a granularity-aware bit controller to perform granularity-aware bit width allocation according to image content. Subsequently, the present invention proposes an entropy-to-bit bit width adjustment mechanism to adaptively fine-tune the bit width of high-bit image blocks. The method of the present invention ensures the optimal bit width allocation for each image block through a granularity-aware bit controller and an entropy-to-bit mechanism, without considering the sensitivity between quantization layers as in previous methods. The dynamic quantization method of image super-resolution models that integrates granularity information can further improve the performance and quantization efficiency of super-resolution model quantization. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention discloses a flow chart of a method for dynamic quantization of an image super-resolution model integrating granularity information.

[0027] Figure 2 A computer system program architecture diagram of a dynamic quantization method for an image super-resolution model integrating granularity information disclosed in an embodiment of the present invention.

[0028] Figure 3 A schematic diagram of the results of a dynamic quantization method for an image super-resolution model integrating granularity information disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0030] like Figure 1 As shown, this embodiment discloses a dynamic quantization method for an image super-resolution model integrating granularity information, comprising the following steps:

[0031] Step 1: construct an image super-resolution quantization network, which includes a granularity-aware bit controller and a generalized distribution statistics module of image entropy. The specific description is as follows:

[0032] (1) In this embodiment, the granularity-aware bit controller obtains a multi-level granularity feature representation of the input image through an encoder with multiple downsampling operations.

[0033] For the input image X, the granularity-aware bit controller obtains a multi-level feature representation Z = E(X) through multiple downsampling operations of the encoder, where: Z = Z1, Z2, ..., Z D , Z1, Z2, ... Z D They are different levels of granularity features, Z1 is the finest granularity feature, Z D It is the coarsest granular feature with the smallest resolution, with a total of D granularities; ε(X) represents the process of feature extraction of image X through encoder E.

[0034] Then, the granularity-aware bit controller uses a differentiable sampling scheme based on a multi-level granularity feature representation to obtain the contribution of each image block in the input image to the entire input image, and corresponds the contribution of each image block to the entire input image to the potential quantization bit width to form a preliminary bit width allocation scheme. The preliminary bit width allocation scheme is as follows:

[0035] First, all granular features are processed by group normalization and then average pooled to the coarsest granularity, expressed as: in They are the features after average pooling to the coarsest granularity.

[0036] Then, all the average pooled granular features are After splicing along the channel dimension, the multi-granularity information is compressed through global average pooling to generate the channel statistics S about the image X, as shown in the following formula:

[0037]

[0038] Among them: GAP() represents the global average pooling operation.

[0039] The channel statistic S is then passed through a linear layer with learnable weights Update and get the gated score Where N is the number of candidate bit widths, and the specific update formula is as follows:

[0040] G=W g S

[0041] Next, for each image block X i Using the principle that unnormalized logarithmic probability can represent the classification distribution of features, the gating score is introduced into the standard Gumbel distribution noise σ n ~Gumble(0,1), and integrate the noise samples to obtain a discrete gating index, as shown in the following formula:

[0042] θ i = argmax(g i,n +σ n )

[0043] Where: g i,n represents the i-th image block X i It corresponds to the gating index value of the nth granularity level, θ i Indicates the final quantization bit width level of the i-th image block.

[0044] Finally, a softmax operation is performed on the sum of the gated score and the corresponding random standard Gumbel distribution noise to obtain the contribution of each image block to the entire input image, thereby determining the granularity level and pointing to the corresponding bit width (b1,...,b n ,...,b N ), the contribution of each image block to the entire input image corresponds to the potential quantization bit width. The calculation process is shown in the following formula:

[0045]

[0046] Where: p i is the probability of contribution, the probability of contribution p i ∈[0,1]; τ is the temperature coefficient. In this embodiment, the temperature coefficient τ=1; n is the index of the candidate bit width; N is the number of candidate bit widths; g i,θi is the i-th image block at the selected granularity level θi The gating index under ...; σ θi For the granularity level θ i Standard Gumbel distribution noise introduced.

[0047] (2) In the image super-resolution quantization network, the image entropy of each image block in the input image is calculated through the generalized distribution statistics module of image entropy, and the image blocks are sorted from small to large according to the image entropy and then the quantile statistics are performed to obtain the generalized distribution statistics of the image entropy. Then, based on the entropy-to-bit quantization bit width adjustment mechanism, the larger bit width in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller is adjusted by the entropy-to-bit mechanism according to the obtained generalized distribution statistics of the image entropy, thereby obtaining the final quantization bit width scheme.

[0048] The quantization bit width adjustment mechanism process is as follows: set the number and size of the quantile thresholds, divide the generalized distribution statistics of the image entropy into multiple intervals, match the number of intervals with the number of adjusted bit widths, compare the image entropy value of each image block in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller with the entropy value corresponding to the quantile threshold, and determine the corresponding adjusted bit width according to the divided intervals, thereby achieving the entropy-to-bit mechanism bit width adjustment of the larger bit width to obtain the final quantization bit width scheme. The specific calculation process is described as follows:

[0049] In this embodiment, the generalized distribution statistics of the image entropy refers to the statistics of the generalized distribution of the image entropy of all low-resolution image blocks in the input image. For an h×w image block x (h is the image block height, w is the image block width), the entropy calculation formula is as follows:

[0050]

[0051] Where: μ and v represent the current pixel and adjacent pixel values. express The probability of appearing in x is Represents the probability of the tuple (μ, v) appearing. According to the above formula, the information entropy of all image blocks can be obtained. All image blocks are sorted from small to large according to the information entropy and expressed as

[0052] Then, the information entropy H just obtained is divided into V sub-intervals, and we have in Represents the image block index at the tth percentile and points to the entropy value M is the number of image blocks; V is the number of divided sub-intervals.

[0053] Set two threshold quantiles t1 and t2 to get the corresponding image block index and And the corresponding image entropy values and Therefore, the information entropy H will be divided into three discrete sub-intervals. Given a random image block and its image entropy E, the candidate bit width of the corresponding interval can be obtained by judging the image entropy value with the quantile threshold, and used as the final bit width quantized by the super-resolution model, as shown in the following formula:

[0054]

[0055] Where: c n Indicates the adjusted bit width; C1, C2, and C3 indicate different candidate bit widths.

[0056] Step 2: Use the training set to train the image super-resolution quantization network constructed in step 1 to obtain a trained image super-resolution quantization network.

[0057] In this embodiment, 1-800 images of a widely used image dataset DIV2K are used as the training set of the original dataset, and the training set is used for the continuous process of the entire bit width allocation, adjustment, and super-resolution model quantization. The test set uses three widely used image datasets, including Urban100, Test2K, and Test4K, to evaluate the performance of the super-resolution model quantization in this embodiment.

[0058] In the first cycle of training, the exponential moving average is used to adaptively update the quantile threshold t, as shown in the following formula:

[0059] t (j) =t (j-1) γ+Norm(E) (1-γ)

[0060]

[0061] Where: j represents the order of iteration in the training phase. (j) represents the threshold value of the jth iteration. Norm() represents the image entropy normalization process. t Indicates the entropy value of the current image. and They represent the maximum and minimum values ​​of the image entropy in all image blocks respectively. γ represents the smoothing parameter of the exponential average moving process, which is fixed to 0.9997 in the method of the present invention.

[0062] In this embodiment, the entire training process only uses the average error between the output image and the high-resolution image pixels as the optimization constraint.

[0063] Step 3: Input the low-resolution image to be tested into the trained image super-resolution quantization network obtained in step 2, and output the final super-resolution image.

[0064] The embodiment also discloses a computer system for implementing a dynamic quantization method for an image super-resolution model integrating granularity information. Figure 2 As shown, it includes a processor and a memory, and the storage medium in the memory stores program instructions, and the program instructions include:

[0065] Training module: Input low-resolution-high-resolution image pairs into the network, feed forward the designed model, calculate the loss, perform backpropagation, update the network parameters and save them.

[0066] Test data preprocessing model: preprocess the test data to match the training model structure, that is, change the data type and pixel range of the low-resolution image to match the resolution during training.

[0067] Test module: used to perform super-resolution processing on the low-resolution data to be tested, and output and save the clear image after super-resolution.

[0068] Referring to Table 1, Table 2 and Table 3, the dynamic quantization method of the image super-resolution model integrating granularity information in this embodiment is compared with the performance of different quantization methods on multiple super-resolution model baselines (including super-resolution methods based on CNN and Transformer structures) at different super-resolution multiples. All the results of this embodiment are completed under the setting of the candidate bit width scheme of the granularity-aware bit controller to [4,6,8], entropy to bit width adjustment for all 8-bit schemes that are initially obtained, and the candidate bit width is set to [4,5,8], and the quantile threshold is [0.5,0.9]. The comparison indicators include two widely used reference indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) and feature average bit width (FAB) for evaluation. In addition, in order to verify the restoration performance and quantization efficiency of this embodiment, the table compares the model size of this embodiment and the compression ratio compared to the baseline model. The result symbol ↑ in the table indicates that the higher the better, and the symbol ↓ indicates that the lower the better. Table 1, Table 2, and Table 3 are as follows:

[0069] Table 1: Comparison of ×2 super-resolution quantization results of this embodiment on the baseline model and various quantization methods

[0070]

[0071] Table 2: Comparison of ×4 super-resolution quantization results of this embodiment on the baseline model and various quantization methods

[0072]

[0073]

[0074] Table 3: The results of this example on the baseline model and various quantization methods ×4 super-resolution quantization on Urban100 test

[0075]

[0076] The experimental results of the above examples show that the method of this embodiment can be effectively used for the quantization of different super-resolution model baselines. The effect is obvious because of the existing super-resolution model quantization method. At the same time, it obtains the best trade-off between restoration performance and quantization performance, which further illustrates that the method of this embodiment can solve the problems encountered in super-resolution model quantization.

[0077] Please see attached Figure 3 , is a schematic diagram of image super-resolution results of a dynamic quantization method for an image super-resolution model integrating granularity information disclosed in this embodiment. The experimental results of the example show that the super-resolution image obtained by the method of this embodiment has clear edges and better restored details.

[0078] In summary: The present invention discloses a method for dynamic quantization of image super-resolution that integrates granularity information. First, the present invention proposes to use a granularity-aware bit controller to allocate a quantization bit width to each image block, and to construct a hierarchical structure from coarse granularity to fine granularity based on the expected contribution of the image block to the overall image, and to allocate a corresponding granularity level to each block, which is consistent with the quantization bit width. In this way, the quantization bit width of the image block is dynamically adjusted according to its importance in the entire image. Secondly, the initially allocated bit width is fine-tuned based on the image entropy value to capture the generalized entropy distribution of the image data, and the entropy value is used as an adaptive factor to accurately calibrate the quantization bit width. The advanced level of this embodiment in the field of super-resolution model quantization is verified on different models, different resolutions, and different test sets. The method of the present invention ensures that both a high quantization efficiency can be maintained during the quantization process and detail retention and quality improvement can be ensured during image reconstruction, especially a higher bit width is allocated to important image blocks to achieve the best reconstruction effect.

[0079] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction, and such combinations should also be regarded as the contents disclosed in the present disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0080] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by technical personnel in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A dynamic quantization method for image super-resolution model integrating granularity information, characterized in that: The following steps are involved: Step 1: construct an image super-resolution quantization network, wherein the image super-resolution quantization network includes a granularity-aware bit controller and a generalized distribution statistics module of image entropy, wherein: The granularity-aware bit controller obtains a multi-level granularity feature representation of the input image through an encoder of multiple downsampling operations, and then obtains the contribution of each image block in the input image to the entire input image based on the multi-level granularity feature representation using a differentiable sampling scheme, and corresponds the contribution of each image block to the entire input image to a potential quantization bit width, thereby forming a preliminary bit width allocation scheme; In the image super-resolution quantization network, the image entropy of each image block in the input image is calculated through the generalized distribution statistics module of the image entropy, and the image blocks are sorted from small to large according to the image entropy and then the quantized statistics are performed, thereby obtaining the generalized distribution statistics of the image entropy. Then, based on the entropy-to-bit quantization bit width adjustment mechanism, the larger bit width in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller is adjusted by the entropy-to-bit mechanism according to the obtained generalized distribution statistics of the image entropy, thereby obtaining the final quantization bit width scheme; Step 2: Use the training set to train the image super-resolution quantization network constructed in step 1 to obtain a trained image super-resolution quantization network; Step 3: Input the low-resolution image to be tested into the trained image super-resolution quantization network obtained in step 2, and output the final super-resolution image.

2. The method for dynamic quantization of image super-resolution model integrating granularity information according to claim 1, characterized in that: In step 1, the particle differentiable sampling scheme process is as follows: First, all granular features are processed by group normalization and then average pooled to the coarsest granularity; Then, all the average pooled granular features are concatenated along the channel dimension, and then global average pooling is used to compress the multi-granularity information to obtain the channel statistics; The channel statistics are then updated through a linear layer of learnable weights to obtain the gating index; Next, based on the principle that unnormalized logarithmic probability can represent the classification distribution of features, the gating index is introduced into the standard Gumbel distribution noise, and the noise samples are integrated to obtain a discrete gating index; Finally, a softmax operation is performed on the sum of the gated score and the corresponding random standard Gumbel distribution noise to obtain the contribution of each image block to the entire input image, thereby determining the granularity level and pointing to the corresponding bit width, thereby achieving the correspondence between the contribution of each image block to the entire input image and the potential quantization bit width.

3. The method for dynamic quantization of an image super-resolution model integrating granularity information according to claim 1, characterized in that: In step 1, the entropy-to-bit quantization bit width adjustment mechanism process is: The number and size of quantile thresholds are set, and the generalized distribution statistics of image entropy are divided into multiple intervals. The number of intervals matches the number of adjusted bit widths. The image entropy value of each image block in the preliminary bit width allocation scheme obtained by the granularity-aware bit controller is compared with the entropy value corresponding to the quantile threshold. The corresponding adjusted bit width can be determined according to the divided intervals, thereby achieving entropy-to-bit mechanism bit width adjustment for larger bit widths, and obtaining the final quantization bit width scheme.

4. The method for dynamic quantization of image super-resolution model integrating granularity information according to claim 1, characterized in that: During training in step 2, the exponential moving average method is used to adaptively update the threshold of the set quantile threshold; When the exponential moving average method is used for threshold adaptive update, the information entropy of the new training image block is normalized in each iteration and used as the external signal of the exponential moving average.

5. The method for dynamic quantization of image super-resolution model integrating granularity information according to claim 1, characterized in that: During training in step 2, the average error between the output image and the high-resolution image pixels is used as the optimization constraint.

Images