Image super-resolution method based on local mean binary quantization
The image super-resolution method using local mean binary quantization addresses the limitation of convolutional neural networks on mobile devices and improves the image super-resolution effect, especially enabling real-time applications and information preservation on edge devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2022-06-30
- Publication Date
- 2026-04-10
AI Technical Summary
Existing convolutional neural network image super-resolution models are limited in their application on mobile devices due to their large model size and computational complexity, and existing binary quantization networks suffer excessive information loss in image super-resolution tasks.
An image super-resolution method based on local mean binary quantization is adopted. The weights and activations of the feature extraction and feature transformation modules are binary quantized, and the local mean is used as the quantization threshold to reduce information loss. The computational load is reduced by combining a bit accumulation mechanism.
It significantly reduces the computational cost and parameter count of super-resolution models, improves the performance of image super-resolution tasks, especially enabling real-time applications on edge devices, and preserves more texture and color information.
Smart Images
Figure CN115311136B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and image processing technology, and relates to an image super-resolution method based on local mean binary quantization. Background Technology
[0002] As a typical low-level vision task, image super-resolution aims to reconstruct high-resolution images from low-resolution images, and it has been widely used in many fields, such as medical imaging, public safety, and remote sensing image processing.
[0003] Recently, with the continuous development of deep learning, image super-resolution models based on convolutional neural networks have been proposed. However, as the complexity of convolutional networks continues to increase, the huge model size and computational complexity also limit the application of image super-resolution tasks on mobile devices.
[0004] Therefore, how to effectively reduce the model size and computational complexity of super-resolution reconstruction networks is a key research issue at present. Summary of the Invention
[0005] To address the aforementioned problems in the existing technology, this invention provides an image super-resolution method based on local mean binary quantization. The technical problem to be solved by this invention is achieved through the following technical solution:
[0006] This invention provides an image super-resolution method based on local mean binary quantization, the image super-resolution method comprising:
[0007] Obtain the low-resolution image to be reconstructed;
[0008] The low-resolution image to be reconstructed is input into a trained super-resolution network to obtain the final high-resolution image. The super-resolution network includes a feature extraction module, a feature transformation module, and an image reconstruction module. The weights and activations of the feature transformation module are both binary quantized.
[0009] In one embodiment of the present invention, the feature extraction module includes a convolutional layer, the feature transformation module includes a plurality of stacked residual blocks, and the image reconstruction module includes a pixel shuffle.
[0010] In one embodiment of the present invention, the binary quantization process of the weights of the feature transformation module is represented as follows:
[0011]
[0012] Among them, w B The weights are quantized, sign(*) is the quantization function, and w is the weights to be quantized.
[0013] In an embodiment of the present application, the binary quantization process of the activation X of the feature transformation module is represented as:
[0014]
[0015] wherein, is the local mean map, is composed of the local mean of all elements, is the average value of the pixels in the preset region.
[0016] In an embodiment of the present application, the quantization process of each element of the activation X is represented as:
[0017]
[0018] wherein, represents the binary quantization of is the pixel value of the element at position (i, j) in the kth channel, 1≤k≤c, and there are c channels in total.
[0019] In an embodiment of the present application, the local mean is calculated in the following manner:
[0020]
[0021] wherein, is the local mean of the elements in all channels within the range of n*n around the element is the calculation process of the local mean .
[0022] In an embodiment of the present application, the training process of the super-resolution network comprises:
[0023] S1, obtaining a training data set, the training data set comprising a plurality of low-resolution training images and high-resolution training images, the low-resolution training images and the high-resolution training images corresponding one by one;
[0024] S2, inputting the low-resolution training images into the super-resolution network to obtain high-resolution output images;
[0025] S3, calculating the gap between the high-resolution output images and the corresponding high-resolution training images, taking the gap as a loss value to perform back propagation on the super-resolution network to optimize the parameters of the super-resolution network;
[0026] S4, repeatedly performing steps S2 to S3 until a preset condition is reached, stopping the training of the super-resolution network, and obtaining a trained super-resolution network.
[0027] In an embodiment of the present application, the S1 comprises:
[0028] S1.1, obtaining a plurality of high-resolution training images;
[0029] S1.2, performing down-sampling processing on each of the high-resolution training images to obtain low-resolution training images, all of the low-resolution training images and the high-resolution training images forming the training data set.
[0030] In an embodiment of the present application, the gap is an average value of the difference values of all corresponding pixel points of the high-resolution output image and the high-resolution training image.
[0031] Compared with the prior art, the present application has the following beneficial effects:
[0032] Firstly, compared with the full-precision super-resolution network model, the binary network proposed in the present application can greatly reduce the calculation amount and parameter amount of the super-resolution model, which is beneficial to the deployment and real-time application of the super-resolution network on edge devices.
[0033] Secondly, compared with the existing binary quantization network, the quantization method based on local mean proposed in the present application can more effectively focus on the detail information of the local area of the feature map, and can more fully retain the texture, color and other information in the full-precision activation in the forward quantization process, thereby reducing the information loss in the quantization process and being more beneficial to the image super-resolution task, and greatly improving the final super-resolution effect.
[0034] Other aspects and features of the present application will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the application. It should be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the application, for which reference should be made to the appended claims. It should be further understood that the drawings are not necessarily drawn to scale and that, unless otherwise specified, they are merely intended to conceptually illustrate the structures and procedures described herein. BRIEF DESCRIPTION OF DRAWINGS
[0035] Figure 1 A flowchart of an image super-resolution method based on local mean binary quantization provided by an embodiment of the present application;
[0036] Figure 2 A process diagram of local mean quantization provided by an embodiment of the present application;
[0037] Figure 3 A visual evaluation diagram based on a VDSR model provided by an embodiment of the present application;
[0038] Figure 4 This is a schematic diagram comparing conventional quantization activation and local mean quantization activation, provided as an embodiment of the present invention.
[0039] Figure 5 This is a schematic diagram of a binary activation distribution provided in an embodiment of the present invention. Detailed Implementation
[0040] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0041] Example 1
[0042] To reduce quantization errors during binary quantization, this technique proposes a bit accumulation mechanism in the quantization process of the super-resolution reconstruction network. This mechanism estimates multi-bit values by accumulating 1-bit values, thereby effectively reducing quantization errors. Through this bit accumulation mechanism, the super-resolution network can effectively reduce quantization errors generated during weight quantization and activation quantization.
[0043] Since currently used binary networks are mainly applied to classification tasks, existing binary super-resolution reconstruction networks do not fully consider the special characteristics of super-resolution tasks during the quantization process. They use a structure similar to that used in binary classification tasks for binary quantization, resulting in excessive information loss during quantization, especially during activation quantization, particularly the loss of texture details, which will greatly affect the effect of super-resolution reconstruction.
[0044] For the reasons stated above, this invention provides an image super-resolution method based on local mean binary quantization. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating an image super-resolution method based on local mean binary quantization (LMB). The present invention provides an image super-resolution method based on LMB, which includes steps 1-2, wherein:
[0045] Step 1: Obtain the low-resolution image to be reconstructed.
[0046] Specifically, the low-resolution image to be reconstructed is the image that needs to be reconstructed into a high-resolution image.
[0047] Step 2: Input the low-resolution image to be reconstructed into the trained super-resolution network to obtain the final high-resolution image. The super-resolution network includes a feature extraction module, a feature transformation module, and an image reconstruction module. The weights and activations of the feature transformation module are both binary quantized.
[0048] Specifically, the low-resolution image to be reconstructed is input into the trained super-resolution network, the network first extracts features of the low-resolution image to be reconstructed through the feature extraction module to generate a set of feature maps, then the feature maps are transmitted to the feature conversion module, the low-resolution feature maps are converted through the feature conversion module to generate high-resolution feature maps, and finally the high-resolution feature maps are up-sampled to different scales according to the pre-set scale factor through the image reconstruction module to be converted into the corresponding high-resolution image. Thus, after obtaining the trained super-resolution network, the low-resolution image to be reconstructed is input into the trained super-resolution network to obtain the corresponding high-resolution image. According to the needs, different scales of super-resolution such as x2, x3, x4, etc. can be realized. In the embodiment, the weights and activations of the feature conversion module are both subjected to binary quantization processing, the binary network can greatly reduce the calculation amount and parameter amount of the super-resolution network, and is beneficial to the deployment and real-time application of the super-resolution network on the edge device.
[0049] In the embodiment, the feature extraction module includes a convolution layer, the feature conversion module includes a plurality of stacked residual blocks, all the residual blocks are connected in sequence, and the image reconstruction module includes a pixelshuffle (pixel recombination module). That is, a layer of convolution layer is used to extract features of the input image, the low-resolution features are converted through the connection of the residual blocks to generate high-resolution features, and the pixelshuffle is used for up-sampling processing, so that different scale up-sampling effects such as x2, x3, x4, etc. can be realized.
[0050] In the embodiment, the weights and activations (i.e. feature maps) of the feature conversion module are both subjected to binary quantization processing, and the binary quantization process of the weights of the feature conversion module is represented as:
[0051]
[0052] wherein w B is the quantized weight, sign(*) is the quantization function, and w is the weight to be quantized.
[0053] Unlike the method of using a fixed threshold for quantization in the conventional binary method, the embodiment uses a spatial level multi-threshold to quantize the full-precision activation. For each element in the full-precision activation, the embodiment determines the quantization threshold according to the surrounding element information, so that each element fully considers the surrounding local information in the quantization process, reducing the information loss in the activation quantization process. For a full-precision activation X, the embodiment defines the element at position (i, j) in the kth channel (feature channel) of X as Therefore, the local mean value of the element can be calculated as follows:
[0054]
[0055] wherein, is the local mean value of the element in the center of the n*n range of the element, is the local mean value of the element in the center of the n*n range of the element, i.e. the pixel mean value of all elements in the n*n range of the element is calculated, and LM(*) is the local mean value the calculation process of the local mean value.
[0056] Therefore, in the quantization process, for each element in the full-precision activation, the embodiment uses the local mean value thereof as the quantization threshold of the element to quantize the element, and the process can be represented as:
[0057]
[0058] wherein, represents the binary quantization of the is the pixel value of the element at position (i, j) in the kth channel, 1≤k≤c, and there are c channels in total.
[0059] For the overall full-precision activation X, the quantization process can be represented as:
[0060]
[0061] wherein, is the local mean value map, is composed of the local mean values of all elements, i.e. the value of the element at each position is the local mean value of the element at the corresponding position of X, and the local mean value is the average value of the pixels in the preset region.
[0062] Therefore, in the local mean value quantization process, the quantization threshold of the element depends on the local peripheral information thereof, and this method can effectively increase the information density in the binary activation.
[0063] Figure 2 The local mean value quantization method of the embodiment is shown, (a) represents an input picture, (b) represents a full-precision activation in a network, (c) represents a conventional binary activation, and (d) represents a local mean value binary activation. It can be seen that, compared with the conventional binary activation method, the local mean value binary activation method quantization produces binary activation containing more texture information and detail information.
[0064] It is worth noting that, for the value of the n*n range used in the local mean value calculation process, considering the influence of the final experimental results and the training speed, the embodiment considers the local information in the 5*5 pixel range when actually calculating the local mean value, but this is not specifically limited, and those skilled in the art can select different ranges according to actual needs.
[0065] In one specific embodiment, the training process of the super-resolution network comprises:
[0066] S1, obtaining a training data set, the training data set comprising a plurality of low-resolution training images and high-resolution training images, the low-resolution training images and the high-resolution training images corresponding one-to-one.
[0067] S1.1, obtaining a plurality of high-resolution training images.
[0068] S1.2, performing down-sampling processing on each high-resolution training image to obtain a low-resolution training image, all low-resolution training images and high-resolution training images forming the training data set.
[0069] Specifically, the high-resolution training images are down-sampled to different scales according to the super-resolution scale, and the down-sampling multiple and the model super-resolution scale factor are both s, generally s = 2, 3 or 4. That is, for a low-resolution image, its height and width are H and W respectively, and after the super-resolution network, the length and width of the corresponding high-resolution image are sH and sW respectively.
[0070] S2, inputting the low-resolution training image into the super-resolution network to obtain a high-resolution output image.
[0071] Specifically, the low-resolution training image is input into the super-resolution network. The super-resolution network first extracts features from the low-resolution training image through a convolutional layer to generate a set of features, then transmits these feature maps to a feature conversion module, converts the low-resolution features through the connection of residual blocks to generate high-resolution features, and finally through an image reconstruction module, i.e. an up-sampling module, the features are up-sampled to different scales according to the pre-set scale factor, and at the same time converted into the corresponding high-resolution output image.
[0072] S3, calculating the difference between the high-resolution output image and the corresponding high-resolution training image, taking the difference as a loss value to back-propagate the super-resolution network to optimize the parameters of the super-resolution network.
[0073] Further, the difference is the average value of the difference of all corresponding pixel points between the high-resolution output image and the high-resolution training image.
[0074] S4, repeatedly performing steps S2 to S3 until a pre-set condition is reached, stopping the training of the super-resolution network, and obtaining a trained super-resolution network.
[0075] Specifically, steps S2 to S3 are repeatedly executed until the observed loss function of the training tends to a stable state, or the iteration number of the model reaches a pre-set iteration upper limit, the model stops training, and the network structure and corresponding model parameters are saved to obtain the trained super-resolution network.
[0076] Firstly, compared with the full-precision super-resolution network model, the binary network proposed in the application can greatly reduce the calculation amount and parameter amount of the super-resolution model, which is beneficial to the deployment and real-time application of the super-resolution network on edge devices.
[0077] Secondly, compared with the existing binary quantization network, the quantization method based on local mean proposed in the application can more effectively focus on the detail information of the local area of the feature map, and can more fully retain the texture, color and other information in the full-precision activation in the forward quantization process, reduce the information loss in the quantization process, and is more beneficial to the image super-resolution task, and greatly improves the final super-resolution effect.
[0078] The effect of the application can be further illustrated by the following simulation experiment.
[0079] 1. Simulation conditions
[0080] The application is programmed on an Inter(R) Core(TM) i7-4790 3.60GHz CPU, a NVIDIA 2080Ti GPU, an Ubuntu 16.04 operating system, a Python programming language, a Pytorch deep learning network framework, and a DIV2K data set, wherein one kind contains 900 images, 800 of which are used as a training set, and the other 100 are used as a verification set. The test data includes: the Set5 data set disclosed by Bevilacqua et al. of the University of Biu, the Set14 data set disclosed by Zeyde et al. of the Technion-Israel Institute of Technology, the BSD100 data set disclosed by Martin et al. of the University of California, Berkeley, and the Urban100 data set disclosed by Huang et al. of the University of Illinois.
[0081] In order to evaluate the performance of the method objectively and fairly, two kinds of existing super-resolution models which have been widely recognized are used for experimental evaluation.
[0082] One is a deep learning method based on residual learning, which is recorded as VDSR in the simulation experiment of the application; and the other is a method based on a residual module and an up-sampling module, which is recorded as SRResNet in the simulation experiment of the application.
[0083] In order to verify that the application has good effect under different super-resolution scale factors, three different scale factors x2, x3 and x4 are used in the simulation to experiment the image super-resolution effect of the model.
[0084] 2. Simulation content
[0085] From Table 1, Table 2, Figure 3 It can be seen that the binarization algorithm of the application is obviously better than other model quantization algorithms in objective evaluation indicators (PSNR: peak signal-to-noise ratio and SSIM: structural similarity) and visual quality evaluation, and has great advantages in image super-resolution reconstruction tasks.
[0086] Table 1 Performance comparison of binarization algorithm based on VDSR
[0087]
[0088]
[0089] Table 2 Performance comparison of binarization algorithm based on SRResNet
[0090]
[0091] In order to further illustrate that the method of the application can retain more information than the conventional quantization method, the application shows the binary activation maps generated by the local mean quantization method and the conventional quantization method. As shown in Figure 3 (a) is the binary activation map generated by the conventional quantization method, (b) is the binary activation map generated by the local mean method, and it can be seen that compared with the conventional quantization method, the local mean quantization method can retain more texture detail information in the place with more complex texture.
[0092] In addition, in order to more objectively show that the local mean binarization can retain more information than the conventional binarization method, the application uses two-dimensional information entropy to measure the information density retained in the activation, and in this experiment, the application calculates the information entropy of the activation of the conventional binary quantization and the activation of the local mean binary quantization, Figure 5 shows the distribution of binary activation data generated by different quantization methods, (a) is the conventional binary quantization method, and (b) is the local mean binary quantization method. It can be seen that compared with the conventional binary quantization method, the distribution of binary activation data generated by the local mean binary quantization method is more balanced, and the information entropy of 3.42 is obviously higher than that of the conventional binary quantization method of 2.62, which shows that the local mean binarization can effectively reduce the information loss in the quantization process. Further verify the effectiveness and advancement of the application.
[0093] In the description of the application, the terms "first", "second", "third", etc. are used only to describe different embodiments and do not imply or suggest relative importance or a specific number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.
[0094] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or specific data points described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or specific data points described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present application.
[0095] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the present application.
Claims
1. A method for image super-resolution based on local mean binary quantization, characterized in that, The image super-resolution method comprises: acquiring a low-resolution image to be reconstructed; inputting the low-resolution image to be reconstructed into a trained super-resolution network to obtain a final high-resolution image, wherein the super-resolution network comprises a feature extraction module, a feature conversion module, and an image reconstruction module, and the weight and activation of the feature conversion module are both subjected to binary quantization processing; wherein the feature extraction module comprises a convolution layer, the feature conversion module comprises a plurality of stacked residual blocks, and the image reconstruction module comprises pixelshuffle; the binary quantization process of the activation X of the feature conversion module is represented as: wherein, is a local mean map, consisting of local means of all elements, said local means are average values of pixels of a predetermined area; The local mean is calculated as follows: wherein, is the element is centered, the perimeter local mean of all elements in the range, LM(*) is the local mean calculation process.
2. The method of claim 1, wherein, the binary quantization process of the weight of the feature conversion module is represented as: wherein, is the quantized weight, is the quantization function, is the weight to be quantized.
3. The method of claim 1, wherein, the quantization process of each element of the activation X is represented as: wherein, representing a binary quantization , is the first pixel value of an element in position of the i-th channel, 1 ≤ ≤ c , i = 1, 2,..., N, where N is the total c number of channels.
4. The method of claim 1, wherein, the training process of the super-resolution network comprises: S1, acquiring a training data set comprising a plurality of low-resolution training images and high-resolution training images, wherein the low-resolution training images and the high-resolution training images correspond one by one; S2, inputting the low-resolution training image into the super-resolution network to obtain a high-resolution output image; S3, calculating the difference between the high-resolution output image and the corresponding high-resolution training image, taking the difference as a loss value, and performing back propagation on the super-resolution network to optimize the parameters of the super-resolution network; S4, repeatedly performing steps S2 to S3 until a preset condition is reached, stopping the training of the super-resolution network, and obtaining a trained super-resolution network.
5. The method of claim 4, wherein, The S1 comprises: S1.1, acquiring a plurality of high-resolution training images; S1.2, performing down-sampling processing on each high-resolution training image to obtain a low-resolution training image, and all the low-resolution training images and the high-resolution training images constitute the training data set.
6. The method of claim 4, wherein, The difference is the average value of the difference values of all corresponding pixel points of the high-resolution output image and the high-resolution training image.