An Image Super-Resolution Reconstruction Method Based on Network Binary Inference Acceleration

Abstract

The present invention relates to an image super-resolution reconstruction method based on network binary inference acceleration, comprising: obtaining a low-resolution image to be reconstructed; inputting the low-resolution image to be reconstructed into a trained image super-resolution reconstruction model to obtain a final high-resolution image, wherein the image super-resolution reconstruction model includes a first convolutional layer, a plurality of stacked binary residual blocks, and a second convolutional layer connected in sequence, and the binary residual block is obtained by performing binary quantization processing on both weights and activations. The present invention first uses multiple corrected thresholds to obtain binary activation representations from multiple perspectives, which can obtain more accurate binary activation expressions and significantly reduce the quantization error caused by the binary process. Subsequently, the convolution operation of the multi-bit network is simulated, and the 1-bit network is used to simulate the multi-bit network to improve the information-carrying capacity of the binary network, thereby improving the recovery of high-frequency detail information by the binary super-resolution network.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence and image processing, and relates to an image super-resolution reconstruction method based on network binary inference acceleration. Background Art

[0002] Image super-resolution reconstruction aims to reconstruct a high-resolution image from a low-resolution image, and has a very broad application space in many fields such as video surveillance and medical imaging. In recent years, thanks to the rapid development of convolutional neural networks, image super-resolution reconstruction algorithms based on deep learning have achieved great success. However, with the continuous improvement of network performance, the network structure has become more complex, thus bringing extremely high storage costs and computational requirements. Although existing super-resolution models have excellent performance, it is difficult to achieve real-time processing, and they rely on high-performance computing devices such as Graphics Processing Units (GPUs) or Tensor Processing Units (TPUs), which severely limits their application and promotion in portable mobile devices. Therefore, image super-resolution reconstruction models need to effectively reduce the consumption of storage and computing resources to meet the needs of devices with limited existing resources.

[0003] Binary neural networks limit the weights and activations in the network to -1 and +1, and at the same time convert the multiplication operations in the network into bit operations, enjoying extremely high model compression ratios and speed gain ratios, and can meet the computational requirements of the super-resolution process for portable mobile devices such as mobile phones. Therefore, the research on image super-resolution reconstruction networks based on binary neural networks is of great significance.

[0004] However, existing methods usually use a standard quantization process to obtain binary activations, which will cause large quantization errors when the feature distribution is significantly uneven. In addition, the network activations in the super-resolution task contain rich image information (colors, textures, etc.), while the information-carrying capacity of binary neural networks is limited, seriously affecting the recovery of high-frequency detail information of the reconstructed image. Summary of the Invention

[0005] To solve the above problems existing in the prior art, the present invention provides an image super-resolution reconstruction method based on network binary inference acceleration. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] An embodiment of the present invention provides an image super-resolution method based on local mean binary quantization. The image super-resolution reconstruction method includes:

[0007] Obtain a low-resolution image to be reconstructed;

[0008] Input the low-resolution image to be reconstructed into the trained image super-resolution reconstruction model to obtain the final high-resolution image. Among them, the image super-resolution reconstruction model includes a first convolutional layer, multiple stacked binary residual blocks, and a second convolutional layer connected in sequence. The binary residual block is obtained from a residual block in which both the weights and activations are binary quantized.

[0009] In an embodiment of the present invention, both the first convolutional layer and the second convolutional layer are 3×3 convolutional layers, and the total number of the binary residual blocks is 9.

[0010] In an embodiment of the present invention, the binary quantization process of the weights of the binary residual module is expressed as:

[0011]

[0012]

[0013] where W b represents the binary weight, Sign represents the quantization function, W represents the real-valued weight, τ represents the weight scale factor, n represents the number of channels of the weight, and ||·|| l1 represents the l1 norm.

[0014] In an embodiment of the present invention, the binary quantization process of the activations of the binary residual module includes:

[0015] S1.1. Decompose the activation a of the residual block into a first subset a1 and a second subset a2 along the channel direction;

[0016] S1.2. Perform binary convolution based on high-order bit approximation on the first subset a1 and the second subset a2 respectively, and correspondingly obtain a first eigenvalue Z1 and a second eigenvalue Z2;

[0017] S1.3. Perform a Concat operation on the first eigenvalue Z1 and the second eigenvalue Z2 along the channel direction, and then pass through a PReLU activation layer to obtain the final activation value.

[0018] In an embodiment of the present invention, the first eigenvalue Z1 is expressed as:

[0019]

[0020] The second eigenvalue Z2 is expressed as:

[0021]

[0022] where W b represents the binary weight, XNOR operation is denoted as, multiplication operation is denoted as ⊙, γ1 is the scale factor after multiplying τ and β1, γ2 is the scale factor after multiplying τ and β2, τ represents the weight scale factor, and both β1 and β2 represent learnable scale factors. represents the binary activation of the first subset a1 quantized based on -α1. represents the binary activation of the first subset a1 quantized based on α2. represents the binary activation of the second subset a2 quantized based on -α1. represents the binary activation of the second subset a2 quantized based on α2, where α1 and α2 represent quantization thresholds.

[0023] In one embodiment of the present invention, the is represented as:

[0024]

[0025] The is represented as:

[0026]

[0027] The is represented as:

[0028]

[0029] The is represented as:

[0030]

[0031] where Sign represents the quantization function.

[0032] In one embodiment of the present invention, the training process of the image super - resolution reconstruction model includes:

[0033] S1. Obtain a training data set, where the training data set includes a number of low - resolution training images and high - resolution training images, and the low - resolution training images and the high - resolution training images are in one - to - one correspondence;

[0034] S2. Crop an image patch of a preset size centered at a random position of the low - resolution training image;

[0035] S3. Input the image patch into the image super - resolution reconstruction model to obtain a high - resolution output image patch;

[0036] S4. Update the weights of the image super - resolution reconstruction model according to the gradient estimation method and the loss function;

[0037] S5. Repeat steps S2 to S4 until a preset condition is reached, and stop the training of the image super-resolution reconstruction model to obtain a trained image super-resolution reconstruction model.

[0038] In one embodiment of the present invention, the gradient estimation method includes activation gradient estimation and weight gradient estimation.

[0039] In one embodiment of the present invention, the activation gradient estimation is expressed as:

[0040]

[0041]

[0042] The weight gradient estimation is expressed as:

[0043]

[0044] where L represents the loss function, x represents the real-valued activation, x b represents the binary activation, w represents the real-valued weight, w b represents the binary weight, and Approx represents the approximate Sign function.

[0045] In one embodiment of the present invention, the loss function is expressed as:

[0046]

[0047] where M represents the number of image patches, α represents the learning rate of the image super-resolution reconstruction model, y represents the high-resolution image patch corresponding to the image patch, represents the image patch reconstructed by the image super-resolution reconstruction model.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] Since the present invention adopts a binary convolution process based on high-order bit approximation, first, multiple corrected thresholds are used to obtain binary activation representations from multiple perspectives, which can obtain more accurate binary activation expressions and significantly reduce the quantization error caused by the binarization process. Subsequently, the convolution operation of the multi-bit network is simulated, and the 1-bit network is used to simulate the multi-bit network, which improves the information-carrying capacity of the binary network and further improves the recovery of high-frequency detail information by the binary super-resolution network.

[0050] Through the following detailed description with reference to the accompanying drawings, other aspects and features of the present invention become apparent. However, it should be understood that the drawings are designed only for the purpose of explanation and not as a limitation of the scope of the present invention, as it should be referred to the appended claims. It should also be understood that, unless otherwise indicated, the drawings are not necessarily drawn to scale, and they are only intended to conceptually illustrate the structures and processes described herein. Description of the Drawings

[0051] Figure 1 Schematic flowchart of an image super-resolution reconstruction method based on network binary inference acceleration provided for an embodiment of the present invention;

[0052] Figure 2 Schematic diagram for comparing the visual quality between the method of the present invention and the BAM method provided for an embodiment of the present invention;

[0053] Figure 3 Schematic diagram for comparing the visual quality between the method of the present invention and the BTM method provided for an embodiment of the present invention. Detailed Embodiments

[0054] The present invention will be further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0055] Embodiment 1

[0056] Please refer to Figure 1 , Figure 1 Schematic flowchart of an image super-resolution reconstruction method based on network binary inference acceleration provided for an embodiment of the present invention. The present invention provides an image super-resolution reconstruction method based on network binary inference acceleration. The image super-resolution reconstruction method includes Step 1 - Step 2, where:

[0057] Step 1: Obtain a low-resolution image to be reconstructed.

[0058] Specifically, the low-resolution image to be reconstructed is an image that needs to be reconstructed into a high-resolution image.

[0059] Step 2: Input the low-resolution image to be reconstructed into a trained image super-resolution reconstruction model to obtain a final high-resolution image. The image super-resolution reconstruction model includes a first convolutional layer, a plurality of stacked binary residual blocks, and a second convolutional layer connected in sequence. The binary residual block is obtained by performing binary quantization processing on both the weights and activations of the residual block.

[0060] Specifically, in this embodiment, the low-resolution image to be reconstructed is input into the trained image super-resolution reconstruction model. The image super-resolution reconstruction model first extracts features from the low-resolution image to be reconstructed through the first convolutional layer, and then transmits these features to multiple stacked binary residual blocks, which transform them to generate a high-resolution feature map. Finally, the high-resolution feature map is reconstructed through the second convolutional layer to obtain the corresponding high-resolution image. Thus, after obtaining the trained image super-resolution reconstruction model, inputting the low-resolution image to be reconstructed into the trained image super-resolution reconstruction model can obtain the corresponding high-resolution image. Super-resolution at different scales such as x2, x3, and x4 can be achieved as needed. In this embodiment, binary quantization processing is performed on the weights and activations of each of the multiple stacked residual blocks.

[0061] Preferably, both the first convolutional layer and the second convolutional layer are 3×3 convolutional layers, both the first convolutional layer and the second convolutional layer are full-precision convolutional layers, the total number of binary residual blocks is 9, and the 9 binary residual blocks are connected in sequence. The first convolutional layer is used for extracting shallow features of the image, and the second convolutional layer is used for image reconstruction, that is, using a first convolutional layer to extract features from the input image, transforming the low-resolution features through the connection of binary residual blocks to generate high-resolution features, and performing image reconstruction through the second convolutional layer.

[0062] In this embodiment, the weights of the binary residual module are quantized using a binarization function (sign function), that is:

[0063]

[0064]

[0065] where W b represents the binary weight, Sign represents the quantization function, W represents the real-valued weight, that is, the full-precision weight, τ represents the weight scale factor, n represents the number of channels of the weight, and ||·|| l1 represents l1 regularization.

[0066] In addition, the activation values are quantized using a binarization function based on different quantization thresholds:

[0067]

[0068]

[0069] where a represents the input full-precision activation, represents the binary activation quantized based on -α1, The binary activation after α2 quantization is shown, where α1 and α2 represent quantization thresholds. Based on different quantization thresholds, two sign functions are combined into a ternary quantization function to approximate the high-order bit activation, i.e.:

[0070]

[0071] where β is a learnable scale factor that adjusts the quantization interval of the sign function.

[0072] Finally, the binary convolution based on high-order bit approximation is implemented:

[0073]

[0074] where Z represents the eigenvalue after binary convolution, represents the XNOR operation. XNOR is the exclusive NOR operation, and the output is 1 if the two inputs are the same and 0 if they are different. ⊙ represents the multiplication operation. γ1 and γ2 are scale factors after multiplying τ and β.

[0075] Therefore, the binary quantization process for the activation value input to each residual block includes:

[0076] S1.1. Decompose the activation a of the residual block into a first subset a1 and a second subset a2 along the channel direction.

[0077] Specifically, the activation a of the residual block is evenly decomposed along the channel dimension to obtain the first subset a1 and the second subset a2.

[0078] S1.2. Perform binary convolutions based on high-order bit approximation on the first subset a1 and the second subset a2 respectively, and correspondingly obtain a first eigenvalue Z1 and a second eigenvalue Z2.

[0079] Specifically, the first eigenvalue Z1 is expressed as:

[0080]

[0081] The second eigenvalue Z2 is expressed as:

[0082]

[0083] where W b represents the binary weight, represents the XNOR operation, ⊙ represents the multiplication operation, γ1 is the scale factor after multiplying τ and β1, γ2 is the scale factor after multiplying τ and β2, τ represents the weight scale factor, and both β1 and β2 represent learnable scale factors, represents the binary activation of the first subset a1 after quantization based on -α1, represents the binary activation of the first subset a1 after quantization based on α2, Indicates the binary activation of the second subset a2 after quantization based on -α1, Indicates the binary activation of the second subset a2 after quantization based on α2, where α1 and α2 represent quantization thresholds.

[0084] Furthermore, It is expressed as:

[0085]

[0086] It is expressed as:

[0087]

[0088] It is expressed as:

[0089]

[0090] It is expressed as:

[0091]

[0092] where Sign represents the quantization function.

[0093] S1.3. Perform a Concat (concatenation) operation on the first eigenvalue Z1 and the second eigenvalue Z2 along the channel direction, and then pass through a PReLU (Parametric Rectified Linear Unit) activation layer to obtain the final activation value. Each binary convolutional residual block completes the above process twice.

[0094] In a specific embodiment, the training process of the image super-resolution reconstruction model includes:

[0095] S1. Obtain a training dataset, which includes a number of low-resolution training images and high-resolution training images, and the low-resolution training images and high-resolution training images correspond one by one.

[0096] Specifically, rotate each high-resolution training image, specifically rotate it by 90 degrees, 180 degrees, and 270 degrees, and use the rotated images as high-resolution training images for data augmentation. Subsequently, perform downsampling on each high-resolution training image to construct low-resolution and high-resolution image pairs, specifically perform 2-fold, 3-fold, and 4-fold downsampling.

[0097] S2. Crop an image patch of a preset size centered at a random position of the low-resolution training image.

[0098] Specifically, in each iteration, the image super-resolution reconstruction model randomly selects a low-resolution training image and crops an image patch of a preset size centered at a random position of the low-resolution training image as the input image. For example, the preset size is 48×48. For different magnification tasks, the high-resolution training image crops an image patch of the same area at a proportional position, that is, the cropped low-resolution training image patch and the high-resolution training image patch are image patches of the same content. For example, image patches of sizes 96×96, 144×144, and 196×196 are cropped from the high-resolution training image according to the magnification factor.

[0099] S3. Input the image patch into the image super-resolution reconstruction model to obtain a high-resolution output image patch.

[0100] S4. Update the weights of the image super-resolution reconstruction model according to the gradient estimation method and the loss function.

[0101] Specifically, the image super-resolution reconstruction model inputs the processed low-resolution image patch. The image super-resolution reconstruction model processes the low-resolution image patch to obtain a reconstructed high-resolution output image patch. Subsequently, the loss function is set, and the gradient estimation method is used to optimize the image super-resolution reconstruction model to guide the weight update in the network.

[0102] Furthermore, the gradient estimation method includes activation gradient estimation and weight gradient estimation.

[0103] Since the binary function (sign function) is not differentiable, a straight-through estimator and a higher-order binary function are introduced to approximate the gradient of the sign function to optimize the image super-resolution reconstruction model. The derivatives of the straight-through estimator and the higher-order binary function are respectively:

[0104]

[0105]

[0106] At this time, for the image super-resolution reconstruction model, the activation gradient estimation is expressed as:

[0107]

[0108]

[0109] The weight gradient estimation is expressed as:

[0110]

[0111] Among them, L represents the loss function, x represents the real-valued activation, that is, the activation that needs to be binarized, x bIndicates binary activation, w represents real-valued weights, that is, the weights to be binarized, w b Indicates binary weights, Approx represents the approximate Sign function, and Approx is the abbreviation of Approximate.

[0112] In addition, the loss function of the image super-resolution reconstruction model is used to measure the difference between the generated image and the real target image, and the pixel loss is the most common loss. In this embodiment, L1 Loss is used as the distance metric function. The loss function of this embodiment is as follows:

[0113]

[0114] Among them, M represents the number of image patches, α represents the learning rate of the image super-resolution reconstruction model, y represents the high-resolution training image patch corresponding to the image patch, represents the image patch reconstructed by the image super-resolution reconstruction model, that is, the high-resolution output image patch.

[0115] S5. Repeatedly execute steps S2 to S4 until a preset condition is reached, stop the training of the image super-resolution reconstruction model, and obtain the trained image super-resolution reconstruction model.

[0116] Specifically, repeatedly execute steps S2 to S4 until the number of iterations of the image super-resolution reconstruction model exceeds the preset iteration value to achieve model convergence, stop the training, and save the network structure and parameters to obtain the trained network model.

[0117] In addition, test and save the output result. Input the low-resolution test set image into the trained network model, and the output result of the network is the predicted high-resolution image.

[0118] Since the present invention adopts a binary convolution process based on high-order bit approximation, first, multiple corrected thresholds are used to obtain binary activation representations from multiple perspectives, which can obtain more accurate binary activation expressions and significantly reduce the quantization error caused by the binarization process. Subsequently, the convolution operation of the multi-bit network is simulated, and the 1-bit network is used to simulate the multi-bit network to improve the information-carrying capacity of the binary network, thereby improving the recovery of high-frequency detail information by the binary super-resolution network.

[0119] The effect of the present invention can be further illustrated by the following simulation experiments.

[0120] 1. Simulation conditions

[0121] The present invention is based on a central processing unit of Inter(R)Core(TM)i7-4790 3.60GHz CPU, NVIDIATitan XP GPU, and Ubuntu 16.04 operating system. The relevant experimental environment is as follows:

[0122] Python=3.7, Pytorch=1.2.0, numpy, skimage, imageio, matplotlib, tdqm, cv2.

[0123] The database uses the DIV2K dataset, which is widely used in the field of super-resolution. The methods compared in the experiment are as follows:

[0124] The first is a binary image super-resolution reconstruction algorithm based on a bit accumulation mechanism, denoted as BAM in the experiment. BAM proposes a bit accumulation mechanism, which estimates multi-bit values ​​by accumulating multiple 1-bit values, and gradually improves the performance of the quantization network along the model reasoning direction. The second is a method based on high-precision binary network training.

[0125] 2. Simulation content

[0126] Experiment: Evaluation of Reconstructed Image Quality

[0127] According to the specific implementation method of the present invention and the binary image super-resolution reconstruction algorithm BAM based on the bit accumulation mechanism and the high-precision binary network training method BTM, the image super-resolution reconstruction network is trained on the public high-definition image dataset DIV2K published by NTIRE2017. The test selected standard test datasets Set5, Set14, Urban100 and BSD100. The experiment selected standard evaluation indicators: image peak signal-to-noise ratio PSNR (peak signal-to-noise ratio, PSNR) and structural similarity SSIM (Structural Similarity Image Metric, SSIM). The larger the values ​​of PSNR and SSIM, the better the reconstructed image quality. The experimental results are shown in Table 1. Bicubic represents the result of image enlargement by bicubic interpolation, and scale represents different magnifications, which are 2x, 3x and 4x methods respectively. From the table, it can be seen that the present invention is significantly superior to the current binary image super-resolution reconstruction algorithm in terms of PSNR and SSIM. Figure 2 , Figure 3 The visual restoration effect of the present invention on the reconstructed image compared with other methods is demonstrated. It can be seen that the image reconstructed by the present invention method has a higher resolution and the synthesized image is clearer, which further verifies the advancement of the present invention.

[0128] Table 1 Performance comparison of different binarization algorithms

[0129]

[0130] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more unless otherwise specifically defined.

[0131] In the description of this specification, the descriptions with reference to terms such as "an embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or specific data points described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or specific data points described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0132] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. An image super-resolution reconstruction method based on network binary inference acceleration, characterized in that The image super-resolution reconstruction method includes: Obtain a low-resolution image to be reconstructed; Input the low-resolution image to be reconstructed into a trained image super-resolution reconstruction model to obtain a final high-resolution image, where the image super-resolution reconstruction model includes a first convolutional layer, multiple stacked binary residual blocks, and a second convolutional layer connected in sequence, and the binary residual block is obtained from a residual block in which both the weights and activations are binary quantized; The binary quantization process of the activation of the binary residual module includes: S1.1: Decompose the activation a of the residual block into a first subset a1 and a second subset a2 along the channel direction; S1.2: Perform binary convolution based on high-order bit approximation on the first subset a1 and the second subset a2 respectively to obtain a first eigenvalue Z1 and a second eigenvalue Z2 correspondingly; S1.3: Perform a Concat operation on the first eigenvalue Z1 and the second eigenvalue Z2 along the channel direction, and then pass through a PReLU activation layer to obtain a final activation value; The first eigenvalue Z1 is expressed as: The second eigenvalue Z2 is expressed as: Among them, W b represents a binary weight, represents an XNOR operation, ⊙ represents a multiplication operation, γ1 is the scaling factor after multiplying τ and β1, γ2 is the scaling factor after multiplying τ and β2, τ represents the weight scaling factor, and both β1 and β2 represent learnable scaling factors, represents the binary activation of the first subset a1 quantized based on -α1, represents the binary activation of the first subset a1 quantized based on α2, represents the binary activation of the second subset α2 quantized based on -α1, represents the binary activation of the second subset α2 quantized based on α2, and α1 and α2 represent quantization thresholds.

2. The image super-resolution reconstruction method according to claim 1, wherein Both the first convolutional layer and the second convolutional layer are 3×3 convolutional layers, and the total number of the binary residual blocks is 9.

3. The image super-resolution reconstruction method according to claim 1, characterized in that, The binary quantization process of the weights of the binary residual module is expressed as: Among them, W b represents the binary weight, Sign represents the quantization function, W represents the real-valued weight, τ represents the weight scale factor, n represents the number of channels of the weight, and ‖·‖ l1 represents l1 regularization.

4. The image super-resolution reconstruction method according to claim 3, characterized in that The said is expressed as: The said is expressed as: The above-mentioned is expressed as: The is expressed as: where Sign represents the quantization function.

5. The image super-resolution reconstruction method according to claim 1, wherein The training process of the image super-resolution reconstruction model includes: S1: Obtain a training data set, where the training data set includes a number of low-resolution training images and high-resolution training images, and the low-resolution training images and the high-resolution training images correspond one by one; S2: Crop an image block of a preset size centered at a random position of the low-resolution training image; S3: Input the image block into the image super-resolution reconstruction model to obtain a high-resolution output image block; S4: Update the weights of the image super-resolution reconstruction model according to the gradient estimation method and the loss function; S5: Repeatedly execute steps S2 to S4 until a preset condition is reached, stop the training of the image super-resolution reconstruction model, and obtain a trained image super-resolution reconstruction model.

6. The image super-resolution reconstruction method according to claim 5, wherein, The gradient estimation method includes the gradient estimation of the activation and the gradient estimation of the weights.

7. The image super-resolution reconstruction method according to claim 6, characterized in that, The gradient estimation of the activation is expressed as: The gradient estimation of the weights is expressed as: Among them, L represents the loss function, x represents the real-valued activation, x b represents the binary activation, w represents the real-valued weight, w b represents the binary weight, and Approx represents the approximate Sign function.

8. The image super-resolution reconstruction method according to claim 7, characterized in that, The loss function is expressed as: Among them, M represents the number of image patches, α represents the learning rate of the image super-resolution reconstruction model, y represents the high-resolution image patch corresponding to the image patch, represents the image patch reconstructed by the image super-resolution reconstruction model.

Images