Up-sampling module circuit based on META-SR algorithm

Abstract

The invention discloses an up-sampling module circuit based on an META-SR algorithm. The up-sampling module circuit has the advantages that hardware implementation resources are few, a low-resolution image with the size smaller than that of a set input image is input in a compatible mode, and up-sampling amplification of any integral multiple can be achieved. The circuit is composed of three basic modules, namely a row buffer module, a weight access module and a convolution module. Compared with a deconvolution layer circuit, the usage amount of a lookup table used as logic, the usage amount of a lookup table used as storage and the usage amount of a trigger are reduced by 85.0%, 31.1% and 34.4% respectively; compatible input of a low-resolution image smaller than a set input image size is realized through 0 complementing operation; according to the up-sampling multiple to be realized, the weight reading unit and the basic convolution unit are added, so that the function of up-sampling of the corresponding multiple can be realized; compared with an up-sampling module based on other algorithms, the method optimizes the imaging effect of the model, and has a wider application range.

Description

Technical Field

[0001] The present invention relates to the field of hardware acceleration of deep learning models in image super-resolution technology, and in particular to an FPGA design implementation of an upsampling module circuit based on META-SR. Background Art

[0002] In recent years, deep learning models have developed rapidly in the field of image super-resolution, especially convolutional neural networks (CNNs), which have greatly improved the imaging effect of super-resolution models. The upsampling modules in different super-resolution models often use different algorithms, such as interpolation, sub-pixel layer, and deconvolution layer. Among them, the interpolation algorithm does not need to be trained, but the imaging effect is poor; the sub-pixel layer and deconvolution layer have better imaging effects, but can only train a single integer multiple upsampling, and the computational complexity is high.

[0003] Due to the complexity of sub-pixel layer and deconvolution layer calculations, a large amount of hardware resources are usually consumed when implementing FPGA design. For example, in his master's thesis "FPGA Design and Implementation of Fast Super-Precision Convolutional Neural Network (FSRCNN)", Li Hanwen used the deconvolution layer to perform 3x upsampling on the super-resolution convolutional neural network FSRCNN and designed and implemented it on FPGA, of which the hardware resources of the deconvolution layer accounted for about 44.7% of the overall design.

[0004] The META-SR algorithm proposed by Xuecai Hu et al. in "Meta-SR: A Magnification-Arbitrary Network for Super-Resolution" published at the CVPR 2019 conference can achieve arbitrary upsampling without repeated training and optimize the imaging effect of the original model. Figure 1 As shown in the figure, according to the calculation process given by Xuecai Hu et al., the META-SR calculation can be divided into three steps: 1) Calculate the horizontal and vertical coordinate offset values ​​of the target high-resolution image pixels after mapping to the low-resolution image. 2) Calculate the convolution kernel weight corresponding to the current high-resolution image pixel based on the horizontal and vertical coordinate offset values ​​and the size factor (target upsampling multiple). 3) Intercept the low-resolution feature map pixels and surrounding pixels (intercept window size is 3*3), use the corresponding convolution kernel weight (convolution kernel size is 3*3) for convolution, and superimpose multiple feature maps to obtain the target high-resolution image pixels.

[0005] According to the calculation steps of the META-SR algorithm, when calculating the convolution kernel weights for 2x upsampling, because the size factor is 2, we only need to pay attention to the horizontal and vertical coordinate offset values. Figure 2As shown, when upsampling by 2 times, the low-resolution image pixel (1, 1) corresponds to 4 high-resolution image pixels (1, 1)(1, 2)(2, 1)(2, 2). After mapping the high-resolution image pixels to the low-resolution image, the coordinates are (0.5, 0.5)(0.5, 1)(1, 0.5)(1, 1), and the horizontal and vertical coordinate offset values ​​are (0.5, 0.5)(0.5, 1)(1, 0.5)(1, 1). Similarly, other high-resolution pixels also correspond to these 4 sets of coordinate offset values. Figure 3 As shown in the figure, the calculation method of the convolution kernel weights shows that in the case of 2x upsampling, there are only 4 sets of relatively independent convolution kernel weights in a feature map. Therefore, the convolution kernel weights calculated in the META-SR algorithm have a periodic property. Applying this property in the design and implementation can significantly reduce the hardware resources used to store the convolution kernel weights. Summary of the invention

[0006] The present invention proposes an upsampling module circuit that can support any integer multiple. The circuit is compatible with low-resolution images that are smaller than the set input image size. When the super-resolution model is deployed in hardware, the use of this circuit can significantly reduce the consumption of hardware resources and optimize the imaging effect of the super-resolution model to a certain extent.

[0007] The technical solution of the present invention is as follows: the upsampling module circuit based on the META-SR algorithm is composed of a line buffer module, a weight reading module and a convolution module.

[0008] The row buffer module implements the "sliding window" operation in the convolution operation, that is, a 3*3 window in the feature map is intercepted in a certain order for convolution. The row buffer module contains a number of row buffers for the feature map channels. The row buffer stores the input feature map data and outputs the required 3*3 window in the feature map during convolution.

[0009] The weight reading module is used to provide the convolution kernel weight in the convolution operation. The weight reading module outputs the corresponding convolution kernel weight according to the feature map channel from which the current convolution data comes.

[0010] The convolution module implements the calculation part of the convolution operation. The convolution module receives the feature map window output by the line buffer module and the convolution kernel weight output by the weight reading module and performs multiplication and addition calculations, and then outputs high-resolution image pixels after cyclic accumulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1Used to illustrate the calculation process of the META-SR algorithm, where r is the size factor, i is the horizontal or vertical coordinate of the high-resolution image pixel, R is the horizontal or vertical coordinate offset value, N is the number of feature map channels, inC is the number of low-resolution image channels, outC is the number of high-resolution image channels, inH and inW are the low-resolution image height and width, H and W are the high-resolution image height and width;

[0012] Figure 2 Used to illustrate the correspondence between pixels and pixels, and between pixels and convolution kernel weights when the META-SR algorithm is upsampling;

[0013] Figure 3 Used to illustrate the specific operation of the META-SR algorithm during convolution;

[0014] Figure 4 It is used to illustrate the circuit module block diagram of the 2, 3, and 4 times upsampling module based on the META-SR algorithm;

[0015] Figure 5 Used to illustrate the basic structure of the line buffer;

[0016] Figure 6 Used to describe the mode of filling and adjusting the row buffer;

[0017] Figure 7 It is used to explain the specific processing of register data when the row buffer is filled and adjusted;

[0018] Figure 8 It is used to illustrate the structure of the basic weight reading unit. Nine groups of shift registers simultaneously output a 3*3 convolution kernel.

[0019] Fig. 9 Used to illustrate the structure of the basic convolution unit;

[0020] Fig.10 It is used to explain the specific operation of padding with zeros when inputting a low-resolution image that is smaller than the set input image size;

[0021] Fig.11 Used to illustrate the PSNR values ​​of different upsampling algorithms on the Set5 test set, where Bicubic is bicubic interpolation, FSRCNN is a super-resolution convolution model based on the deconvolution layer upsampling module, and Meta-FSRCNN is a super-resolution convolution model based on the Meta-SR upsampling module;

[0022] Fig.12 Used to illustrate the comparison of hardware resource usage between the deconvolution layer of FSRCNN and the upsampling module circuit based on META-SR. DETAILED DESCRIPTION

[0023] To further illustrate the technical solutions and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] The up-sampling module circuit of the present invention is as follows Figure 4 As shown, the structure includes a row buffer module, a weight reading module and a convolution module.

[0025] Next, the calculation process of the upsampling module circuit is described based on an example: the input low-resolution image feature map channel number of the upsampling module set to 2 times is 56, the input low-resolution image size is 64*64 (the feature map size is the same as the low-resolution image size), and the input data circulates the same coordinate pixel value of the 56-channel feature map every 56 clock cycles (1 pixel cycle), and the coordinates change from left to right and from top to bottom. After the input data is 56*65 clock cycles, that is, 65 pixel cycles, in the 56 clock cycles from the beginning to the end of the 66th pixel cycle, the line buffer meets the output conditions, and the padded and adjusted data from line buffer 1 to line buffer 56 will be output to the convolution module through the 56-to-1 multiplexer. At the same time, the 2x weight reading unit outputs 56 groups of corresponding convolution kernel weights in sequence (each group of convolution kernel weights contains 4 3*3 convolution kernels), and the 4 basic convolution units in the convolution module receive the corresponding convolution kernels in each cycle. Each basic convolution unit completes one convolution per clock cycle and performs cyclic accumulation. When 56 clock cycles of convolution are completed and cyclic accumulation is performed, one high-resolution image pixel is obtained. At this point, the 66th pixel cycle ends, and the convolution module outputs 4 high-resolution image pixels. The above operation is repeated in the subsequent pixel cycles until the last 4 high-resolution pixels are output.

[0026] The basic structure of a line buffer is as follows Figure 5 As shown, the row buffer consists of 2 groups of shift registers and 9 registers that can read data in parallel. Figure 5 The output values ​​of the nine registers numbered 1, 2, ..., 9 are respectively recorded as R1, R2, ..., R9. The length of the shift register is the set input image size minus the convolution kernel size, which is 61 in this example.

[0027] The output of the row buffer consists of 9 registers that can read data in parallel, which are filled and adjusted. The filling operation means that the output is 0 when there is no data in the register, and the adjustment operation means that the output is 0 when there is data in the register but it is invalid. For example, when the pixel data of the second row and the second column of the input feature map is stored in register No. 9, the pixel data is stored, and the pixel data of the first row and the first column of the input feature map is stored in register No. 5. Figure 60 mode. There is no data in registers 1, 2, 3, and 4, so a filling operation is performed. There is invalid data in register 7, so an adjustment operation is performed. Registers 5, 6, 8, and 9 output normally. Similarly, other filling and adjustment modes are processed according to Figure 6 as well as Figure 7 conduct.

[0028] The basic weight reading unit is as follows Figure 8 As shown in the figure, the basic weight reading unit uses 9 groups of shift registers to store the convolution kernel weights. The length of the shift register is the number of feature map channels. In this example, the length is 56. According to the feature map channel from which the current convolution data comes, the output corresponds to 9 convolution kernel weights, that is, the convolution kernel weights of size 3*3. Figure 2 and Figure 3 As shown in the figure, a 2x weight reading unit consists of 4 basic weight reading units. Similarly, a 3x and 4x weight reading unit consists of 9 and 16 basic weight reading units, respectively.

[0029] The basic convolution unit is Fig. 9 As shown in the figure, the basic convolution unit consists of 9 multipliers, 8 adders and 1 circular accumulator. It accepts the convolution kernel weights and feature map data of size 3*3, performs multiplication and addition operations to obtain the current output, that is, completes one convolution. Then, Fig. 9 The convolution kernel weights and feature map data in the image are switched to the convolution kernel weights and feature map data of another channel, and another convolution calculation is performed. After the feature map channels are switched several times, the feature map channels are convolved several times. In this example, the number of feature map channels is 56. The output pixels of all feature map channels are cyclically summed to obtain high-resolution image pixels.

[0030] The META-SR-based upsampling module circuit design proposed in the present invention can freely expand the upsampling multiple and is compatible with inputting low-resolution images that are smaller than the set input image size.

[0031] like Figure 4 As shown, the upsampling circuit module is capable of 2, 3, and 4 times upsampling. If other upsampling multiples need to be added, it is only necessary to increase the corresponding multiple weight reading unit and increase the basic convolution unit to the square of the highest upsampling multiple.

[0032] This design is compatible with inputting low-resolution images that are smaller than the set input image size. When inputting low-resolution images that are smaller than the set input size, the line buffer module does not need to be modified, such as Fig.10 As shown, only the input low-resolution image needs to be padded with zeros.

[0033] The present invention implements an embodiment in the hardware implementation of a super-resolution convolutional neural network FSRCNN. The network outputs a 56-channel feature map in the feature extraction part and uses a deconvolution layer for upsampling. In the case of using a conventional deconvolution calculation circuit, the upsampling module in the hardware implementation of the network can only achieve single-multiple upsampling, and the hardware resource consumption is large; however, the design of the present invention can achieve any integer multiple upsampling and reduce hardware resource overhead.

[0034] Fig.11 and Fig.12 The upsampling module circuit based on META-SR proposed in the present invention is compared with the deconvolution layer circuit of the super-resolution convolutional neural network FSRCNN implemented by Li Hanwen in his master's thesis "FPGA Design and Implementation of Fast Super-Precision Convolutional Neural Network (FSRCNN)". In terms of function, the upsampling module circuit based on META-SR proposed in the present invention can achieve any integer multiple upsampling. Fig.11 It can be seen that the PSNR value of the upsampling module based on META-SR on the Set5 test set is better than that of the deconvolution layer upsampling module of FSRCNN. The number of multipliers (DSP) used in the present invention when only upsampling by 3 times is 81, which is much smaller than the 135 multipliers used in the deconvolution layer circuit. Fig.12 It can be seen that even when the present invention implements 2, 3, and 4 times upsampling, the usage of the lookup table (LUT) and flip-flop (FF) hardware resources is much smaller than that of the deconvolution layer circuit.

[0035] In summary, the upsampling module circuit based on META-SR proposed in the present invention has the following advantages:

[0036] Compared with the existing upsampling module, the upsampling module circuit based on META-SR proposed in the present invention can achieve upsampling of any integer multiple, requires fewer hardware resources to implement the circuit, has good scalability, and has a wider range of application scenarios.

[0037] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An upsampling module circuit based on the META-SR algorithm, characterized in that: Aiming at the hardware implementation and optimization requirements of the super-resolution neural network FSRCNN, a computing circuit was proposed based on the META-SR algorithm, which can support any integer multiple upsampling module and is compatible with low-resolution images whose input size is smaller than the set input image size.

2. A computing circuit capable of supporting any integer multiple upsampling module according to claim 1, characterized in that: The circuit supports the expansion of upsampling functions of arbitrary integer multiples. The upsampling module circuit includes three modules, namely the row buffer module, the weight access module and the convolution module. Among them: 1) The row buffer module is composed of row buffers of the number of feature map channels, and the row buffer performs sliding window output on the input feature map by filling and adjusting. 2) The weight access module is composed of a basic weight reading unit, which is composed of 9 groups of shift registers with a length of the number of feature map channels. The corresponding convolution kernel weight is output according to the feature map channel from which the current convolution pixel comes. 3) The convolution module is composed of a basic convolution unit, which is composed of 9 multipliers, 8 adders and 1 circular accumulator. After completing the accumulation of the feature map channels several times, it outputs high-resolution image pixels. When adding other upsampling multiples, it is only necessary to increase the corresponding multiple weight reading unit and increase the basic convolution unit to the square of the highest upsampling multiple.

3. The computing circuit according to claim 1, wherein the compatible input image size is smaller than the set input image size, is characterized in that: When the input image size is smaller than the set input image size, zero values ​​are filled in the row buffer module to ensure the adaptation of the input feature map to the fixed row buffer structure, thereby completing the upsampling operation and improving the module's adaptability to different input image sizes.

Images