A dual cubic interpolation circuit based on FPGA

Abstract

The application provides a dual cubic interpolation circuit based on FPGA, and the working mode of the circuit is as follows: firstly, a control unit reads pixels from an external DDR memory into an input buffer through DMA, the read pixels are written into the input buffer in a loop, then a PE directly reads the input buffer into an internal operation register, the PE calculates high-definition image pixels, and the output of the PE is rearranged through an output buffer, so that the hardware output is directly connected with a high-definition video interface. The application reduces the number of line buffers, improves the calculation amount of each time period, maximizes the throughput, reduces the delay time required by calculation, and designs the output buffer, so that the system can be connected with a conventional video interface.

Description

Technical Field

[0001] This invention belongs to the field of electronic device technology, and in particular relates to a bicubic interpolation circuit based on FPGA. Background Technology

[0002] In recent years, due to advancements in display technology, the cost of monitor manufacturing has gradually decreased, making televisions supporting 4K Ultra High Definition (UHD) resolution mainstream in the market. However, mainstream video sources are still primarily High Definition (HD) and Full High Definition (qHD). Displaying low-definition content directly on a high-definition screen results in poor image quality. Real-time super-resolution systems are hardware systems that scale low-definition video down to high-definition video, ensuring the video's clarity matches the high-definition screen. High-quality real-time video super-resolution systems are crucial for 4K audio-visual systems. Super-resolution methods have been extensively researched, and various solutions have been proposed.

[0003] Bicubic interpolation, a classic super-resolution algorithm, is capable of high-quality image reconstruction, but its computational complexity is extremely high. For a 2x super-resolution problem, assuming the image's height and width are m×n, the complexity is O(64mn). Therefore, designing an efficient hardware architecture to accelerate bicubic interpolation is a challenge. Bicubic interpolation can be used not only in super-resolution systems but also embedded in various image processing systems, making its wide applicability significant. Therefore, designing a hardware accelerator for bicubic interpolation is of profound importance.

[0004] Technical solution of existing technology 1

[0005] In its early years, Nuno et al. [https: / / doi.org / 10.1109 / RECONFIG.2005.34] proposed a bicubic interpolation hardware architecture that could achieve speeds up to 10 times faster than the software implementation on a 4-core 2.4GHz CPU at the time. Their proposed bicubic interpolation architecture had a 4-stage pipeline in its computational core, enabling it to calculate one pixel every 4 time cycles.

[0006] Disadvantages of existing technology 1

[0007] However, Technique 1 has higher latency, requiring at least four rows of pixels to be cached before calculation can begin. Furthermore, each calculation unit requires four time cycles to calculate a single pixel, resulting in lower data throughput. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a bicubic interpolation circuit based on FPGA.

[0009] This invention mainly addresses the following problems in the current design of bicubic interpolation circuits:

[0010] 1) Reduce the number of line buffers; 2) Increase the computation amount per time period to maximize throughput; 3) Reduce the latency required for computation; 4) Design an output buffer so that the system can interface with conventional video interfaces, such as VGA interfaces.

[0011] The present invention adopts the following technical solution:

[0012] A bicubic interpolation circuit based on FPGA includes a pre-amplifier (PE), an input buffer, an output buffer, and an internal arithmetic register. The circuit operates as follows: First, the control unit reads pixels from external DDR memory into the input buffer via DMA. The read pixels are then cyclically written back into the input buffer. Next, the PE directly reads the input buffer into the internal arithmetic register. After calculating the high-definition image pixels, the PE's output is rearranged through the output buffer, allowing the hardware output to directly interface with the high-definition video interface.

[0013] Furthermore, the input buffer contains three circular row buffers. When the last pixel of the last row buffer is written, the next pixel will be written again starting from the first pixel of the first row buffer, using a shift register to provide the first 3 columns of the input image block.

[0014] Furthermore, the values ​​of the first three columns of the input graph block will be updated from the input of the three row caches and the external DDR.

[0015] Furthermore, the PE is a two-stage pipeline, with interpolation operations consisting of four MAC (Macro-Macro-Interpolation) operations. Stage 1 is the X-direction interpolation, and Stage 2 is the Y-direction interpolation. The input pixel passes through the MAC pyramid to obtain the result of four MAC operations. This result is the result of Stage 1, i.e., the X-direction interpolation. Subsequently, the calculation result is input into a trigger, and after a one-beat delay, it is output to the circuit of Stage 2. Stage 2 uses the same MAC pyramid structure as Stage 1, with the result of Stage 1 (i.e., the result of four X-direction interpolations) as the input, and the final pixel value as the output.

[0016] Furthermore, the output buffer consists of two congruent buffer arrays. The congruent buffer arrays operate as follows: for a row of pixels, four BRAM buffers are used, namely BRAM0, BRAM1, BRAM2, and BRAM3, to form a congruent buffer. The pixel with coordinate k is stored in the BRAM. kmod4In this configuration, the address is floor(k / 4). The BRAM cache is configured as a dual-port RAM, capable of writing one pixel and reading four pixels simultaneously. Thus, four BRAMs can write four pixels and read 16 pixels within one clock cycle. Since the PE output is a 4×4 matrix containing four rows, 16 BRAMs are used to form a congruent cache array. Within one time cycle, 16 pixels from the PE output can be written to the congruent cache simultaneously, and 16 consecutive pixel values ​​from a row can be read simultaneously, achieving input / output rate matching and thus enabling data rearrangement.

[0017] Furthermore, to avoid conflicts between read and write operations, the hardware system uses a ping-pong cache to separate data writing and reading. First, one congruent cache array is filled, and then the congruent cache array is read. At the same time, another congruent cache array is written. The two congruent cache arrays are read and written alternately to ensure that data can be continuously input and output.

[0018] The beneficial effects of this invention are:

[0019] This invention proposes a bicubic interpolation circuit based on FPGA. Its advantages are: 1) An optimal input buffering scheme is designed, reducing the number of line buffers to 3 lines. 2) Each time-cycle calculation unit can calculate one pixel, achieving the theoretically maximum throughput achievable when inputting pixels sequentially per frame. 3) The calculation latency is reduced to the time cycle required for buffering only one line, theoretically achieving the optimal latency. 4) An output buffer is designed, enabling the system to interface with conventional video interfaces, such as VGA interfaces. Attached Figure Description

[0020] Figure 1 This is the hardware design data flow for the present invention;

[0021] Figure 2(a) is a schematic diagram of the last pixel update in the row buffer (nth frame);

[0022] Figure 2(b) shows a schematic diagram of the last pixel update in the row buffer (in the (n+1)th frame);

[0023] Figure 3(a) is a schematic diagram of the current output status of the line buffer;

[0024] Figure 3(b) shows the composition of the input image blocks;

[0025] Figure 4 This is a diagram of the PE structure.

[0026] Figure 5(a) Logical arrangement of the congruent cache array;

[0027] Figure 5(b) shows the actual layout of the congruent cache array. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0029] The present invention provides a bicubic interpolation circuit based on FPGA, such as... Figure 1 As shown, an FPGA-based accelerator typically includes a Processing Element (PE), on-chip memory, registers, and a LUT. The control unit first reads data from external DDR (Dual Data Rate) memory into on-chip memory via DMA (Direct Memory Access), and then the PE directly reads data from on-chip memory into its internal arithmetic registers. Figure 1 The D flip-flop in the chip calculates the high-definition image pixels, and after rearranging the pixels through the output buffer, it can be output to a high-definition interface such as a VGA interface. This invention uses on-chip memory (BRAM) to implement three circular line buffers; pixels read from external DDR memory are cyclically written into these three line buffers. For example... Figure 2(a)-Figure 2(b) As shown, after the last pixel of the last row buffer is written, the next pixel will be written again starting from the first pixel of the first row buffer. This invention uses a shift register to provide the first three columns of the input image blocks. These values ​​are updated from the input of the three row buffers and the external DDR, as shown in the following update method. Figures 3(a)-3(b) As shown in Figure 3(a), the output of the BRAM is the pixel currently pointed to by the pointer. Figure 3(b) shows the structure of the input image block. The solid squares are flip-flops, and the last column of dashed squares represent the outputs of the three row buffer BRAMs, namely the output of the BRAM shown in Figure 3(a), and the output of the DDR through DMA. In this way, using the shift register, row buffer, and data provided by the external DDR, the input image block can be updated in each clock cycle, achieving the effect of shifting one pixel to the right, and writing it into the internal arithmetic calculator of the PE.

[0030] like Figure 4 As shown, the design of the pipelined PE is presented. The rectangle between the two stages represents the triggers, f11 to f14 represent the first row of pixels of the LR image patch, and p is the output pixel (the portion from the second to the fourth row of pixels is omitted in the figure). This invention designs a two-stage pipelined PE to reduce data latency and increase the operating frequency. The interpolation operation is completed by four MAC operations. This invention adopts a pyramid structure design for the four MAC operations to reduce data latency. The design structure is as follows: Figure 4As shown. Stage 1 is the X-direction interpolation, and Stage 2 is the Y-direction interpolation. The input pixel is processed through the MAC pyramid to obtain the result of 4 MAC operations. This result is the result of Stage 1, i.e., the X-direction interpolation. Subsequently, the calculation result is input into the trigger, and after a one-beat delay, it is output to the circuit of Stage 2. Stage 2 uses the same MAC pyramid structure as Stage 1, with the result of Stage 1 as the input, i.e., the result of 4 X-direction interpolations, and the final pixel value as the output.

[0031] For the output section, this invention designs a congruent buffer array. The PE output is rearranged, avoiding the need to rewrite the output results back to external DDR, allowing the hardware output to directly interface with high-definition video interfaces, such as VGA. For a row of pixels, this invention uses four BRAM buffers, namely BRAM0, BRAM1, BRAM2, and BRAM3, to form a congruent buffer. The pixel at coordinate k is stored in the BRAM. kmod4 In this configuration, the address is floor(k / 4). The BRAM cache is configured as a dual-port RAM, capable of writing one pixel and reading four pixels simultaneously. Thus, four BRAMs can write four pixels and read 16 pixels within one clock cycle. Since the PE output is a 4×4 matrix containing four rows, this invention uses 16 BRAMs to form a congruent cache array. Within one time cycle, 16 pixels from the PE output can be written to the congruent cache simultaneously, and 16 consecutive pixel values ​​from a row can be read simultaneously, achieving input / output rate matching and thus data rearrangement. Figures 5(a)-(b) illustrate this process in detail.

[0032] As shown in Figures 5(a) and 5(b), the cache array consists of 4×4 BRAM blocks, caching 4 rows of pixels. Each BRAM block can simultaneously write one pixel or read four pixels. Figure 5(a) shows the arrangement of cached pixels in the image, and Figure 5(b) shows the actual physical storage arrangement.

[0033] To avoid conflicts between reading and writing, this invention uses a ping-pong cache to separate data writing and reading. First, a congruent cache array is filled, and then the congruent cache array is read. At the same time, another congruent cache array is written. The two congruent cache arrays are read and written alternately to ensure that data can be continuously input and output.

[0034] Example

[0035] This invention implements the proposed hardware on a Zynq-7020 FPGA. The source image size is 960×540, and the super-resolution image size is 3840×2160. The bicubic interpolation hardware implemented on the FPGA reads the image data from memory via the AXI (Advanced deXtensible Interface) bus and calculates the result in real time, outputting it to the HDMI interface. The system operates at a frequency of 100MHz, and the measured frame rate reaches 192.9.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bicubic interpolation circuit based on FPGA, characterized in that, It includes a PE (Programmable Array), input buffer, output buffer, and internal arithmetic register. The working mode of the FPGA-based bicubic interpolation circuit is as follows: First, the control unit reads pixels from external DDR memory into the input buffer via DMA. The read pixels are cyclically written into the input buffer. Then, the PE directly reads the input buffer into the internal arithmetic register. After the PE calculates the high-definition image pixels, the output of the PE is rearranged through the output buffer so that the hardware output can directly connect to the high-definition video interface. The PE is a two-stage pipeline. The interpolation operation consists of four MAC operations. Stage 1 is the X-direction interpolation, and Stage 2 is the Y-direction interpolation. The input pixel passes through the MAC pyramid to obtain the result of four MAC operations. This result is the result of Stage 1, i.e., the X-direction interpolation. Subsequently, the calculation result is input into the trigger, and after a one-beat delay, it is output to the circuit of Stage 2. Stage 2 uses the same MAC pyramid structure as Stage 1. The input is the result of Stage 1, which is the result of four X-direction interpolations, and the output is the final pixel value.

2. The FPGA-based bicubic interpolation circuit according to claim 1, characterized in that, The input buffer consists of three circular line buffers. After the last pixel of the last line buffer is written, the next pixel will be written again starting from the first pixel of the first line buffer. The three circular line buffers can provide three pixels for a small block of the input image.

3. The FPGA-based bicubic interpolation circuit according to claim 2, characterized in that, The values ​​of the first three columns of the input graph block will be updated from the input of the three row caches and the external DDR.

4. The FPGA-based bicubic interpolation circuit according to claim 1, characterized in that, The output buffer consists of two congruent buffer arrays. The congruent buffer arrays work as follows: for a row of pixels, four BRAM buffers are used for storage, namely BRAM0, BRAM1, BRAM2, and BRAM3. The congruent buffers store the pixel at coordinate k into the BRAM. k mod 4 In this configuration, the address is floor (k / 4), and the BRAM cache is configured as a dual-port RAM, simultaneously writing 1 pixel and reading 4 pixels. Thus, 4 BRAMs can write 4 pixels and read 16 pixels in one clock cycle. Since the PE output is a 4×4 matrix containing four rows, 16 BRAMs are used to form a congruent cache array. In one time cycle, 16 pixels from the PE output are simultaneously written to the congruent cache, and 16 consecutive pixel values ​​in one row are read simultaneously, achieving input and output rate matching, thereby achieving the effect of data rearrangement.

5. The FPGA-based bicubic interpolation circuit according to claim 4, characterized in that, In addition, to avoid conflicts between reading and writing, the hardware system uses the two congruent cache arrays to form a ping-pong cache to separate the writing and reading of data. First, one congruent cache array is filled, and then the congruent cache array is read. At the same time, the other congruent cache array is written. The two congruent cache arrays are read and written alternately to ensure continuous data input and output.

Images