Bilinear interpolation algorithm real-time hardware system and method based on FPGA

Abstract

The invention discloses a bilinear interpolation algorithm real-time hardware implementation method and system based on an FPGA, and belongs to the technical field of image processing. In order to solve the problems that traditional software is high in implementation delay and an existing hardware scheme is large in resource consumption, the invention designs a full-parallel pipeline architecture: an input interface module analyzes image data; the coordinate calculation module maps a target pixel coordinate to an original image continuous coordinate, and generates four nearest neighbor reference pixel coordinates and weight coefficients; the weight generation module adopts Q16 fixed-point quantitative weight; the reference pixel cache module caches two rows of reference data through a dual-port BRAM; the multiply-add calculation module realizes parallel multiply-add operation by using multiplier time division multiplexing; and the control module integrates boundary detection and clamping logic to ensure the legality of coordinates. According to the invention, single-cycle processing of 1 pixel is realized, the throughput is greater than 150M pixel / second, 4K and 30fps real-time processing is supported, the DSP occupancy rate is reduced by more than 60%, single-chip integration on a low-cost FPGA (Field Programmable Gate Array) can be realized, and the method is suitable for scenes such as video monitoring, machine vision and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a real-time hardware system and method for a bilinear interpolation algorithm based on FPGA. BACKGROUND

[0002] In the field of digital image processing, image scaling, as one of the most basic operations, is widely used in video surveillance, medical imaging, machine vision, high-definition display and other scenarios. The bilinear interpolation algorithm can achieve a good balance between computational complexity and reconstruction quality by weighted averaging of the four nearest neighbor reference pixels around the target pixel. Its precision is higher than that of the nearest neighbor interpolation, and the computational complexity is significantly lower than that of the bicubic interpolation. Therefore, it has become the preferred algorithm for real-time image processing systems.

[0003] Currently, the implementation of the bilinear interpolation algorithm mainly includes software and hardware methods. Software implementation is usually based on general-purpose CPU or GPU platforms, and the calculation is completed through instruction-level parallelism or thread-level parallelism. However, with the evolution of image resolution to 4K / 8K and the increase of frame rate to 60fps or even higher, software implementation faces severe challenges: the single-frame data volume of high-resolution images is huge, and memory access bandwidth becomes a bottleneck; the delay introduced by instruction scheduling and thread synchronization makes it difficult to meet the hard real-time requirements (usually single-frame processing time ≤ 16ms); although GPU has parallel computing capability, it has high power consumption and high cost, which is not suitable for embedded scenarios.

[0004] To address the real-time bottleneck, the industry has turned to FPGA hardware acceleration solutions. Traditional FPGA implementations mostly use serial or semi-parallel architectures, which decompose the bilinear interpolation into sequentially executed coordinate calculation, weight generation, pixel reading and multiplication-addition operation steps. Although this architecture significantly improves the speed compared to software, it still has the following shortcomings: first, resource consumption is high, some designs use floating-point operation units, resulting in high occupancy of DSP Slice, registers and memory resources, making it difficult to deploy on low-cost FPGA devices; second, the computational efficiency is low, the key resources such as multipliers lack sharing mechanisms, and each pixel calculation needs to monopolize hardware units, resulting in low DSP utilization; third, the boundary processing is complex, when the target pixel coordinates exceed the reference image boundary, additional conditional judgment and jump logic need to be inserted, causing pipeline stall or resource redundancy, affecting throughput stability.

[0005] In addition, existing hardware solutions are not optimized for storage architecture, frequently accessing external memory to read reference pixels, which increases system power consumption and introduces uncertain memory access delay. In summary, existing technologies cannot simultaneously meet the multiple demands of high real-time performance, low resource consumption, high robustness and low cost, and there is an urgent need for a bilinear interpolation hardware implementation solution that is deeply optimized for FPGA architecture. SUMMARY

[0006] The purpose of this invention is to provide a real-time hardware system and method for a bilinear interpolation algorithm based on FPGA, so as to solve the problem that the existing technology mentioned in the background art is unable to simultaneously meet the multiple requirements of high real-time performance, low resource consumption, high robustness and low cost.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A real-time hardware system for a bilinear interpolation algorithm based on FPGA, comprising: The input interface module is used to receive raw image data and synchronization signals, and parse them to obtain pixel coordinates and pixel values. The coordinate calculation module is used to map the target pixel coordinates to continuous coordinates of the original image based on the scaling ratio between the target image and the original image, and to calculate the integer coordinates and weight coefficients of the four nearest neighbor reference pixels. The continuous coordinates are calculated according to the formula:

[0008] Calculation, where , For the target pixel coordinates, For scaling ratio, , For the original image resolution, , The target image resolution; The weight generation module is used to represent the weight coefficients in a fixed-point quantized form, wherein the weight coefficients are based on the formula:

[0009] Calculation, where The fractional part of the continuous coordinates. , The integer coordinates; The reference pixel cache module uses a dual-port BRAM to cache the row buffer data of the original image and reads four reference pixel values ​​according to the integer coordinates. The multiply-add module contains a parallel multiplier and an adder tree, used to perform calculations based on formulas.

[0010] The target pixel value is obtained by performing a multiply-add operation, where , , , These are the reference pixel values ​​at four integer coordinates; The control module is used to perform timing control of each module using a multi-stage pipeline architecture, and includes boundary detection and clamping units; The output interface module is used to output the target pixel value and the synchronization signal.

[0011] Furthermore, the cache depth of the reference pixel cache module is equal to the width of the target image. The amount is doubled to cover the two adjacent rows of data where the four nearest neighbor reference pixels are located.

[0012] Furthermore, the multiply-accumulate module uses time-division multiplexing technology to share multiplication resources, and when the scaling ratio is... When n is a positive integer, the weighting coefficient , , , For a fixed value, the multiplication-addition calculation module is implemented using combinational logic.

[0013] Furthermore, the multi-stage pipeline architecture is a four-stage pipeline, including four stages: coordinate calculation, weight generation, reference pixel reading, and multiply-accumulate calculation.

[0014] Furthermore, the system supports real-time scaling of 4K@30fps or higher resolution images, with a throughput of over 150M pixels / second. When implemented on a Xilinx Artix-7 series FPGA, the DSP Slice utilization rate does not exceed 10%, and the system latency does not exceed 3 clock cycles.

[0015] A real-time hardware implementation method for a bilinear interpolation algorithm based on FPGA includes the following steps: Step S1: Data Input: Receive raw image data and synchronization signal, and parse to obtain pixel coordinates and pixel values; Step S2: Coordinate Calculation: Based on the target image resolution Compared to the original image resolution scaling ratio , target pixel coordinates ( , Mapped to continuous coordinates of the original image ( , ) = ( × , × ); Step S3: Calculate the decimal part: According to the formula Calculate the fractional part of the continuous coordinates, where The integer part of the continuous coordinates Integer coordinates of the four nearest neighbor reference pixels; Step S4: Weight Calculation Step: According to the formula Calculate the weight coefficients for the four reference pixels; Step S5: Weight quantization step: The weight coefficients are represented by fixed-point quantization, using Q16 format or Q1 format selected according to the scaling ratio; Step S6: Pixel caching step: Use a dual-port BRAM to cache the row buffer data of the original image, and according to the integer coordinates Read four reference pixel values; , , , .

[0016] Step S7: Parallel computation step: Using parallel multipliers and adder trees, according to the formula... Perform multiplication and addition operations to obtain the target pixel value; Step S8: Pipeline processing steps: A four-stage pipeline architecture is used to control the timing of each step; Step S9: Output Results: Output the target pixel value and synchronization signal.

[0017] Furthermore, the fixed-point quantization adopts the Q16.16 format, which consists of a 16-bit integer part and a 16-bit fractional part, ensuring that the weighting coefficients maintain computational accuracy when the scaling ratio is any rational number.

[0018] Furthermore, when the scaling ratio and When both are powers of 2, the fractional parts u and v take only 0 or 0.5, and the weighting coefficients... , , , For a fixed combination of values ​​0, 0.25, or 0.5, the multiplication-addition operation is simplified to a shift and addition operation.

[0019] Furthermore, the cache depth of the dual-port BRAM is twice the width of the target image, and among the four integer coordinates... and Row data is read in parallel through dual ports. and The column data is obtained within the same clock cycle through address decoding logic.

[0020] Furthermore, it also includes boundary detection and clamping steps: when integer coordinates , , , Beyond the boundaries of the original image At that time, the outbound coordinates are clamped to the boundary values, i.e.

[0021] .

[0022] Compared with the prior art, the beneficial effects of the present invention are: (1) High real-time performance and high throughput: The four-stage pipeline architecture enables processing of one target pixel in a single clock cycle, with a throughput of over 150M pixels / second. It supports real-time processing of 4K / 8K high frame rate (such as 30fps / 60fps) images, with a single frame latency of ≤16ms. (2) Low resource consumption and high energy efficiency: The use of Q16 fixed-point quantization and DSP slice time-division multiplexing reduces resource consumption by more than 60% compared with traditional floating-point design (DSP utilization rate on Artix-7 is only 8%); the dual-port BRAM only caches two rows of reference pixels, without the need to access external memory, reducing system power consumption by 30%-40%; (3) High integration and low cost: All functional modules (including interfaces, computing, caching and control) are implemented based on FPGA on-chip resources (LUT, registers, DSP, BRAM), without the need for any external chips. The single-chip solution significantly reduces BOM cost and PCB complexity, and is compatible with low-cost Artix-7 and other mid-to-low-end devices. (4) Strong robustness and high adaptability: Built-in boundary detection and coordinate clamping logic automatically handles input of any resolution (including non-standard resolution) and prevents out-of-bounds access; supports multiple formats such as RGB / YUV and 8 / 10 / 12 bit width, and the system is stable and reliable. Attached Figure Description

[0023] Fig. 1 This is an overall architecture diagram of the FPGA-based bilinear interpolation real-time hardware system described in this invention. Fig. 2 This is a schematic diagram of the four-stage processing sequence of the pipeline described in this invention (Stage 1 to Stage 4). Fig. 3 This is a logic block diagram of the coordinate calculation module described in this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figs. 1 to 3 The core idea of ​​this invention is to decompose the bilinear interpolation algorithm into parallelizable hardware computing units, utilize the parallel processing capabilities and on-chip storage resources of FPGAs, reduce latency through pipelined design, and reduce resource consumption through fixed-point arithmetic and resource sharing. The specific technical solution is as follows: The hardware system of this invention includes: an input interface module, a coordinate calculation module, a weight generation module, a reference pixel cache module, a multiply-accumulate calculation module, an output interface module, and a control module. The functions of each module are as follows: Input interface module: Receives raw image data (supports RGB, YUV formats, 8 / 10 / 12-bit bit width), performs synchronization signal detection (horizontal synchronization HSYNC, field synchronization VSYNC, data validity signal DATA_VALID) on the input data, and outputs pixel coordinates. and pixel values.

[0026] Coordinate calculation module: based on target image resolution Compared to the original image resolution scaling ratio , target pixel coordinates ( , Mapped to continuous coordinates of the original image: ( , ) = ( × , × ), and calculate the integer coordinates of its four nearest neighbor reference pixels. and weighting coefficients (where x1 = x0 + 1, y1 = y0 + 1).

[0027] Weight generation module: Generates weights for four reference pixels based on u and v. To avoid floating-point operations, fixed-point quantization (such as Q16.16 format, 16-bit integer + 16-bit decimal) is used to convert the weighting coefficients into 16-bit fixed-point numbers.

[0028] Reference pixel cache module: Employs a dual-port BRAM (block RAM) to cache the row buffer data of the original image, with a cache depth equal to the target image width W. out Twice the size (covering rows y0 and y1), the four reference pixel values ​​are quickly read through address decoding logic. , , , .

[0029] Multiply-accumulate module: Contains four multipliers (each performing a multiplication and addition operation). ) and an addition tree (which adds four products to get the target pixel value P) out The multiplier is implemented using an on-chip DSP slice on the FPGA, and shares some multiplication resources through time-division multiplexing technology (the same set of multipliers is reused when processing different target pixels).

[0030] Output interface module: Outputs the calculated target pixel values ​​as line synchronization and field synchronization signals, supporting interface with subsequent display or storage devices.

[0031] The following is an example to illustrate this: Example: Real-time scaling system for 4K@30fps YUV images This embodiment takes a real-time scaling scenario of a YUV image with a target resolution of 3840×2160 (4K), an original resolution of 1920×1080 (1080p), and a scaling ratio of 2:1 as an example to illustrate the specific implementation process of the present invention.

[0032] Step 1: Input Interface Module Configuration

[0033] The input interface module receives 1080p@30fps YUV422 format image data (8 bits for Y component, 8 bits for U / V component, 1920 pixels for Y component and 960 pixels for U / V component per line), locks the input clock (74.25MHz) through the clock data recovery (CDR) circuit, and detects the HSYNC (line synchronization), VSYNC (vertical synchronization) and DATA_VALID (data valid) signals, and outputs a pixel stream of Y component and U / V component valid line by line.

[0034] Step 2: Design of coordinate calculation module

[0035] The target resolution is 3840×2160, and the original resolution is 1920×1080, therefore For target pixel coordinates ( , () ), continuous coordinates are calculated as follows: , because and It is 0.5 (a power of 2). and decimal part It can only be 0 or 0.5 (e.g., When u is even, u = 0; when u is odd, u = 0.5. To simplify calculations, u and v are represented using Q1 format fixed-point numbers (1 integer digit + 1 decimal place), where... (Adjustments may be needed based on actual quantification accuracy).

[0036] Step 3: Design of the weight generation module

[0037] The weighting coefficients are calculated as follows:

[0038] Since u and v can only take the values ​​0 or 0.5, the weighting coefficients can be pre-calculated as fixed values ​​(e.g., when u=0.5 and v=0.5). = = = =0.25), but this embodiment uses a general quantization method: representing u and v in Q1 format (1 decimal place), then ( (where u is an integer representation), therefore the weighting coefficients can be converted to Q2 format (2 decimal places), for example, u=0.5 ( When 1 = 1, 1 - u = 0.5 → Q2 format is 0b010 (i.e. 0.5 × 4 = 2).

[0039] Step 4: Reference Pixel Cache Module Design

[0040] The original image's Y component is 1920 pixels per row. A dual-port BRAM is used to cache two rows of Y data (current row y0 and previous row y1), with a cache depth of 1920 pixels per row and an address range of 0-1919. For the target pixel ( , ), (Boundary checks are required; if y1 ≥ 1080, then y1 = 1079).

[0041] The address for reading the Y component is: ,because (Divisible) (when If the number is odd, x1 = x0 + 1; otherwise, x1 = x0.

[0042] Step 5: Design of the multiplication and addition calculation module

[0043] The bilinear interpolation of the Y component is calculated as follows: ,in, .

[0044] Since u and v in this embodiment only take the values ​​0 or 0.5, and the weighting coefficients are 0, 0.25, or 0.5, the multiplication-addition calculation can be simplified to: when Even number and When it is even, x1=x0, y1=y0, both with a weight of 0.25. ; when odd number and When it is even, Weight ; In other cases, the weighting method can be directly selected through combinational logic.

[0045] In this embodiment, the multiply-accumulate calculation module is implemented using combinational logic (eliminating the need for a multiplier), further reducing resource consumption.

[0046] Step 6: Output Interface Module and Timing Verification

[0047] The output interface module outputs the calculated Y and U / V components (the U / V components are processed independently through downsampling or bilinear interpolation) at a timing of 4K@30fps, with HSYNC and VSYNC signal frequencies of 60kHz and 30Hz, respectively. Actual measurements show a system processing latency of 2 clock cycles (approximately 27ns@74.25MHz), a single-frame processing time of 33.3ms (meeting the 30fps real-time requirement), and a DSP slice utilization rate of 8% (using a Xilinx Artix-7 series FPGA XC7A100T), verifying the feasibility of the solution.

[0048] The beneficial effects of this solution are achieved through the coordinated optimization of architecture, algorithm, and circuit: Taking the above embodiment as an example, when the original 1080p image is magnified by 2 times, the scaling ratio Scale_x = Scale_y = 0.5 is a power of 2. After the coordinate calculation module maps the target pixel coordinates (x_out, y_out), its decimal parts u and v are always 0 or 0.5. Based on this, the weight generation module pre-calculates the weight coefficients, so that they only take three fixed values: 0, 0.25, and 0.5. The multiply-add calculation module further degenerates the floating-point operation that originally required four DSP multipliers into a shift-add operation implemented by combinational logic (such as ×0.25 is equivalent to right shift by 2 bits), so that the DSP Slice utilization is reduced to 8% (traditional floating-point solutions require >20%); the reference pixel cache module uses dual-port BRAM to read two lines of Y data in parallel, and obtains four reference pixels simultaneously through address decoding, eliminating memory access wait; the four-stage pipeline decomposes the processing flow into single-cycle tasks, compressing the system latency to 2 clock cycles (27ns@74.25MHz), ultimately achieving a throughput of 150M pixels / second and a single frame processing time of 33.3ms, strictly meeting the 4K@30fps real-time requirements. This design, based on a general architecture, simplifies the triggering logic for power-of-two scaling scenarios, achieving triple optimization of resources, speed, and cost.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time hardware system for a bilinear interpolation algorithm based on FPGA, characterized in that, include: The input interface module is used to receive raw image data and synchronization signals, and parse them to obtain pixel coordinates and pixel values. The coordinate calculation module is used to map the target pixel coordinates to continuous coordinates of the original image based on the scaling ratio between the target image and the original image, and to calculate the integer coordinates and weight coefficients of the four nearest neighbor reference pixels. The continuous coordinates are calculated according to the formula: Calculation, where , For the target pixel coordinates, For scaling ratio, , For the original image resolution, , The target image resolution; The weight generation module is used to represent the weight coefficients in a fixed-point quantized form, wherein the weight coefficients are based on the formula: Calculation, where The fractional part of the continuous coordinates. , The integer coordinates; The reference pixel cache module uses a dual-port BRAM to cache the row buffer data of the original image and reads four reference pixel values ​​according to the integer coordinates. The multiply-add module contains a parallel multiplier and an adder tree, used to perform calculations based on formulas. The target pixel value is obtained by performing a multiply-add operation, where , , , These are the reference pixel values ​​at four integer coordinates; The control module is used to perform timing control of each module using a multi-stage pipeline architecture, and includes boundary detection and clamping units; The output interface module is used to output the target pixel value and the synchronization signal.

2. The system according to claim 1, characterized in that, The cache depth of the reference pixel cache module is equal to the width of the target image. The amount is doubled to cover the two adjacent rows of data where the four nearest neighbor reference pixels are located.

3. The system according to claim 1, characterized in that, The multiply-accumulate calculation module uses time-division multiplexing technology to share multiplication resources, and when the scaling ratio is... When n is a positive integer, the weighting coefficient , , , For a fixed value, the multiplication-addition calculation module is implemented using combinational logic.

4. The system according to claim 1, characterized in that, The multi-stage pipeline architecture is a four-stage pipeline, including four stages: coordinate calculation, weight generation, reference pixel reading, and multiply-accumulate calculation.

5. The system according to claim 1, characterized in that, The system supports real-time scaling of 4K@30fps or higher resolution images, with a throughput of over 150M pixels / second. When implemented on a Xilinx Artix-7 series FPGA, the DSPSlice utilization rate does not exceed 10%, and the system latency does not exceed 3 clock cycles.

6. A real-time hardware implementation method for a bilinear interpolation algorithm based on FPGA, characterized in that, Includes the following steps: Step S1: Data Input: Receive raw image data and synchronization signal, and parse to obtain pixel coordinates and pixel values; Step S2: Coordinate Calculation: Based on the target image resolution W_out×H_out and the original image resolution... × scaling ratio , target pixel coordinates ( , Mapped to continuous coordinates of the original image ( , ) = ( × , × ); Step S3: Calculate the decimal part: According to the formula Calculate the fractional part of the continuous coordinates, where The integer part of the continuous coordinates Integer coordinates of the four nearest neighbor reference pixels; Step S4: Weight Calculation Step: According to the formula Calculate the weight coefficients for the four reference pixels; Step S5: Weight quantization step: The weight coefficients are represented by fixed-point quantization, using the Q16.16 format or the Q1.1 format selected according to the scaling ratio; Step S6: Pixel caching step: Use a dual-port BRAM to cache the row buffer data of the original image, and according to the integer coordinates Read four reference pixel values; , , , ; Step S7: Parallel computation step: Using parallel multipliers and adder trees, according to the formula... Perform multiplication and addition operations to obtain the target pixel value; Step S8: Pipeline processing steps: A four-stage pipeline architecture is used to control the timing of each step; Step S9: Output Results: Output the target pixel value and synchronization signal.

7. The method according to claim 6, characterized in that, The fixed-point quantization adopts the Q16.16 format, which consists of a 16-bit integer part and a 16-bit fractional part, ensuring that the weighting coefficients maintain calculation accuracy when the scaling ratio is any rational number.

8. The method according to claim 6, characterized in that, When the scaling ratio and When both are powers of 2, the fractional parts u and v take only 0 or 0.5, and the weighting coefficients... , , , For a fixed combination of values ​​0, 0.25, or 0.5, the multiplication-addition operation is simplified to a shift and addition operation.

9. The method according to claim 6, characterized in that, The dual-port BRAM has a cache depth of twice the width of the target image, and the four integer coordinates are... and Row data is read in parallel through dual ports. and The column data is obtained within the same clock cycle through address decoding logic.

10. The method according to claim 6, characterized in that, It also includes boundary detection and clamping steps: when the integer coordinates , , , Beyond the boundaries of the original image When this happens, the out-of-bounds coordinates are clamped to the boundary values, i.e. 。

Images