A Real-Time Pole Line Correction FPGA Pipeline Architecture with Uncompressed LUTs

Abstract

This invention relates to the field of image correction, specifically a real-time epipolar correction FPGA pipeline architecture without compressed LUTs. The invention proposes a real-time, high-efficiency epipolar correction pipeline architecture without compressed LUTs. Firstly, it abandons the traditional compressed LUT scheme, reading the image to be corrected in blocks according to a set ratio. Then, it designs and constructs a high-speed image cache array based on four dual-port RAMs. Through address rearrangement, random four-point interpolation is converted into parallel reading of cache array data addresses, quickly achieving access to neighboring grayscale values ​​and eliminating the random access bottleneck of DDR memory in the FPGA system. This invention can be used to achieve high-throughput, low-latency real-time epipolar correction.

Description

Technical Field

[0001] This invention relates to the field of image correction, and in particular to a real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs. Background Technology

[0002] Epipolar alignment is a fundamental calibration process in multi-camera view systems, used for epipolar alignment, also known as epipolar correction. In binocular (multi-camera) stereo matching systems, epipolar correction, as a preprocessing step, effectively reduces the computational complexity of disparity estimation and is an indispensable part. To output disparity / depth images in real time, epipolar correction must also be performed in real time (the real-time performance of epipolar correction directly affects the real-time performance of the stereo matching system), but currently, there is relatively little research focus on this aspect.

[0003] FPGA-accelerated real-time stereo matching systems offer advantages such as customizability, low power consumption, and low latency compared to traditional CPUs or GPUs. Accelerating algorithms using hardware like FPGAs is considered one of the most effective methods currently available. However, traditional general-purpose stereo matching techniques typically only require real-time epipolar correction for a single image pair, placing relatively low processing pressure on the FPGA. When spatiotemporal modulation techniques are introduced to improve stereo matching accuracy, the system needs to complete epipolar correction for N≥2 image pairs within milliseconds. As the number of image pairs increases, the real-time performance of epipolar correction faces significant challenges with limited FPGA resources.

[0004] There are generally two common FPGA epipolar correction schemes: one is the non-LUT correction scheme, which uses real-time pixel-by-pixel calculation of mapping relationships to achieve image correction within the FPGA. This scheme implements image distortion correction and mapping calculations through the FPGA, calculating the mapping coordinates of input pixels in real time. Because it requires a large amount of real-time computation, it consumes significant resources from the FPGA's internal logic units and DSP. The other is the lookup table (LUT) correction scheme, which pre-calculates the image mapping relationships and stores them in memory. During operation, the mapping result for each pixel is obtained through the LUT. The LUT scheme requires pre-storing the mapping matrix LUT for the entire left and right images. Although it consumes more memory resources than the first scheme, it avoids the complex floating-point operations within the FPGA, greatly reducing the performance requirements of the FPGA and the consumption of logic units. Especially when the system has inexpensive and high-capacity memory like DDR (due to limited internal FPGA resources, LUTs are usually stored in external DDR), this scheme has significant advantages and has become the mainstream application.

[0005] In the early days, FPGA polar correction typically used compressed LUTs. This was done to reduce storage resource consumption, but for FPGA systems now widely equipped with inexpensive, high-capacity DDR, the benefits are no longer significant. On the other hand, the aim was to trade "small data volume" for faster read and write speeds. However, it is difficult to balance the complexity of compression algorithms (involving compression ratio, lossy compression, etc.) with the time and resources required for decompression, which may be counterproductive (increasing additional latency and complexity).

[0006] Furthermore, in existing technologies, epipolar correction requires four-point (typically four-point) bilinear interpolation for each output pixel, which necessitates reading the grayscale values ​​of four neighboring pixels from the original image. Since the positions of these four pixels are determined by the LUT, their positions in the original image are typically discontinuous, representing a typical random access pattern. In CPU systems, the random access performance issue of DDR is mitigated through multi-level caches (L1 / L2 / L3) and hardware prefetching mechanisms, resulting in faster access speeds perceived by software developers. However, in FPGA systems, due to the lack of a CPU-level automatic cache mechanism, direct random access to DDR exposes the true latency bottleneck. Specifically, while DDR burst sequential access bandwidth can reach 10 GB / s, random access requires frequent row activation (RAS), column selection (CAS), and precharge operations, reducing the actual bandwidth to around 100 MB / s. For high-resolution images, if each pixel requires four random accesses, the total number of accesses reaches millions. Using DDR random access in an FPGA system will lead to a severe performance bottleneck, failing to meet real-time processing requirements.

[0007] In summary, to address the aforementioned issues, this study proposes a simple and efficient real-time epipolar correction FPGA pipeline architecture, eliminating latency and complex LUT compression steps. Furthermore, it constructs a high-speed image cache array suitable for fast access to random points in images at the macroscopic architecture level, thus solving the problem of huge resource consumption for image random point access on FPGA systems. Summary of the Invention

[0008] The purpose of this invention is to overcome the problem of low efficiency in epipolar correction in the prior art and to provide a real-time epipolar correction FPGA pipeline architecture without compressed LUTs.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0010] A real-time polar line correction FPGA pipeline architecture with uncompressed LUTs includes an input port, an output port, an ARM module, and an FPGA module. The FPGA module includes a high-speed data processing unit, a polar line correction unit, a first DDR3, and a cache array. The ARM module includes a second DDR3.

[0011] The input port is used to receive the image to be corrected, read the sub-image data and corresponding LUT parameters of the image to be corrected, store the sub-image data in the first DDR3, and store the LUT parameters in the second DDR3;

[0012] The high-speed data processing unit is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio, and send them to the epipolar correction unit.

[0013] The epipolar correction unit is used to arrange the read sub-image data according to a set rule and store it into the cache array, and generate the corrected image according to the read LUT parameters;

[0014] The output port is used to receive and output the corrected image;

[0015] The cache array includes a FIFO, four dual-port RAMs, and an address decoding circuit.

[0016] The FIFO's data output interface is 8 bytes, used to communicate with the epipolar correction unit; the four dual-port RAMs are used to store sub-image data arranged according to a set rule; the address decoding circuit is used to convert random interpolation into cache array data addresses for parallel reading through address rearrangement.

[0017] As a preferred embodiment of the present invention, the high-speed data processing unit includes a DDR data high-speed read module, a DDR data high-speed write module, a control state machine, and three AXI buses;

[0018] Of the three AXI buses, one group is responsible for reading sub-image data from the first DDR3, another group reads LUT parameters from the second DDR3, and the last group is responsible for writing the calculation results of the epipolar correction unit into the first DDR3.

[0019] The control state machine is used to receive external commands and control the DDR data high-speed read module and the DDR data high-speed write module.

[0020] The DDR high-speed data read module is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio; and write the data to the data cache unit through the DDR high-speed data write module.

[0021] As a preferred embodiment of the present invention, the ratio set in the high-speed data processing unit is:

[0022] M+1 rows of sub-image data and M rows of LUT parameters, where M = 1 / 3 × the number of rows in one image of the image to be corrected.

[0023] As a preferred embodiment of the present invention, the epipolar correction unit includes a partition correction control state machine, a high-speed image cache module, and a computing module;

[0024] The partition correction control state machine is used to control the high-speed image cache module and the computing module;

[0025] The high-speed image cache module is used to arrange the read sub-image data according to a set rule and store it into the cache array;

[0026] The computing module is used to perform epipolar correction calculations based on the read LUT parameters;

[0027] The setting rules are as follows:

[0028] The sub-image data of the read image to be corrected are converted from horizontal to vertical arrangement and then stored sequentially in the cache array.

[0029] As a preferred embodiment of the present invention, the generation of the corrected image in the epipolar correction unit includes the following steps:

[0030] Traverse the cache array;

[0031] Epipolar correction is performed on each pixel in the sub-image data according to the corresponding LUT parameters. After all pixels have completed epipolar correction, the corrected image is output.

[0032] As a preferred embodiment of the present invention, the polar line correction step includes:

[0033] Acquire grayscale values ​​of several adjacent pixels around the pixel;

[0034] Obtain the LUT parameters of the grayscale values ​​of the adjacent pixels;

[0035] Multiply the gray values ​​of several adjacent pixels by their corresponding LUT parameters, sum the products, and then divide the sum by the set parameters.

[0036] As a preferred embodiment of the present invention, when the number of adjacent pixels is 4, the expression for epipolar correction is:

[0037] Result=(d1×c1+d2×c2+d3×c3+d4×c4) / 256,

[0038] Where Result is the corrected grayscale value of pixel (X, Y), d1, d2, d3, and d4 are the grayscale values ​​of the four neighboring pixels of the pixel (X, Y) to be corrected, and c1, c2, c3, and c4 are the LUT parameters of the pixel (X, Y) to be corrected.

[0039] As a preferred embodiment of the present invention, the calculation of the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected includes the following steps:

[0040] Obtain the access addresses in the cache array corresponding to the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected;

[0041] Read the corresponding grayscale value based on the access address.

[0042] As a preferred embodiment of the present invention, the calculation of the corresponding access address in the cache array includes the following steps:

[0043] The address parameter Di, i={1,2,3,4}, is calculated using the following expression:

[0044] D1 = Y·W + X,

[0045] D2 = Y·W + X + 1,

[0046] D3 = Y·W + X + W,

[0047] D4 = Y·W + X + W + 1;

[0048] Where X and Y are the horizontal and vertical coordinates of the pixel to be corrected, respectively, and W is the width of the image to be corrected;

[0049] Di is divided by 4, and the quotient and remainder are obtained.

[0050] The calculated quotient is output as the column number of the access address corresponding to di in the cache array, and the remainder is output as the row number of the access address corresponding to di in the cache array.

[0051] As a preferred embodiment of the present invention, the FPGA module includes two sets of high-speed data processing units and an epipolar correction unit, which are used to process the left and right images of the binocular camera, respectively.

[0052] As a preferred embodiment of the present invention, the pipeline processing architecture uses a Zynq chip.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0054] This invention proposes a real-time, high-efficiency pipeline architecture for epipolar correction without the need for compressed LUTs. Firstly, it abandons the traditional compressed LUT scheme, reading the image to be corrected in blocks according to a set ratio. Then, a high-speed image cache array based on four dual-port RAM chips is designed and constructed. Address rearrangement transforms random four-point interpolation into parallel reading of cache array data addresses, enabling rapid access to neighboring grayscale values ​​and eliminating the bottleneck of random access to DDR memory in FPGA systems. This invention can be used to achieve high-throughput, low-latency real-time epipolar correction. Secondly, through a resource-saving block prefetching scheme, each correction of a portion of sub-image data allows left and right binocular image pairs (N ≥ 2 frames) to share the same pipeline, significantly increasing throughput with the same logic resources. This meets the stringent requirements of spatiotemporal modulation stereo matching for "millisecond-level multi-frame correction," providing a feasible case for real-time correction of multi-frame spatiotemporal image pairs. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the FPGA pipeline architecture for real-time pole line correction without compression LUT as described in Embodiment 1 of the present invention;

[0056] Figure 2 This is a schematic diagram of the overall design framework of a real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs as described in Embodiment 4 of the present invention.

[0057] Figure 3 This is a schematic diagram of the high-speed data processing unit in a real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs as described in Embodiment 4 of the present invention.

[0058] Figure 4 This is a schematic diagram of the structure of the epipolar correction unit in a real-time epipolar correction FPGA pipeline architecture with uncompressed LUT as described in Embodiment 4 of the present invention.

[0059] Figure 5 This is a schematic diagram of the mapping relationship of the high-speed image buffer array in a real-time epipolar correction FPGA pipeline architecture with uncompressed LUT as described in Embodiment 4 of the present invention.

[0060] Figure 6 This is a schematic diagram of the FPGA implementation architecture of the high-speed image buffer unit in the real-time epipolar correction FPGA pipeline architecture without compression LUT as described in Embodiment 4 of the present invention.

[0061] Figure 7This is a schematic diagram of the computing unit architecture in the epipolar correction unit of a real-time epipolar correction FPGA pipeline architecture with uncompressed LUT as described in Embodiment 4 of the present invention. Detailed Implementation

[0062] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments. All technologies implemented based on the content of the present invention fall within the scope of the present invention. Unless otherwise specified, in the description of the specific embodiments of the present invention, the use of terms such as "first," "second," and "third" is merely for distinguishing the description of the same or similar components, and should not be construed as emphasizing or implying the relative importance of a particular component.

[0063] Example 1

[0064] like Figure 1 As shown, a real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs includes an input port, an output port, an ARM module, and an FPGA module. The FPGA module includes a high-speed data processing unit, an epipolar correction unit, a first DDR3, and a cache array; the ARM module includes a second DDR3.

[0065] The input port is used to receive the image to be corrected, read the sub-image data and corresponding LUT parameters of the image to be corrected, store the sub-image data in the first DDR3, and store the LUT parameters in the second DDR3.

[0066] The high-speed data processing unit is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio, and send them to the epipolar correction unit.

[0067] The epipolar correction unit is used to arrange the read sub-image data according to a set rule and store it into the cache array, and generate the corrected image according to the read LUT parameters.

[0068] The output port is used to receive and output the corrected image.

[0069] The cache array includes a FIFO, four dual-port RAMs, and an address decoding circuit.

[0070] The FIFO's data output interface is 8 bytes, used to communicate with the epipolar correction unit; the four dual-port RAMs are used to store sub-image data arranged according to a set rule; the address decoding circuit is used to convert random interpolation into cache array data addresses for parallel reading through address rearrangement.

[0071] Furthermore, the high-speed data processing unit includes a DDR data high-speed read module, a DDR data high-speed write module, a control state machine, and three AXI buses;

[0072] Of the three AXI buses, one group is responsible for reading sub-image data from the first DDR3, another group reads LUT parameters from the second DDR3, and the last group is responsible for writing the calculation results of the epipolar correction unit into the first DDR3.

[0073] The control state machine is used to receive external commands and control the DDR data high-speed read module and the DDR data high-speed write module.

[0074] The DDR high-speed data read module is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio; and write the data to the data cache unit through the DDR high-speed data write module.

[0075] Furthermore, the ratio set in the high-speed data processing unit is as follows:

[0076] M+1 rows of sub-image data and M rows of LUT parameters, where M = 1 / 3 × the number of rows in one image of the image to be corrected.

[0077] Furthermore, the epipolar correction unit includes a partition correction control state machine, a high-speed image cache module, and a computing module;

[0078] The partition correction control state machine is used to control the high-speed image cache module and the computing module;

[0079] The high-speed image cache module is used to arrange the read sub-image data according to a set rule and store it into the cache array;

[0080] The computing module is used to perform epipolar correction calculations based on the read LUT parameters;

[0081] The setting rules are as follows:

[0082] The sub-image data of the read image to be corrected are converted from horizontal to vertical arrangement and then stored sequentially in the cache array.

[0083] Furthermore, the setting rules in the epipolar correction unit are as follows:

[0084] The sub-image data of the read image to be corrected is converted from horizontal to vertical arrangement and then stored sequentially into the cache array;

[0085] Furthermore, the epipolar correction unit generates the corrected image by including the following steps:

[0086] Traverse the cache array;

[0087] Epipolar correction is performed on each pixel in the sub-image data according to the corresponding LUT parameters. After all pixels have completed epipolar correction, the corrected image is output.

[0088] Furthermore, the epipolar correction step includes:

[0089] Acquire grayscale values ​​of several adjacent pixels around the pixel (in this embodiment, several are greater than or equal to 2, and 4 are used as an example in this embodiment);

[0090] Obtain the LUT parameters of the grayscale values ​​of the adjacent pixels;

[0091] Multiply the gray values ​​of several adjacent pixels by their corresponding LUT parameters, sum the products, and then divide the sum by the set parameters.

[0092] Furthermore, when the number of adjacent pixels is 4, the expression for epipolar correction is:

[0093] Result=(d1×c1+d2×c2+d3×c3+d4×c4) / 256,

[0094] Where Result is the corrected grayscale value of pixel (X, Y), d1, d2, d3, and d4 are the grayscale values ​​of the four neighboring pixels of the pixel (X, Y) to be corrected (arbitrarily positioned in the original image, subject to slow random read / write), and c1, c2, c3, and c4 are the LUT parameters of the pixel (X, Y) to be corrected. Dividing by 256 compresses the weighted sum to the "valid range of image grayscale values ​​(0~255)".

[0095] Furthermore, in this embodiment, the calculation of the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected includes the following steps:

[0096] Obtain the access addresses in the cache array corresponding to the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected;

[0097] Read the corresponding grayscale value based on the access address.

[0098] Furthermore, the calculation of the access address includes the following steps:

[0099] The address parameter Di, i={1,2,3,4}, is calculated using the following expression:

[0100] D1 = Y·W + X,

[0101] D2 = Y·W + X + 1,

[0102] D3 = Y·W + X + W,

[0103] D4 = Y·W + X + W + 1;

[0104] Where X and Y are the horizontal and vertical coordinates of the pixel to be corrected, respectively, and W is the width of the image to be corrected;

[0105] Di is divided by 4, and the quotient and remainder are obtained.

[0106] The calculated quotient is output as the column number of the access address corresponding to di in the cache array, and the remainder is output as the row number of the access address corresponding to di in the cache array.

[0107] Example 2

[0108] The difference between this embodiment and Embodiment 1 is that the FPGA module includes two sets of high-speed data processing units and an epipolar correction unit, which are used to process the left and right images of the binocular camera, respectively.

[0109] Example 3

[0110] This embodiment is a specific implementation of the real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs described in Embodiment 2. This embodiment integrates the real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs into the Zynq series chips.

[0111] Specifically, the Zynq chip in this embodiment includes input / output ports, an ARM terminal, and an FPGA terminal.

[0112] The first DDR3 is configured on the ARM side; the high-speed data processing unit, the polar correction unit, the second DDR3, and the cache array are configured on the FPGA side.

[0113] Furthermore, the Zynq chip includes the following operational steps:

[0114] S1: Use a binocular stereo vision system to acquire N pairs (2N images in total).

[0115] S2: Read the 2N images to be corrected into the DDR3 (i.e., the first DDR3) of the Zynq chip ARM terminal; read the left and right sets of LUT data into the DDR3 (i.e., the second DDR3) of the Zynq chip FPGA terminal.

[0116] S3: On the FPGA side, the high-speed data processing unit first parses the AXI protocol and reads M+1 rows of sub-image data from the ARM side DDR3 using a block prefetching method, reads M rows of LUT parameters from the FPGA side DDR3, and then outputs them to the epipolar correction unit.

[0117] S4: The epipolar correction unit rearranges the M+1 rows of sub-image data and caches them into the 4 RAM blocks (i.e., cache array) inside the FPGA, regenerates a high-speed image cache array suitable for random point access of the image, and then uses the M-row LUT and the high-speed image cache array to design the FPGA implementation architecture of the high-speed image cache unit, and finally generates the M-row epipolar corrected image.

[0118] Due to the limitations of internal storage resources in the FPGA, the high-speed image cache array proposed in this embodiment can support a maximum of M+1 lines of image data correction at a time. Preferably, M is 1 / 3 of the data of the entire image. For example, when outputting a 1000×800 resolution corrected image, M can be set to 400 lines to achieve calibration of 300 lines of image data each time. Therefore, calibrating a complete image requires reading data and correcting it 3 times.

[0119] S5: The corrected images are sequentially output to DDR3 on the ARM side.

[0120] Furthermore, the high-speed data processing unit includes a DDR data high-speed read module, a DDR data high-speed write module, and a control state machine. The control state machine is responsible for receiving external (CPU-side) commands and controlling the commands of each module within this unit. The DDR data high-speed read module includes an AXI protocol parsing unit, a data read unit, and a data cache. The DDR data high-speed write module includes a data cache, a data write unit, and an AXI protocol parsing unit. This unit uses three AXI buses to communicate with the ARM / FPGA DDR3: one bus reads image data from the ARM DDR3, one bus reads LUTs from the FPGA DDR3, and the third bus writes the epipolar correction unit's calculation results to the ARM DDR3.

[0121] Because this invention requires rapid reading and writing of large amounts of DDR3 data from the external ARM / FPGA side of the ZYNQ chip, and the entire polar correction module on the FPGA side is connected to the AXI bus, this unit is designed with an AXI protocol parsing unit specifically responsible for parsing the AXI bus protocol. The AXI protocol has five channels: read address channel, write address channel, read data channel, write data channel, and write acknowledge channel; each channel corresponds to a different timing sequence. The AXI protocol parsing unit performs AXI protocol parsing on all channels. After the protocol parsing is complete and the DDR3 data is obtained, the data is written to the cache array for use by the polar correction unit.

[0122] Furthermore, the epipolar correction unit includes a partition correction control state machine, a high-speed image cache module, and a computing module. The partition correction control state machine is responsible for scheduling the unit, the computing module uses the FPGA's internal DSP unit to perform correction calculations, and the high-speed image cache module is used to cache the original sub-image (M+1 rows) using a high-speed image cache array, enabling fast access to random points in the image.

[0123] Furthermore, due to the low random read / write efficiency of DDR3, directly searching for random pixels in the original image sequence results in extremely slow access speeds. While caching the entire image in the FPGA's internal RAM is feasible, it consumes a significant amount of FPGA resources. Therefore, this invention innovatively designs a high-speed image cache array suitable for random point access in the epipolar correction unit to meet the real-time correction requirements of binocular cameras. This scheme can guarantee a fast read of the grayscale values ​​d1, d2, d3, and d4 of four adjacent pixels at any position, thus solving the bottleneck of huge resource consumption for random point access on the FPGA. Specifically, the array re-caches the pre-fetched M+1 rows of sub-image data in a vertical arrangement into four dual-port RAMs within the FPGA to achieve fast access to random pixels, thereby improving the read / write speed of the uncompressed LUT.

[0124] Furthermore, for the point to be matched (X, Y), its four neighboring pixel grayscale values ​​d1, d2, d3, and d4, the corresponding epipolar correction formula is expressed as follows:

[0125] Result=(d1×c1+d2×c2+d3×c3+d4×c4) / 256,

[0126] Wherein, Result is the grayscale value of pixel (X, Y) after correction, d1, d2, d3, and d4 are the grayscale values ​​of the four neighboring pixels of the pixel (X, Y) to be corrected (these grayscale information are four grayscale values ​​at any position in the original image arrangement, and their access belongs to random read and write). Through a high-speed image cache array and characteristic mapping algorithm, the grayscale values ​​of four adjacent pixels at any position can be read quickly at one time, and c1, c2, c3, and c4 are the LUT parameters of the pixel (X, Y) to be corrected (i.e., the parameters used to calibrate (X, Y) obtained through camera calibration).

[0127] Furthermore, the calculation of the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected includes the following steps:

[0128] Obtain the access addresses corresponding to the gray values ​​d1, d2, d3, and d4 of the four pixels in the neighborhood of the pixel to be corrected (X, Y).

[0129] Read the corresponding grayscale value based on the access address.

[0130] Furthermore, the calculation of the access addresses corresponding to the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected includes the following steps:

[0131] The address parameter Di, i={1,2,3,4}, is calculated using the following expression:

[0132] D1 = Y·W + X,

[0133] D2 = Y·W + X + 1,

[0134] D3 = Y·W + X + W,

[0135] D4 = Y·W + X + W + 1;

[0136] Where X and Y are the horizontal and vertical coordinates of the pixel to be corrected, respectively, and W is the width of the image to be corrected;

[0137] Di is divided by 4, and the quotient and remainder are obtained.

[0138] The calculated quotient is output as the column number of the access address corresponding to di, and the remainder is output as the row number of the access address corresponding to di.

[0139] For example, when D1=0, the quotient q1=0 and the remainder r1=0, indicating that the value of d1 is located in column 0 of address and row 0 of RAM in the high-speed image cache array.

[0140] Furthermore, in this embodiment, the high-speed image cache array has the following storage rules: the first row is the ADDR address row, and the second to fifth rows are the RAM0 to RAM3 storage rows; the original horizontal image rows are transformed into a vertical arrangement on the RAM0 to RAM3 addresses until the M+1 rows of data are arranged.

[0141] For example, pixels 0, 1, 2, and 3 in the first row of the original image are rearranged in the ADDR0 address column, and pixels 4, 5, 6, and 7 are rearranged in the ADDR1 address column.

[0142] Furthermore, the cache array is an image block cache area composed of four dual-port on-chip RAMs, capable of caching a maximum of M rows of images. When an image is read from the ARM DDR3, it is first cached in a FIFO. This FIFO has an 8-byte data output interface, corresponding to the two input ports of the four RAMs. After passing through the address decoding circuit, it enters the high-speed image cache unit for caching using the high-speed image cache array. Through this special arrangement and dual-port RAM structure, given any (X, Y), the corresponding grayscale values ​​d1, d2, d3, and d4 can be obtained through address decoding.

[0143] Furthermore, this embodiment includes two high-speed data processing units and an epipolar correction unit, which can simultaneously correct the left and right images of the binocular camera to improve correction speed. This is mainly due to the fine-grained scheduling of resources through block prefetching, which allows the two data processing units to run at full capacity in parallel, resulting in a significant increase in throughput.

[0144] To address the real-time computational challenge of epipolar correction in spatiotemporal modulation stereo matching technology, this invention proposes a real-time, high-efficiency pipeline architecture for epipolar correction that eliminates the need for compressed LUTs. First, it abandons the traditional compressed LUT scheme, significantly improving LUT access speed by regenerating a high-speed image cache array suitable for random point access in images. This eliminates the round-trip latency and logical overhead of compressed / decompressed LUTs, resulting in higher read / write efficiency. Second, it employs a high-speed cache array consisting of "block prefetching + 4 dual-interface RAMs." An effective formula converts random four-point interpolation into parallel reading of cache array data addresses, enabling the acquisition of neighboring pixel grayscale values ​​in a single cycle and completely eliminating the DDR3 random access bottleneck. Furthermore, through the resource-saving block prefetching scheme, left and right binocular image pairs (N ≥ 2 frames) can share the same pipeline. By using dual data processing units and a time-division multiplexed DSP array, throughput is significantly increased under the same logical resources, meeting the stringent requirements of spatiotemporal modulation stereo matching for "millisecond-level multi-frame correction," providing a feasible case for real-time correction of multi-frame spatiotemporal image pairs.

[0145] Example 4

[0146] This embodiment is a specific simulation application example of the real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs described in Embodiment 3, such as... Figure 2 As shown, this includes data input, a real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs, and image output. This embodiment implements epipolar correction based on Xilinx's Zynq XC7Z100 platform, which uses an ARM+FPGA architecture. The specific steps in implementing this embodiment are as follows:

[0147] S1. A 3D camera using speckle structured light acquires 3 pairs (6 images in total) of speckle images with a resolution of 1024×1280.

[0148] S2. Read the 6 images to be corrected into DDR3 (i.e., the first DDR3) at the ARM end of the Zynq chip; read the left and right sets of LUT parameters into DDR3 (i.e., the second DDR3) at the FPGA end of the Zynq chip.

[0149] S3. On the FPGA side, the high-speed data processing unit first parses the AXI protocol, reads 500 lines of image data from the ARM side DDR3, reads 300 lines of LUT parameters from the FPGA side DDR3, and then outputs them to the epipolar correction unit.

[0150] S4. The epipolar correction unit rearranges 400 lines of image data and caches them into 4 RAM blocks (i.e., cache arrays) inside the FPGA. It then regenerates a high-speed image cache array suitable for random point access of the image, and finally uses 300 lines of LUT parameters and the high-speed image cache array to generate 300 lines of epipolar corrected image.

[0151] Specifically, the processing architecture in this embodiment includes two high-speed data processing units and an epipolar correction unit, which can simultaneously correct the left and right images of the binocular camera to improve the correction speed. When the FPGA receives a command from the ARM, the correction module begins (or interrupts) image correction.

[0152] S3-1, the high-speed data processing unit as follows: Figure 3 As shown.

[0153] This unit mainly includes high-speed DDR data reading, high-speed DDR data writing, and a control state machine. The control state machine is responsible for receiving external (CPU-side) commands and controlling the commands of each module within this unit. High-speed DDR data reading includes an AXI protocol parsing unit, a data reading unit, and a data cache. High-speed DDR data writing includes a data cache, a data writing unit, and an AXI protocol parsing unit. This unit uses three AXI buses to communicate with the ARM / FPGA DDR3: one bus reads image data from the ARM DDR3, one bus reads LUTs from the FPGA DDR3, and the third bus writes the epipolar correction unit's calculation results to the ARM DDR3.

[0154] Furthermore, since this invention requires rapid reading and writing of large amounts of DDR3 data from the external ARM / FPGA side of the ZYNQ chip, and the entire polar correction module on the FPGA side is connected to the AXI bus, this unit is designed with an AXI protocol parsing unit specifically responsible for parsing the AXI bus protocol. The AXI bus is divided into five channels: read address channel, write address channel, read data channel, write data channel, and write acknowledge channel; each channel corresponds to a different timing sequence. The AXI protocol parsing unit performs AXI protocol parsing for all channels. After the protocol parsing is completed and the DDR3 data is obtained, the data is written to the data buffer unit for use by the polar correction unit.

[0155] S3-2, the polar correction unit as follows: Figure 4As shown, it includes a partition correction control state machine, a high-speed image buffer unit, and a calculation unit. Image data and LUT parameters read from the high-speed data processing unit are input into this unit for correction.

[0156] First, the LUT parameter structure stored in DDR3 on the FPGA side is as follows:

[0157] {X,Y,C1,C2,C3,C4},

[0158] Where (X, Y) represents the pixel to be corrected. Since a 1000×800 resolution image is desired, X and Y each require 2 bytes, while C1, C2, C3, and C4 are parameters (known parameters) obtained through camera calibration used to calibrate (X, Y), each occupying 1 byte. Therefore, to correct one (X, Y) pixel, 8 bytes of LUT parameters need to be stored (so outputting a 1000×800 resolution image requires 0.8×8 = 6.4 M DDR). The correction formula is as follows:

[0159] Result=(d1×c1+d2×c2+d3×c3+d4×c4) / 256,

[0160] Where d1, d2, d3, and d4 are the grayscale values ​​of the four pixels in the (X, Y) neighborhood of the pixel to be corrected, and Result represents the corrected grayscale value. The key to improving the algorithm's speed lies in how quickly the grayscale information of d1, d2, d3, and d4 can be obtained. Based on this, the following considerations are made:

[0161] The grayscale information of d1, d2, d3, and d4 represents four grayscale values ​​at arbitrary positions in the original image arrangement, and their access is a random read / write operation. Furthermore, DDR3 random read / write efficiency is low; directly searching for random pixels in the original image sequence for reading results in extremely slow access speeds. While caching the entire image in the FPGA's internal RAM is feasible, it consumes enormous FPGA resources. Therefore, this invention innovatively designs a high-speed image cache array suitable for random point access in the epipolar correction unit to meet the real-time correction requirements of a binocular camera. This solution guarantees a fast read of the grayscale values ​​d1, d2, d3, and d4 of four adjacent pixels at any position, thus overcoming the bottleneck of excessive resource consumption for random point access on the FPGA.

[0162] (1) Design principle of high-speed image buffer array

[0163] The grayscale information of d1, d2, d3, and d4 at any position in the original image arrangement is as follows: Figure 5As shown in (a) above. For example, if a corrected image with a resolution of 1000×800 is to be output, then the four adjacent pixel positions corresponding to the first pixel (0, 0) are 0, 1, 800, and 801. In order to enable the FPGA to quickly access the positions of these pixels, the epipolar correction unit (202) rearranges the 500 rows of image data and caches them into the four RAMs inside the FPGA to generate a high-speed image cache array, as shown in (a). Figure 5 As shown in (b), according to the storage rules of the high-speed image cache array, pixels 0, 1, 2, and 3 in the first row of the original image are rearranged in the ADDR0 address column, pixels 4, 5, 6, and 7 are rearranged in the ADDR1 address column, and so on. Figure 5 As shown. Therefore, the access correspondence between the two can be described as:

[0164] For an image with H rows and W columns, the formula for calculating the addresses corresponding to the gray values ​​d1, d2, d3, and d4 is:

[0165] D1 = Y·W + X,

[0166] D2 = Y·W + X + 1,

[0167] D3 = Y·W + X + W,

[0168] D4 = Y·W + X + W + 1;

[0169] Where X and Y are the horizontal and vertical coordinates of the pixel to be corrected, respectively, and W is the width of the image to be corrected.

[0170] Then, divide each of the calculated D1, D2, D3, and D4 by 4, and take the quotient and remainder to obtain the result. Figure 5 The exact access location corresponding to (b) in the text.

[0171] Where the calculated quotient qi is the corresponding ADDR address column in the high-speed image cache array, and the remainder ri represents the corresponding RAM chip. For an output image with W=800 columns, D1=0, D2=1, D3=800, and D4=801 are calculated using the above formula. For example, when calibrating the first pixel (0, 0) of the image, its position in the original image and its corresponding adjacent pixel is determined by... Figure 5 (a) in the middle changed to Figure 5 (b) (gray-filled square), that is:

[0172] When D1=0, q1=0, r1=0, it means that the value of d1 is located in column 0 of address and row 0 of RAM in the high-speed image cache array;

[0173] When D1=1, q1=0, r1=1, it means that the value of d1 is located in column 0 of address and row 1 of RAM in the high-speed image cache array;

[0174] When D1=800, q1=200, and r1=0, it means that the value of d1 is located in column 0 of address and row 0 of RAM in the high-speed image cache array;

[0175] When D1=801, q1=200, and r1=1, it means that the value of d1 is located in column 0 of address and row 1 of RAM in the high-speed image cache array.

[0176] (2) FPGA implementation of high-speed image buffer unit

[0177] The FPGA implementation structure of the high-speed image buffer unit is as follows: Figure 6 As shown, this array has two functions: caching of the original sub-images (400 rows) using a high-speed image cache array, and fast access to random points in the image. Internally, this unit contains an image block cache area composed of four dual-port on-chip RAMs, capable of caching up to 300 rows of images. When an image is read from the ARM's DDR3, it is first cached in a FIFO. This FIFO has an 8-byte data output interface, corresponding to the two input ports of the four RAMs. After passing through the address decoding circuit, it enters the high-speed image cache unit. The FPGA structure of this unit is shown below. Figure 6 As shown, the pixels of the image are arranged vertically. With this special arrangement and the dual-port RAM structure, given any (X, Y), the corresponding gray values ​​d1, d2, d3, and d4 can be obtained through the address decoding circuit.

[0178] Furthermore, in the epipolar correction unit, the partitioned correction control state machine is responsible for the scheduling of this unit, while the calculation unit uses the FPGA's internal DSP unit to complete the correction calculations, such as... Figure 7 As shown, this unit completes the computational task of address access.

[0179] S5. The corrected image (resolution 1000×800) is sequentially output to the DDR3 of the Zynq chip's ARM side.

[0180] Using the architecture described in this embodiment in an FPGA can greatly improve the correction speed. Each stereo matching requires three frames of images, while the solution of this invention can achieve 40 FPS / s, 3*40=120, that is, to achieve epipolar correction of 120 images per second.

[0181] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs, comprising input ports, output ports, an ARM module, and an FPGA module, characterized in that, The FPGA module includes a high-speed data processing unit, a polarity correction unit, a first DDR3, and a cache array; the ARM module includes a second DDR3. The input port is used to receive the image to be corrected, read the sub-image data and corresponding LUT parameters of the image to be corrected, store the sub-image data in the first DDR3, and store the LUT parameters in the second DDR3; The high-speed data processing unit is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio, and send them to the epipolar correction unit. The epipolar correction unit is used to arrange the read sub-image data according to a set rule and store it into the cache array, and generate the corrected image according to the read LUT parameters; The output port is used to receive and output the corrected image; The cache array includes a FIFO, four dual-port RAMs, and an address decoding circuit. The FIFO's data output interface is 8 bytes, used to communicate with the epipolar correction unit; the four dual-port RAMs are used to store sub-image data arranged according to a set rule; the address decoding circuit is used to convert random interpolation into cache array data addresses for parallel reading through address rearrangement.

2. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 1, characterized in that, The high-speed data processing unit includes a DDR data high-speed read module, a DDR data high-speed write module, a control state machine, and three AXI buses; Of the three AXI buses, one group is responsible for reading sub-image data from the first DDR3, another group reads LUT parameters from the second DDR3, and the last group is responsible for writing the calculation results of the epipolar correction unit into the first DDR3. The control state machine is used to receive external commands and control the DDR data high-speed read module and the DDR data high-speed write module. The DDR high-speed data read module is used to read sub-image data from the first DDR3 and LUT parameters from the second DDR3 according to a set ratio; and write the data to the data cache unit through the DDR high-speed data write module.

3. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 2, characterized in that, The ratio set in the high-speed data processing unit is: M+1 rows of sub-image data and M rows of LUT parameters, where M = 1 / 3 × the number of rows in one image of the image to be corrected.

4. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 1, characterized in that, The epipolar correction unit includes a partition correction control state machine, a high-speed image cache module, and a computing module; The partition correction control state machine is used to control the high-speed image cache module and the computing module; The high-speed image cache module is used to arrange the read sub-image data according to a set rule and store it into the cache array; The computing module is used to perform epipolar correction calculations based on the read LUT parameters; The setting rules are as follows: The sub-image data of the read image to be corrected are converted from horizontal to vertical arrangement and then stored sequentially in the cache array.

5. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 4, characterized in that, The epipolar correction unit generates the corrected image by the following steps: Traverse the cache array; Epipolar correction is performed on each pixel in the sub-image data according to the corresponding LUT parameters. After all pixels have completed epipolar correction, the corrected image is output.

6. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 5, characterized in that, The polar correction steps include: Acquire grayscale values ​​of several adjacent pixels around the pixel; Obtain the LUT parameters of the grayscale values ​​of the adjacent pixels; Multiply the gray values ​​of several adjacent pixels by their corresponding LUT parameters, sum the products, and then divide the sum by the set parameter.

7. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 6, characterized in that, When the number of adjacent pixels is 4, the expression for epipolar correction is: Result=(d1×c1+d2×c2+d3×c3+d4×c4) / 256, Where Result is the corrected grayscale value of pixel (X, Y), d1, d2, d3, and d4 are the grayscale values ​​of the four neighboring pixels around the pixel (X, Y) to be corrected, and c1, c2, c3, and c4 are the LUT parameters of the pixel (X, Y) to be corrected.

8. A real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 7, characterized in that, The calculation of the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected includes the following steps: Obtain the access addresses in the cache array corresponding to the gray values ​​d1, d2, d3, and d4 of the four pixels in the (X, Y) neighborhood of the pixel to be corrected; Read the corresponding grayscale value based on the access address.

9. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 8, characterized in that, The calculation of the corresponding access address in the cache array includes the following steps: The address parameter Di, i={1,2,3,4}, is calculated using the following expression: D1 = Y·W + X, D2 = Y·W + X + 1, D3 = Y·W + X + W, D4 = Y·W + X + W + 1; Where X and Y are the horizontal and vertical coordinates of the pixel to be corrected, respectively, and W is the width of the image to be corrected; Di is divided by 4, and the quotient and remainder are obtained. The calculated quotient is output as the column number of the access address corresponding to di in the cache array, and the remainder is output as the row number of the access address corresponding to di in the cache array.

10. The real-time epipolar correction FPGA pipeline architecture with uncompressed LUTs according to claim 1, characterized in that, The FPGA module includes two sets of high-speed data processing units and an epipolar correction unit, which are used to process the left and right images of the binocular camera, respectively.

Images