Ultrahigh-resolution video real-time distortion correction device based on FPGA

Abstract

The invention relates to an ultra-high resolution video real-time distortion correction device based on an FPGA (Field Programmable Gate Array), which belongs to the technical field of video image processing and comprises a distortion correction calculation unit, a Flash memory, a DDR4 memory and a QDR II + SRAM (Quadrature Data Rate II + Static Random Access Memory) memory which run on an FPGA chip in a helmet display system, and the distortion correction calculation unit is used for distortion correction of image data processed by a GPU (Graphic Processing Unit) on a helmet; the Flash memory pre-stores a lookup table for distortion correction which is sequenced according to the sequence of input pixels, and the lookup table is used for querying the mapping relation between the input pixels and the output pixels; the DDR4 memory is used for reading the lookup table in real time in the operation process of the distortion correction calculation unit; and the QDR II + SRAM memory is used for performing pixel caching and outputting on the data calculated by the distortion correction calculation unit, so that the problem that the pixel calculation sequence and the pixel output sequence are inconsistent is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing methods, and in particular relates to an ultra-high resolution video real-time distortion correction device based on FPGA. Background Art

[0002] With the development of display technology and imaging technology, more and more ultra-high resolution displays and imaging sensors are applied to airborne display and imaging equipment. At present, the new generation of large field of view optical systems have been applied to the design of helmet displays. Under the premise of meeting the angular resolution requirements, improving the field of view of the helmet display must increase the resolution of the helmet display input video, so ultra-high resolution display technology must be applied. On the other hand, due to the inherent distortion of the current off-axis optical system, pre-distortion processing must be performed by digital methods to make the video displayed on the helmet display undistorted. It is necessary to design an ultra-high resolution video real-time distortion correction method and device based on FPGA, which can meet the correction of the inherent distortion of the optical display system while ensuring the real-time display of the helmet display, resulting in low image display quality of the helmet display. Summary of the invention

[0003] In view of this, the present invention provides an ultra-high resolution video real-time distortion correction device based on FPGA, which reduces the delay of ultra-high resolution video distortion correction and reduces the resource consumption of FPGA by the distortion correction system.

[0004] An ultra-high-resolution video real-time distortion correction device based on FPGA is suitable for image display of an airborne helmet display, including a distortion correction calculation unit and a Flash memory, a DDR4 memory and a QDR II+SRAM memory running on an FPGA chip in a helmet display system, wherein:

[0005] The distortion correction calculation unit is used for distortion correction of image data processed by the GPU processor on the helmet;

[0006] The Flash memory pre-stores a distortion correction lookup table sorted in the order of input pixels, and the lookup table is used to query the mapping relationship between input pixels and output pixels;

[0007] The DDR4 memory is used to read the lookup table in real time during the operation of the distortion correction calculation unit;

[0008] The QDR II+SRAM memory is used for pixel buffering and outputting the data calculated by the distortion correction calculation unit.

[0009] The technical beneficial effects of the present invention are:

[0010] By building a multi-pixel binding parallel computing unit on the FPGA and pre-sorting the lookup table according to the input pixel order, the internal storage resources of the FPGA can meet the pre-interpolation cache consumption under ultra-high-resolution video input, and at the same time ensure that multiple pixel values ​​calculated in parallel can be written to the same address of the external QDR II+SRAM. Through the external high-bandwidth QDR II+SRAM, the interpolated pixels are cached, effectively solving the problem of inconsistent pixel calculation order and pixel output order. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 This is a hardware block diagram of the ultra-high resolution video real-time distortion correction device of the present invention;

[0013] Figure 2 This is a structural diagram of the ultra-high resolution real-time distortion correction calculation unit of the present invention;

[0014] Figure 3 This is an embodiment of the front-level cache unit in the present invention;

[0015] Figure 4 It is an equivalent model of the QDR II+SRAM pixel cache unit in the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0017] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0018] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0019] like Figures 1 to 4 The FPGA-based ultra-high-resolution video real-time distortion correction device shown is suitable for image display of an airborne helmet display, including a distortion correction calculation unit and a Flash memory, a DDR4 memory, and a QDR II+SRAM memory running on an FPGA chip in a helmet display system, wherein:

[0020] The distortion correction calculation unit is used for distortion correction of image data processed by the GPU processor on the helmet, and is also used for selecting the pixel parallelism N according to the video pixel clock of the image data processed by the GPU processor on the helmet, where the pixel parallelism N is the number of pixels processed at one time, such as N is 1, 2, 4, 8, etc.;

[0021] The Flash memory pre-stores a distortion correction lookup table sorted according to the order of input pixels, and the lookup table is used to query the mapping relationship between input pixels and output pixels;

[0022] DDR4 memory is used to read the lookup table in real time during the operation of the distortion correction calculation unit;

[0023] The QDR II+SRAM memory is used for pixel buffering and outputting the data calculated by the distortion correction calculation unit.

[0024] As a specific implementation method provided in this case, Figure 2 The distortion correction calculation unit shown includes an HDMI / DP input video decoding unit, a Flash reading unit, a DDR4 read-write control unit, N pre-caching units and N interpolation calculation units, a QDR write address calculation unit, a QDR read address calculation unit, a QDR read-write control unit and an HDMI / DP output video encoding unit, wherein:

[0025] The HDMI / DP input video decoding unit is used to decode the video inputted from the HDMI / DP interface and convert it into a parallel video with a pixel parallelism of N;

[0026] The Flash read unit is used to read the lookup table from the Flash memory and send it to the DDR4 read-write control unit;

[0027] The DDR4 read / write control unit is used to write the lookup table read by the Flash read unit into the DDR4 SDRAM memory, and sequentially read and write the lookup table of the DDR4 read / write control unit during the distortion calculation process of each frame of video, and send it to N front-level cache units, N interpolation calculation units and QDR write address calculation units respectively;

[0028] N front-stage cache units are used to cache the parallel video output by the HDMI / DP input video decoding unit, and look up the corresponding input pixel value according to the lookup table read by the DDR4 read-write control unit, and send it to N interpolation calculation units. For example, within one line of the current input video, all the lookup table items mapped to the previous line of input pixels are calculated. At this time, the number of video lines cached by the front-stage cache unit can be small enough. Considering that the front-stage cache unit needs to cover N pixels calculated at the same time and the continuity of the image distortion itself, the front-stage cache unit does not exceed N lines at most;

[0029] N interpolation calculation units perform interpolation calculations based on the input pixel values ​​found by the N previous cache units and the lookup table read by the DDR4 read-write control unit, and output the final N output pixels, wherein one previous cache unit and one interpolation calculation unit form a calculation pipeline, and each calculation pipeline uses bilinear interpolation for calculation;

[0030] The QDR write address calculation unit is used to calculate the addresses of the final N output pixels corresponding to the QDR memory according to the lookup table read by the DDR4 read-write control unit;

[0031] The QDR read address calculation unit generates an output video timing after delaying a specific number of video lines according to the parallel video timing output by the HDMI / DP input video decoding unit, and calculates the address in the QDR memory corresponding to the encoded output pixel according to the output video timing;

[0032] The QDR read / write control unit is used to write the final N output pixels into the QDR memory according to the write address output by the QDR write address calculation unit, and read out the coded output pixel according to the address in the QDR memory corresponding to the coded output pixel output by the QDR read address calculation unit;

[0033] The HDMI / DP output video encoding unit is used to perform video encoding on the encoded output pixels output by the QDR read / write control unit to output HDMI / DP video, and send it to the helmet display for display.

[0034] In general, since image distortion may cause multiple output pixels to be mapped near the original single input pixel, in order to ensure that all output pixels mapped to the previous row of input pixels are calculated within one row of the input video, the operating frequency of the calculation pipeline can be set higher than the pixel clock of the parallel video. The external input HDMI / DP ultra-high-resolution video is decoded by the HDMI / DP input video decoding unit and converted into a parallel video with a pixel parallelism of N. The Flash reading unit is responsible for reading the lookup table from the Flash and sending it to the DDR4 read-write control unit to write it into the DDR4 SDRAM memory. Each pre-level cache unit and interpolation calculation unit form a calculation pipeline, which is responsible for calculating an output pixel in the lookup table entry. According to the output pixel row and column values ​​of the lookup table entry, the QDR storage address corresponding to the output pixel is calculated, and the output pixels calculated on the N calculation pipelines are simultaneously written into the QDR II+SRAM. According to the input video timing, after a specific number of rows are delayed, the output timing is generated, and the output pixel values ​​are sequentially read from the QDR II+SRAM according to the output timing, and sent to the HDMI / DP video output video encoding unit to generate the output video.

[0035] As a specific implementation method provided in this case, the distortion correction lookup table stored in the Flash memory stores the position mapping relationship between the input video pixels and the output video pixels before and after the distortion correction, and each lookup table item in the lookup table corresponds to N output pixels; wherein, the order of the lookup table items is sorted according to the row and column of the largest input pixel corresponding to each lookup table item. It is composed of (input pixel row 1, input pixel column 1, row-wise interpolation coefficient 1, column-wise interpolation coefficient 1, input pixel row 2, input pixel column 2, row-wise interpolation coefficient 2, column-wise interpolation coefficient 2, ..., input pixel row N, input pixel column N, row-wise interpolation coefficient N, column-wise interpolation coefficient N, output pixel row, output pixel column) 4N+2 items, corresponding to the mapping relationship between the adjacent N pixels and the input pixel starting from the N*K+1th (K=0, 1, 2...)th pixel of each output row.

[0036] Furthermore, the order of the lookup table entries is sorted by the row and column corresponding to the largest input pixel of each lookup table entry, including,

[0037] When the row count of the input video exceeds the row and column of the largest input pixel of the current lookup table item, each computing pipeline can read the pixel value of the input pixel and its neighborhood corresponding to the current lookup table item from the previous cache unit, perform interpolation calculation based on the interpolation coefficient, and obtain the pixel value of the output pixel corresponding to the pipeline in the current lookup table item.

[0038] Since the order of the lookup table items is sorted according to the row and column of the largest input pixel corresponding to each lookup table item, when the row count of the input video exceeds the row and column of the largest input pixel of the current lookup table item, each computing pipeline can read the pixel value of the input pixel corresponding to the current lookup table item and its neighborhood from the previous cache unit, perform interpolation calculation according to the interpolation coefficient, and obtain the pixel value of the output pixel corresponding to the pipeline in the current lookup table item.

[0039] Within one line of the current input video, all lookup table entries mapped to the previous line of input pixels are calculated. At this time, the number of video lines cached by the front-level cache unit can be small enough. Considering that the front-level cache unit needs to cover N pixels calculated at the same time and the continuity of the image distortion itself, the front-level cache unit does not exceed N lines at most. The N consecutive output pixels corresponding to a lookup table entry are calculated simultaneously on N calculation pipelines. According to the row and column values ​​of the first pixel in these N output pixels and the input video resolution, the write address of the QDR can be calculated, and these N output pixels can be written to the continuous addresses of the QDR. According to the output timing, the continuous N output pixels are read out from the QDR in sequence and sent to the HDMI / DP output video encoding unit to output the HDMI / DP video. Since image distortion may cause multiple output pixels to be mapped to the vicinity of the original single input pixel, in order to ensure that all output pixels mapped to the previous line of input pixels are calculated within one line of the input video, the operating frequency of the calculation pipeline can be set higher than the pixel clock of the parallel video.

[0040] Example

[0041] The present invention is further introduced below using a 4K@120Hz 12bit (3840*2160@120Hz) video distortion correction embodiment. The pixel clock of a 3840*2160@120Hz video is 1188MHz. Considering that the video processing pipeline designed on an FPGA can reach up to 300MHz or more, a computational pipeline with N=4 pixels in parallel is selected. When 4 pixels are in parallel, the corresponding parallel pixel clock is 297MHz.

[0042] One lookup table entry corresponds to four output pixels, including (input pixel row 1, input pixel column 1, row interpolation coefficient 1, column interpolation coefficient 1, input pixel row 2, input pixel column 2, row interpolation coefficient 2, column interpolation coefficient 2, ..., input pixel row N, input pixel column N, row interpolation coefficient N, column interpolation coefficient N, output pixel row, output pixel column). Considering that both the input pixel row and column values ​​and the output pixel row and column values ​​are quantized with 16 bits, and the row interpolation coefficients and column interpolation coefficients are quantized with 8 bits, the total number of bits of one lookup table entry is 224 bits. The total lookup table size is 3840*2160 / 4*224=453.6Mbit, such as Figure 1As shown in the figure, the use of 1Gbit QSPIFlash can meet the non-volatile storage requirements of the lookup table. When 4 pixels are calculated in parallel, the total read bandwidth of the lookup table can reach 297*224=66528Mbit / s. When the 32-bit DDR4 runs at a rate of 2666Mbps, it can meet the bandwidth requirements.

[0043] like Figure 2 As shown, the external 3840*2160@120Hz video is input through the HDMI / DP interface and converted into a 4-pixel parallel video with a pixel clock of 297MHz after being decoded by the HDMI / DP input video decoding unit. The Flash reading unit is responsible for reading the lookup table from the Flash and sending it to the DDR4 read / write control unit to write it into the DDR4SDRAM memory. Each pre-level cache unit and interpolation calculation unit form a calculation pipeline, which is responsible for calculating an output pixel in the lookup table entry. Each calculation pipeline uses bilinear interpolation for calculation. At this time, Figure 3 As shown, the front-level cache unit can be designed as a storage unit of 8 memory banks, corresponding to the even row modulo 4 remainder 0 storage bank, the even row modulo 4 remainder 1 storage bank, the even row modulo 4 remainder 2 storage bank, the even row modulo 4 remainder 3 storage bank, the odd row modulo 4 remainder 0 storage bank, the odd row modulo 4 remainder 1 storage bank, the odd row modulo 4 remainder 2 storage bank, and the odd row modulo 4 remainder 3 storage bank. The 8 storage banks can ensure that when 4 pixels are input in parallel, four storage banks can be written at the same time, and at the same time, 4 adjacent pixels can be read out to participate in the bilinear interpolation calculation of each pipeline.

[0044] The four computation pipelines are responsible for calculating one output pixel in the lookup table entry. For 12-bit color pixel values, the total pixel data bit width calculated in a single cycle is 12*3*4=144 bits. Using a 2*18-bit wide QDR II+SRAM, the maximum operating frequency can reach 632MHz. Its equivalent read-write model is shown in Figure 4, which can be equivalent to a dual-port SRAM that can operate at a frequency of 316MHz. Its read-write port bit width is 36*4=144 bits, and the storage depth is 4M. It can meet the bandwidth requirements of 4 pixels written in parallel in the computation pipeline. The write address of QDR can be calculated as (3840*TY+TX) / 4 based on the output pixel row TY and the output pixel column TX. According to the input video timing, after a specific number of rows are delayed, the output timing is generated. According to the output timing, the output pixel value is read from the QDR II+SRAM in address accumulation order, and sent to the HDMI / DP video output video encoding unit to generate the output video.

[0045] The total storage size of the pre-caching unit corresponds to 4 rows of pixels, and each storage unit corresponds to 0.5 rows of pixels. The designed parallel computing pipeline clock matches the interface clock of QDR, which is 316MHz, which can effectively meet the burst computing throughput requirements when multiple output pixels are mapped to a single input pixel. The maximum storage depth of QDR is 4M, which can save 1080 rows of output video, and the corresponding output video can be recycled, that is, the 1080th row starts again from address 0.

[0046] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An ultra-high resolution video real-time distortion correction device based on FPGA, suitable for image display of airborne helmet display, characterized in that: The invention comprises a distortion correction calculation unit and a Flash memory, a DDR4 memory and a QDR II+SRAM memory running on an FPGA chip in a helmet display system, wherein: The distortion correction calculation unit is used for distortion correction of image data processed by the GPU processor on the helmet; The Flash memory pre-stores a distortion correction lookup table sorted in the order of input pixels, and the lookup table is used to query the mapping relationship between input pixels and output pixels; The DDR4 memory is used to read the lookup table in real time during the operation of the distortion correction calculation unit; The QDR II+SRAM memory is used for pixel caching and outputting the data calculated by the distortion correction calculation unit.

2. The ultra-high resolution video real-time distortion correction device according to claim 1, characterized in that: The distortion correction calculation unit is also used to select a pixel parallelism N according to a video pixel clock of image data processed by a GPU processor on the helmet, where the pixel parallelism N is the number of pixels processed in one calculation.

3. The ultra-high resolution video real-time distortion correction device according to claim 2, characterized in that: The distortion correction calculation unit includes an HDMI / DP input video decoding unit, a Flash reading unit, a DDR4 read-write control unit, N pre-stage cache units and N interpolation calculation units, a QDR write address calculation unit, a QDR read address calculation unit, a QDR read-write control unit and an HDMI / DP output video encoding unit, wherein: The HDMI / DP input video decoding unit is used to decode the video inputted from the HDMI / DP interface and convert it into a parallel video with a pixel parallelism of N; The Flash reading unit is used to read the lookup table from the Flash memory and send it to the DDR4 read-write control unit; The DDR4 read-write control unit is used to write the lookup table read by the Flash read unit into the DDR4 SDRAM memory, and sequentially read and write the lookup table of the DDR4 read-write control unit during the distortion calculation process of each frame of video, and send it to N previous cache units, N interpolation calculation units and QDR write address calculation units respectively; The N front-stage cache units are used to cache the parallel video output by the HDMI / DP input video decoding unit, and search for the corresponding input pixel value according to the lookup table read by the DDR4 read-write control unit, and send it to the N interpolation calculation units; The N interpolation calculation units perform interpolation calculations according to the input pixel values ​​found by the N previous cache units and the lookup table read by the DDR4 read-write control unit, and output the final N output pixels, wherein one previous cache unit and one interpolation calculation unit form a calculation pipeline, and each calculation pipeline uses bilinear interpolation for calculation; The QDR write address calculation unit is used to calculate the addresses of the final N output pixels corresponding to the QDR memory according to the lookup table read by the DDR4 read-write control unit; The QDR read address calculation unit generates an output video timing after delaying a specific number of video lines according to the parallel video timing output by the HDMI / DP input video decoding unit, and calculates the address in the QDR memory corresponding to the encoded output pixel according to the output video timing; The QDR read / write control unit is used to write the final N output pixels into the QDR memory according to the write address output by the QDR write address calculation unit, and read out the coded output pixel according to the address in the QDR memory corresponding to the coded output pixel output by the QDR read address calculation unit; The HDMI / DP output video encoding unit is used to perform video encoding on the encoded output pixels output by the QDR read / write control unit to output HDMI / DP video, and send it to the helmet display for display.

4. The ultra-high resolution video real-time distortion correction device according to claim 3, characterized in that: The distortion correction lookup table stored in the Flash memory stores the position mapping relationship between input video pixels and output video pixels before and after distortion correction, and each lookup table item in the lookup table corresponds to N output pixels; wherein the order of the lookup table items is sorted according to the row and column of the largest input pixel corresponding to each lookup table item.

5. The method and device for real-time distortion correction of ultra-high resolution video according to claim 4, characterized in that: The order of the lookup table entries is sorted by the row and column of the largest input pixel corresponding to each lookup table entry, including: When the row count of the input video exceeds the row and column of the largest input pixel of the current lookup table item, each computing pipeline can read the pixel value of the input pixel and its neighborhood corresponding to the current lookup table item from the previous cache unit, perform interpolation calculation based on the interpolation coefficient, and obtain the pixel value of the output pixel corresponding to the pipeline in the current lookup table item.

Images