A bicubic interpolation optimization method for video image acquisition system based on FPGA

Abstract

The present invention relates to a video image acquisition system based on FPGA and a bicubic interpolation optimization method. The bicubic interpolation optimization method of the video image acquisition system based on FPGA adopts a lightweight upsampling system so that it has a smaller model scale and fewer parameters during use, so its memory occupation is relatively low, and it can effectively adapt to resource-constrained devices; through innovative algorithms and optimization strategies, it can achieve high-quality upsampling effects while maintaining low computational complexity; by integerizing floating-point numbers, integer operations only involve integer parts, the calculation process is relatively simple, and the overall computational complexity is reduced; it can reduce memory occupation, which helps to reduce hardware costs and improve the applicability of the algorithm.

Description

Technical Field

[0001] The invention relates to the technical field of FPGA video image acquisition, and in particular to a bicubic interpolation optimization method of a video image acquisition system based on FPGA. Background Art

[0002] The real-time upsampling system of video images is a system that can perform real-time upsampling of video images through different technical means. In today's society, video images have become an indispensable part of people's daily lives. They are widely used in monitoring systems, video conferencing, distance education and other fields. However, since there may be many problems in the process of video image transmission, such as network delay, data loss, etc., an efficient real-time upsampling system is needed to solve these problems. FPGA (Field-Programmable Gate Array) is a programmable logic device with a wide range of applications in the field of image processing.

[0003] FPGA image upsampling refers to the process of converting low-resolution images into high-resolution images by enlarging images on the FPGA hardware platform. The upsampling related technologies include interpolation algorithms, convolutional neural networks, super-resolution algorithms, etc. Resource optimization, parallel processing, and real-time requirements should be considered in the algorithm implementation.

[0004] In terms of resource consumption, video image upsampling requires powerful data processing capabilities. The system must be able to quickly analyze the original video data and use complex algorithms to enlarge and supplement the image details; the upsampling process involves a large amount of memory access and data storage, and the system needs to save the original video data, intermediate calculation results, and the final high-definition image; video image upsampling also requires powerful graphics processing capabilities, and the system needs to execute complex image processing algorithms, such as interpolation and filtering, especially in the interpolation algorithm, which involves a large number of division operations. These operations have a high dependence on the performance of the graphics card and the capacity of the video memory; therefore, a bicubic interpolation optimization method for a video image acquisition system based on FPGA is proposed to solve the above problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a bicubic interpolation optimization method for a video image acquisition system based on FPGA, which has the advantages of effectively reducing the calculation amount of the bicubic interpolation algorithm when upsampling video data, improving the calculation efficiency and reducing resource occupation, and solves the problems of large algorithm calculation amount and large resource occupation during the caching process of video data.

[0006] To achieve the above object, the present invention provides the following technical solution: a bicubic interpolation optimization method for a video image acquisition system based on FPGA, which performs high-definition display of video images, comprising the following steps:

[0007] S1, the image acquisition module acquires the video data and transmits the video data to the upsampling module;

[0008] S2, the upsampling module performs bicubic interpolation optimization method on the received video data to upsample and amplify the video image, and transmits the amplified video data to the storage module;

[0009] The S2 further comprises the following sub-steps, characterized in that:

[0010] S2.1. Input the 16 original pixels in the upper left corner of the original image, i.e. a 4×4 pixel block. Take the first pixel in the upper left corner as the reference point and mark the coordinates of the 16 pixels as ,in ;

[0011] S2.2, The distance between the target pixel and the reference point is recorded as , establish the weight coefficient mapping formula and bring it into Calculate the weight group , for the initial weight group Optimize to get the weight group ;

[0012] S2.3. Input optimized weight group For 16 original pixels Perform horizontal interpolation calculation on each row to obtain four transition pixel points , for four transition pixels Perform interpolation calculation in the vertical direction to obtain a target pixel point ;

[0013] S2.4. Reorganization through substitution rights With different weight values ​​in , 16 original pixels are repeatedly interpolated to obtain 16 target pixels;

[0014] S2.5, traverse all 4×4 pixel blocks in the entire original image from left to right and from top to bottom, calculate the target pixel values ​​of all 4×4 pixel blocks in the original image, and output all target pixel points;

[0015] S3, the storage module converts the format of the up-sampled and amplified video data and stores it. After the storage is completed, the storage module reads out the stored video data again and transmits it to the display module;

[0016] S4. After receiving the up-sampled and amplified video data, the display module converts the video data into a high-definition display format and realizes high-definition display of the video data.

[0017] Furthermore, for the 16 original pixels The general formula for bicubic interpolation is as follows:

[0018] ;

[0019] The weight expression is:

[0020] ;

[0021] in, , , ; represents the horizontal weight, represents the vertical weight, is the pixel value of the original pixel, where r is the distance between the target pixel and the reference point.

[0022] Furthermore, the horizontal interpolation calculation is performed on each row of the 16 original pixels to obtain four transition pixels. The calculation formula is:

[0023] ;

[0024] In the above formula, ;

[0025] For four transition pixels The formula for obtaining the target pixel point by vertical interpolation is:

[0026] .

[0027] Furthermore, when performing bicubic interpolation calculation on the 16 original pixels, first select is the reference point. At this time, the distance between the target pixel and the reference point is , ,in is the horizontal distance between the target pixel and the reference point, , is the vertical distance between the target pixel and the reference point, ,Will Substitute the weight calculation function to get the weight group The specific formula for each weight coefficient in is as follows:

[0028] ;

[0029] in, .

[0030] Further, because So The way of changing periodically can Convert floating point integers to get ,in , the weight coefficient expression after simplification by floating-point integer is as follows:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in .

[0036] Furthermore, a video image acquisition system based on FPGA includes an image acquisition module, an upsampling module, a storage module, a control module and a display module;

[0037] The image acquisition module is connected to the up-sampling module, the up-sampling module is connected to the storage module, the storage module is connected to the display module, and the control module is connected to the storage module and the display module;

[0038] The image acquisition module includes a camera and an image acquisition chip adapted thereto, and the image acquisition chip is connected to the upsampling module;

[0039] The storage module includes a first interface conversion unit, a second interface conversion unit, a read-write unit and a storage, the storage includes a GP interface and an HP interface, the display module includes a first data conversion unit, a second data conversion unit and a display unit, the second interface conversion unit is connected to the first data conversion unit, the first data conversion unit is connected to the second data conversion unit, and the second data conversion unit is connected to the display unit;

[0040] The control module includes a control unit, a video configuration unit and a clock frequency unit. The control unit is connected to the GP interface, and the control unit is connected to the video configuration unit and the clock frequency unit.

[0041] Furthermore, the image acquisition chip is used to convert the video acquired by the camera into video data and transmit it to the up-sampling module.

[0042] Furthermore, the first interface conversion unit is used to convert the format of the up-sampled and amplified video data and transmit it to the read-write unit, and the read-write unit writes the up-sampled and amplified video data into the storage for storage;

[0043] The read / write unit is further used to read out the stored video data and transmit it to the second interface conversion unit, and the second interface conversion unit performs secondary format conversion on the video data and then transmits it to the display module.

[0044] Furthermore, the first data conversion unit is used to perform three format conversions on the video data read by the read-write unit, and transmit the video data after three format conversions to the fourth format conversion unit, and the fourth format conversion unit is used to perform four format conversions on the video data after three format conversions and transmit it to the display unit to achieve high-definition display.

[0045] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0046] 1. The bicubic interpolation optimization method of the FPGA-based video image acquisition system adopts a lightweight upsampling system to have a smaller model size and fewer parameters during use, so its memory usage is relatively low and can be effectively adapted to resource-constrained devices; through innovative algorithms and optimization strategies, it can achieve high-quality upsampling effects while maintaining low computational complexity.

[0047] 2. The bicubic interpolation optimization method of the FPGA-based video image acquisition system converts floating-point numbers into integers. Integer operations only involve the integer part, and the calculation process is relatively simple, which reduces the overall calculation complexity. It can reduce memory usage because integers usually take up less storage space than floating-point numbers, which helps to reduce hardware costs and improve the applicability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the upsampling system of the present invention;

[0049] Figure 2 Bicubic interpolation algorithm flow chart of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of the interpolation pixel point schematic diagram of the present invention;

[0051] Figure 4 It is a schematic diagram of the arrangement of interpolation points of the present invention;

[0052] Figure 5 It is a horizontal target image mapping diagram of the present invention;

[0053] Figure 6 This is a schematic diagram of implementing data row cache based on the ping-pong principle of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Embodiment 1:

[0056] See also Figure 1-2 , a video image acquisition system based on FPGA in this embodiment includes an image acquisition module, an upsampling module, a storage module, a control module and a display module;

[0057] The image acquisition module is connected to the up-sampling module, the up-sampling module is connected to the storage module, the storage module is connected to the display module, and the control module is connected to the storage module and the display module;

[0058] The display module includes a camera and an image acquisition chip adapted thereto, and the image acquisition chip is connected to the up-sampling module;

[0059] The storage module includes a first interface conversion unit, a second interface conversion unit, a read-write unit and a storage, the storage includes a GP interface and an HP interface, the display module includes a first data conversion unit, a second data conversion unit and a display unit, the second interface conversion unit is connected to the first data conversion unit, the first data conversion unit is connected to the second data conversion unit, and the second data conversion unit is connected to the display unit;

[0060] The control module includes a control unit, a video configuration unit and a clock frequency unit. The control unit is connected to the GP interface, and the control unit is connected to the video configuration unit and the clock frequency unit.

[0061] In this embodiment, when the video image needs to be upsampled, the video data can be collected through the display module, wherein the display module includes a camera and an image acquisition chip adapted thereto, and the image acquisition chip is used to convert the video collected by the camera into video data and transmit it to the upsampling module.

[0062] Preferably, the image acquisition chip in this embodiment can adopt an OV5640 single-chip image sensor. The present invention uses the OV5640 single-chip image sensor to acquire image data, and can achieve the fastest image acquisition capability of 90fps VGA (640×480) resolution.

[0063] When the upsampling module receives the video data, it will quadruple the 640×480 resolution image to a 2560×1920 resolution image.

[0064] The image after upsampling will enter the storage module for storage, wherein the storage module includes a first interface conversion unit, a second interface conversion unit, a read-write unit and a storage, the first interface conversion unit is used to perform a format conversion on the video data after upsampling and amplification and transmit it to the read-write unit, and the read-write unit writes the video data after upsampling and amplification into the storage for storage;

[0065] Preferably, the upsampling module in this embodiment adopts a parallel output method, and calculates and outputs the values ​​of four pixels in one clock cycle. Calculating the interpolation data of the target image requires four lines of data of the original image. In order to improve the operating efficiency of the FPGA, the row cache and data request system are implemented using the ping-pong principle.

[0066] like Figure 6 As shown, when calculating the first four rows of data of the target image, only the first four rows of the original image are used, so the first four rows of data of the original image are instantiated into the first four RAMs before calculation. When calculating and inputting the first group of data of the target image, the next row of data of the target image is stored in the fifth RAM. Similarly, when calculating the next group of data, the data in ram2, ram3, ram4, and ram5 are used. In the calculation, the next row of data of the target image is stored in ram1. The state machine loop transformation is used to realize the ping-pong principle, and this algorithm is used to improve the computing efficiency.

[0067] The read / write unit in this embodiment can also read out the stored video data and transmit it to the second interface conversion unit, which performs secondary format conversion on the video data and transmits it to the display module; the storage device is provided with an HP interface and a GP interface, and the control module is connected to the GP interface. It is worth noting that the storage module end in this example is a ps end, and the other modules are all pl ends.

[0068] Preferably, in the above embodiment, the first interface conversion unit may adopt a Video Into AXI4-Stream unit, the read / write unit includes a VDMA unit and an AXI-SmartConnect unit, and the second data conversion unit adopts an AXI4-Stream to Video out unit. When performing read / write operations, VDMA is connected to AXI-SmartConnect to achieve writing of data from the PL end to the HP interface of the PS end, and at the same time, data from the PL end can be read out to the PS end to achieve high-definition display;

[0069] In this embodiment, the image acquisition chip is used to convert the video captured by the camera into video data and transmit it to the upsampling module. The first interface conversion unit is used to perform a format conversion on the video data after upsampling and amplification and transmit it to the read-write unit, and the read-write unit writes the video data after upsampling and amplification into the storage for storage; the read-write unit is also used to read out the stored video data and transmit it to the second interface conversion unit, and the second interface conversion unit performs a second format conversion on the video data and transmits it to the display module. The first data conversion unit is used to perform a third format conversion on the video data read by the read-write unit, and transmit the video data after the third format conversion to the fourth format conversion unit, and the fourth format conversion unit is used to perform a fourth format conversion on the video data after the third format conversion and transmit it to the display unit to achieve high-definition display.

[0070] Preferably, the first data conversion unit in the above embodiment may adopt an RGB 2 YcrCb unit, the second format conversion unit may adopt an input Zed_hdmi_out unit, the display unit adopts an ADV7511 chip, the control unit may adopt a VideoTiming Controller unit, and the Video Timing Controller unit is responsible for controlling the operation of the AXI4-Stream to Videoout module; the video configuration unit may adopt an AXI-iic unit, and the AXI-iic unit is responsible for correctly configuring the ADV7511 chip, and the clock frequency unit may adopt a CLK WIX 1 unit for configuring the correct clock.

[0071] As a preferred technical solution in this embodiment: when the collected video data is displayed in high definition, firstly, the RGB image is collected by the OV5640 camera, transmitted to the upsampling module, and the video image data is subjected to bicubic linear interpolation processing. The processed data is in RGB format, and after passing through the Video In to AXI-Stream module, it is converted into AXI-Stream data and written to the HP interface of the storage, thereby realizing the data interaction between the PL and PS ends. Next, the AXI-Stream data is read out from the storage unit DDR, and the data is converted into RGB format through AXI4-Stream to Video Out; since HDMI is to be used for display, the RGB format must be converted into YCrCb format, so the RGB format is passed to RGB2YCrCb to obtain YCrCb data, and finally the data in YCrCb format is displayed in high definition using HDMI.

[0072] Embodiment 2:

[0073] See also Figure 3-6 In order to reduce the amount of calculation and resource occupation during the calculation of bicubic interpolation, a bicubic interpolation optimization method for a video image acquisition system based on FPGA in this embodiment includes the following steps:

[0074] S2.1. Input the 16 original pixels in the upper left corner of the original image, i.e. a 4×4 pixel block. Take the first pixel in the upper left corner as the reference point and mark the coordinates of the 16 pixels as ,in ;

[0075] S2.2, The distance between the target pixel and the reference point is recorded as , establish the weight coefficient mapping formula and bring it into Calculate the weight group , for the initial weight group Optimize to get the weight group ;

[0076] S2.3. Input optimized weight group For 16 original pixels Perform horizontal interpolation calculation on each row to obtain four transition pixel points , for four transition pixels Perform interpolation calculation in the vertical direction to obtain a target pixel point ;

[0077] S2.4. Reorganization through substitution rights With different weight values ​​in , 16 original pixels are repeatedly interpolated to obtain 16 target pixels;

[0078] S2.5. Traverse all 4×4 pixel blocks in the entire original image from left to right and from top to bottom, calculate the target pixel values ​​of all 4×4 pixel blocks in the original image, and output all target pixel points.

[0079] For 16 original pixels The general formula for bicubic interpolation is as follows:

[0080] ;

[0081] The weight expression is:

[0082] ;

[0083] in, , , ; represents the horizontal weight, represents the vertical weight, is the pixel value of the original pixel, where r is the distance between the target pixel and the reference point.

[0084] Perform horizontal interpolation calculation on each row of 16 original pixels to obtain four transition pixels The calculation formula is:

[0085] ;

[0086] In the above formula, ;

[0087] For four transition pixels The formula for obtaining the target pixel point by vertical interpolation is:

[0088] .

[0089] When performing bicubic interpolation calculation on 16 original pixels, first select is the reference point. At this time, the distance between the target pixel and the reference point is , ,in is the horizontal distance between the target pixel and the reference point, , is the vertical distance between the target pixel and the reference point, ,Will Substitute the weight calculation function to get the weight group The specific formula for each weight coefficient in is as follows:

[0090] ;

[0091] in, .

[0092] In the above formula, , after being substituted into the above formula, a large number of division operations will be involved. In the process of bicubic difference calculation, the division operation has a higher computational complexity than addition and multiplication. In applications such as image processing that require processing of large amounts of data, the increase in division operations will lead to a longer overall calculation time. Therefore, it is necessary to convert the floating-point integer to enable the integer part to be calculated first during the calculation, thereby effectively reducing the division operations and increasing the calculation speed. The specific implementation method is as follows:

[0093] See also Figure 6 , in each 4×4 pixel block So The way of changing periodically can be Convert floating point integers to get ,in , the weight coefficient expression after simplification by floating-point integer is as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] in .

[0099] When converting floating point numbers to integers, The value is , because its vertical and horizontal weights are fixed, the simplified calculation formula is as follows:

[0100] when hour

[0101] ;

[0102] when hour

[0103] ;

[0104] when hour

[0105] ;

[0106] when hour

[0107] ;

[0108] It is worth noting that, in the embodiment itself, it is preferred to calculate the floating-point integer part, and finally perform a division operation on it, so as to achieve the effect of reducing the amount of calculation; finally, the values ​​of the 16 pixel points around the interpolation position and the horizontal and vertical weights are substituted into the overall formula of bicubic interpolation to obtain the value of the interpolation point.

[0109] The present invention optimizes the algorithm, completes parallel optimization processing, pipeline operation design and data reuse operation, realizes real-time processing of video stream data, and the image performance parameters are relatively ideal. The present invention analyzes and compares the three parameters of the average PSNR value (peak signal-to-noise ratio), SSIM value (structural self-similarity operator) and LPIPS value (learning perceptual image fast similarity) of 50 images sampled, and the operation results are shown in Table 1. The three performance index characteristics are: the higher the PSNR value, the closer the SSIM is to 1, and the lower the LPIPS is, the better the upsampling quality is, refer to Table 1:

[0110] Table 1 Parameters of bicubic interpolation in different environments

[0111]

[0112] As shown in Table 1, the parameters of bicubic interpolation implemented based on FPGA are more ideal than those implemented in Python environment, and hardware acceleration in FPGA can ensure the real-time performance of video image processing.

[0113] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A bicubic interpolation optimization method for a video image acquisition system based on FPGA, characterized in that: Displaying a video image in high definition includes the following steps: S1, the image acquisition module acquires the video data and transmits the video data to the upsampling module; S2, the upsampling module performs bicubic interpolation optimization method on the received video data to upsample and amplify the video image, and transmits the amplified video data to the storage module; The S2 further comprises the following sub-steps, characterized in that: S2.

1. Input the 16 original pixels in the upper left corner of the original image, i.e. a 4×4 pixel block. Take the first pixel in the upper left corner as the reference point and mark the coordinates of the 16 pixels as ,in ; S2.2, The distance between the target pixel and the reference point is recorded as , establish the weight coefficient mapping formula and bring it into Calculate the weight group , for the initial weight group Optimize to get the weight group ; S2.

3. Input optimized weight group For 16 original pixels Perform horizontal interpolation calculation on each row to obtain four transition pixel points , for four transition pixels Perform interpolation calculation in the vertical direction to obtain a target pixel point ; S2.

4. Reorganization through substitution rights With different weight values ​​in , 16 original pixels are repeatedly interpolated to obtain 16 target pixels; S2.5, traverse all 4×4 pixel blocks in the entire original image from left to right and from top to bottom, calculate the target pixel values ​​of all 4×4 pixel blocks in the original image, and output all target pixel points; S3, the storage module converts the format of the up-sampled and amplified video data and stores it. After the storage is completed, the storage module reads out the stored video data again and transmits it to the display module; S4. After receiving the up-sampled and amplified video data, the display module converts the video data into a high-definition display format and realizes high-definition display of the video data.

2. The bicubic interpolation optimization method of a video image acquisition system based on FPGA according to claim 1, characterized in that: For 16 original pixels The general formula for bicubic interpolation is as follows: ； The weight expression is: ； in, , , ; represents the horizontal weight, represents the vertical weight, is the pixel value of the original pixel, where r is the distance between the target pixel and the reference point.

3. The bicubic interpolation optimization method of a video image acquisition system based on FPGA according to claim 2, characterized in that: Perform horizontal interpolation calculation on each row of 16 original pixels to obtain four transition pixels The calculation formula is: ； In the above formula, ; For four transition pixels The formula for obtaining the target pixel point by vertical interpolation is: 。 4. The bicubic interpolation optimization method of a video image acquisition system based on FPGA according to claim 3, characterized in that: When performing bicubic interpolation calculation on 16 original pixels, first select is the reference point. At this time, the distance between the target pixel and the reference point is , ,in is the horizontal distance between the target pixel and the reference point, , is the vertical distance between the target pixel and the reference point, ,Will Substitute the weight calculation function to get the weight group The specific formula for each weight coefficient in is as follows: ； in, .

5. The bicubic interpolation optimization method of a video image acquisition system based on FPGA according to claim 4, characterized in that: because So The way of changing periodically can Convert floating point integers to get ,in , the weight coefficient expression after simplification by floating-point integer is as follows: ； ； ； ； in .

6. A video image acquisition system based on FPGA according to any one of the methods described in claims 1-5, characterized in that: It includes an image acquisition module, an up-sampling module, a storage module, a control module and a display module; The image acquisition module is connected to the up-sampling module, and the up-sampling module is connected to the storage The storage module is connected to the display module, and the control module is connected to the storage module and the display module; The image acquisition module includes a camera and an image acquisition chip adapted thereto, and the image acquisition chip is connected to the upsampling module; The storage module includes a first interface conversion unit, a second interface conversion unit, a read-write unit and a storage, the storage includes a GP interface and an HP interface, the display module includes a first data conversion unit, a second data conversion unit and a display unit, the second interface conversion unit is connected to the first data conversion unit, the first data conversion unit is connected to the second data conversion unit, and the second data conversion unit is connected to the display unit; The control module includes a control unit, a video configuration unit and a clock frequency unit. The control unit is connected to the GP interface, and the control unit is connected to the video configuration unit and the clock frequency unit.

7. The FPGA-based video image acquisition system according to claim 6, characterized in that: The image acquisition chip is used to convert the video acquired by the camera into video data and transmit it to the up-sampling module.

8. The FPGA-based video image acquisition system according to claim 6, characterized in that: The first interface conversion unit is used to convert the format of the up-sampled and amplified video data and transmit it to the read-write unit, and the read-write unit writes the up-sampled and amplified video data into the storage for storage; The read / write unit is further used to read out the stored video data and transmit it to the second interface conversion unit, and the second interface conversion unit performs secondary format conversion on the video data and then transmits it to the display module.

9. The FPGA-based video image acquisition system according to claim 6, characterized in that: The first data conversion unit is used to perform three format conversions on the video data read by the read-write unit, and transmit the video data after three format conversions to the fourth format conversion unit. The fourth format conversion unit is used to perform four format conversions on the video data after three format conversions and transmit it to the display unit to achieve high-definition display.

Images