A real-time image display system based on ZYNQ

By combining the collaborative design of the CMOS camera, SPI bus control module and block design system in the ZYNQ system, the problems of slow image transmission speed and color deviation are solved, and efficient real-time display of images is achieved.

CN115174763BActive Publication Date: 2024-11-19SHENZHEN SAIDESI INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210782869.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-05
Publication Date
2024-11-19
Estimated Expiration
2042-07-05

AI Technical Summary

Technical Problem

With the improvement of image quality, the amount of data operated by image acquisition, processing, transmission and real-time display increases exponentially, resulting in slow image transmission speed, and real-time display of images is often accompanied by problems of image tearing and color deviation.

Method used

Adopt a ZYNQ-based image real-time display system, including a CMOS camera, SPI bus control module and block design system, through image analysis and conversion module, grayscale processing module, data writing and reading memory module, VGA control module and HDMI sending module The collaborative design uses the advantages of FPGA parallel processing and the high-speed data interaction of the AXI bus to realize real-time display of images.

Benefits of technology

It improves data processing and transmission efficiency, avoids slowing image frame rate, solves color deviation problem, and improves data transmission rate and stability through high-performance interfaces and pipeline structures, ensuring real-time display of images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115174763B_ABST
    Figure CN115174763B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of image processing and computer vision, and specifically relates to a ZYNQ-based real-time image display system, comprising: a CMOS camera, an SPI bus control module, a block design system and a display, wherein the block design system is externally connected to the CMOS camera and the display; an original image acquired by the CMOS camera is input into the block design system for image processing, and the processed image is sent to the display to complete image display; the present invention adopts a pipeline structure by combining the advantages of FPGA parallel processing at the PS end to improve the efficiency of data processing and transmission; the problem of color deviation is solved by the gray world algorithm module in the pipeline structure; a data interaction module between the PS end and the PL end is independently designed according to the AXI timing to further improve the speed and stability of data transmission.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and in particular relates to a ZYNQ-based real-time image display system. Background Art

[0002] ZYNQ-7000 is a programmable system-on-chip development platform launched by Xilinx. Its essential feature is the combination of a high-performance dual-core ARM Cortex-A9 processor and a traditional field programmable gate array (FPGA) logic component. Among them, the dual-core ARM processor is the core of the processing system (PS), and the FPGA logic unit and DSP resources are the core of the programmable logic (PL). The interaction between PS and PL on this development platform is interconnected through the AXI (Advanced eXtensible Interface) bus to achieve high-speed data interaction. The programmable system-on-chip development platform not only has the advantages of ASIC in energy consumption, performance and compatibility, but also has the advantages of FPGA hardware programmability. At present, the programmable system-on-chip development platform is widely used in embedded image processing and can meet the real-time requirements.

[0003] As image quality continues to improve, the amount of data required for image acquisition, processing, transmission and real-time display has increased exponentially, resulting in slow image transmission speeds, and real-time image display is often accompanied by image tearing and color deviation. If a single ARM processor is used to solve this problem, its serial processing architecture reduces the image processing speed, and if DSP+FPGA is used, the development difficulty and cost are increased.

[0004] To sum up, the problem of the existing technology is that as the image quality continues to improve, the amount of data operated for image acquisition, processing, transmission and real-time display has increased exponentially, resulting in slow image transmission speed, and the real-time display of images is often accompanied by problems such as image tearing and color deviation. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a ZYNQ-based real-time image display system, comprising: a CMOS camera, an SPI bus control module, a block design system and a display, wherein the block design system is externally connected to the CMOS camera and the display; the SPI bus control module is connected to the CMOS camera;

[0006] The block design system includes 6 IP modules: image analysis and conversion module, grayscale processing module, data writing memory module, data reading memory module, VGA control module, and HDMI sending module;

[0007] The SPI bus control module initializes the CMOS camera and configures the built-in registers of the CMOS camera through the SPI bus, and obtains that the original image captured by the CMOS camera has a frame rate of 30 Hz, a size of 1920x1080, and a format of Bayer format;

[0008] The image analysis and conversion module extracts each frame of the original image collected by the CMOS camera and analyzes the field synchronization signal and the line synchronization signal of the image, and converts the image into RGB888 format by using the interpolation method of the field synchronization signal and the line synchronization signal;

[0009] The grayscale processing module performs white balance processing on the three primary colors of each frame of image to obtain a processed image;

[0010] The data writing memory module writes the processed image data into the DDR3 SDRAM memory via the AXI4 bus, and temporarily caches the image via the asynchronous FIFO;

[0011] The data read memory module reads the image temporarily cached in the DDR3 SDRAM memory through the AXI4 bus;

[0012] The VGA control module scans and synchronizes the image read out by the data read memory module through the VGA timing, and then sends the synchronized data to the HDMI sending module;

[0013] The HDMI sending module encodes, performs DC balance and serial-to-parallel conversion on the data synchronized by the VGA control module, and sends the processed image to the display to complete image display.

[0014] Preferably, the image analysis and conversion module includes: a state machine and a row and column counter, and the image analysis and conversion module is divided into two parts: image analysis and image conversion. The image analysis part analyzes the number of pixels effectively displayed by the original image collected by the sensor in the CMOS camera through the state machine, obtains the row synchronization signal, the field synchronization signal and the three primary colors, and accurately counts each pixel through the row and column counter to accurately control each pixel; the image conversion part converts the analyzed image bayer format into the RGB888 image format for display on the display.

[0015] Furthermore, the original image in the Bayer format captured by the CMOS camera reserves 8 circles of edge pixels outside the effective display area of ​​1920x1080. The 8 circles of edge pixels outside the effective display area are used to perform image enhancement processing on the collected original image. The edge pixels outside the effective display area are expanded by one circle of edge pixels using a 3x3 interpolation method, so as to convert the original image in the Bayer format into an image in RGB888 format for display on a display.

[0016] Preferably, the grayscale processing module uses the grayscale world algorithm to further process the converted image, specifically: pixel detection is performed on the extended area through the line synchronization signal and the field synchronization signal, three accumulators are set and cleared, each accumulator corresponds to a primary color channel, when the image data of the extended area can be used for effective image display, the three accumulators respectively accumulate the channel values ​​of the three primary colors of one frame of the image, the accumulated data is divided by 1920x1080 to obtain the average value of each channel, white balance processing is performed on a new frame of the image, and the three channels are merged, and the above operation is repeated until the white balance processing is completed for each frame of the image.

[0017] Preferably, the data write memory module and the data read memory module are built based on ZYNQ's high-performance interface HP, and the HP port follows the AXI protocol. A FIFO is added to cache the data written by the data write memory module, and the data read memory module reads the data cached in the FIFO. When the image data cached in the FIFO is less than one line, the AXI bus is started to read the data in DDR3 to fill the FIFO.

[0018] Preferably, the memory space accessed by the data writing memory module and the data reading memory module is divided into area A and area B of the same size, and area A and area B are continuous. A flag signal is introduced at the source end of the data. When the flag signal is at a high level, the image data in area A is read and the read image data is written to area B. When the flag signal is at a low level, the image data in area B is read and the read image data is written to area A.

[0019] Preferably, the VGA control module scans and synchronizes the read image data, changes the image data parameters to 1920x1080@60hz, and outputs the image data to the HDMI sending module.

[0020] Preferably, the HDMI sending module receives the image data output by the VGA control module, performs 8-bit to 10-bit encoding, DC balancing and parallel-to-serial conversion on the data, and outputs the data to a display having an HDMI interface.

[0021] Preferably, the clock of the FIFO write port is set to be consistent with the synchronous clock of the AXI bus, and the clock of the read port is set to be consistent with the driving clock 148.5 MHz of the VGA control module, to ensure that the filling efficiency of the FIFO is greater than the reading efficiency of the VGA control module, so that the image can be displayed in real time.

[0022] Beneficial effects of the present invention:

[0023] 1). Using a software-hardware collaborative design method, the PS side completes the initialization and register configuration of the camera, and the PL side uses a pipeline structure combined with the advantages of FPGA parallel processing to improve the efficiency of data processing and transmission;

[0024] 2) The image format conversion is completed in the image parsing and conversion module of the pipeline structure to avoid slowing down the acquisition frame rate when the conversion is completed inside the camera;

[0025] 3). The grayscale world algorithm module in the pipeline structure provides a solution to the problem of color deviation;

[0026] 4). High-speed access to PS-side memory DDR3 SDRAM is achieved through the high-performance interface in the ZYNQ system, and the data interaction module between the PS and PL is independently designed according to the AXI timing to further improve the speed and stability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall framework diagram of the real-time image display system of the present invention;

[0028] Figure 2 It is a 3x3 interpolation conversion map of the effective edge of the original image of the present invention;

[0029] Figure 3 This is a 3x3 interpolation calculation asynchronous FIFO pipeline operation diagram of the present invention;

[0030] Figure 4 It is a structural diagram of a data writing module and a data reading module of the present invention;

[0031] Figure 5 A diagram of the area division of the memory space accessed by the data writing module and the data reading module of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, a real-time image display system based on ZYNQ includes a CMOS camera, an SPI bus control, a block design system, and a display. The block design system is externally connected to the CMOS camera and the display. The block design system includes 6 IP modules: an image analysis and conversion module, a grayscale processing module, a data writing memory module, a data reading memory module, a VGA control module, and an HDMI sending module.

[0034] The SPI bus control module completes the initialization and register configuration of the CMOS camera through the SPI bus, so that the frame rate of the original image collected by the CMOS camera is 30hz and the size is 1920x1080;

[0035] The image analysis and conversion module receives the raw image data collected by the CMOS camera, and then analyzes and converts the raw image data and sends it to the grayscale processing module;

[0036] The grayscale processing module receives the image data sent by the image analysis and conversion module, performs grayscale processing on the image data using the original grayscale algorithm, and then sends it to the data writing memory module;

[0037] The data writing memory module writes the image data into the DDR3 SDRAM memory via the AXI4 bus, and temporarily caches the image via the asynchronous FIFO;

[0038] The data read memory module reads the image data in the DDR3 SDRAM memory through the AXI4 bus, and caches the image data through the asynchronous FIFO. The memory address is partitioned while the data is written into and read from the memory, thereby solving the problem of image tearing.

[0039] The VGA control module receives the image data cached in the asynchronous FIFO in the data read memory module, scans and synchronizes the image through the VGA timing, and then sends the synchronized data to the HDMI sending module;

[0040] The HDMI sending module receives the image data synchronized by the VGA control module, and then sends it to the display to complete the image display.

[0041] The image parsing and conversion module is divided into two parts: image parsing and image conversion. The ultimate goal of image parsing is to parse a frame of image collected by the camera and generate a row synchronization signal and a field synchronization signal as a sign signal for subsequent image processing and transmission, so that the image will not be transmitted across frames. The present invention parses the collected original image through a state machine and generates a row and field synchronization signal, and accurately counts each pixel through a row and column counter to facilitate the control of each pixel; the image conversion part converts the collected original image format into an RGB888 image format for display on a display. The original image format is a bayer format. The bayer format is a single-channel pixel with a bit width of 8 bits. It is necessary to convert the bayer format image into an RGB888 format through 3x3 interpolation. The RGB888 format is a 24-bit three-channel pixel. The CMOS camera used in the present invention collects a frame of image. In addition to the effective display area of ​​1920x1080, 8 circles of pixels are reserved for image processing. The 3x3 interpolation method is adopted here, so a circle of pixels is extended outside the effective display area to perform interpolation calculation on the edge pixels of the effective display area, such as Figure 2 shown.

[0042] In the present invention, the logic implementation of the 3x3 interpolation calculation requires two asynchronous FIFOs to complete the pipeline operation, such as Figure 3 As shown: a 3x3 register matrix is ​​constructed to store pixels. FIFO0 caches the first row of pixels of a frame of image, FIFO1 caches the second row of pixels of a frame of image, and the third row of pixels of a frame of image is written into the third row of the register (c2, c1, c0). Then FIFO0 receives the pixels read out from FIFO1, and FIFO1 receives the pixels collected by the camera. When the row counter counts to the second row and the 1082nd row, FIFO0 and FIFO1 turn on the read enable to write the stored row of pixels into the first and second rows of the register matrix, and at the same time write the newly transmitted row of pixels into the third row of the register matrix. The row and column counters are counted, and an interpolation calculation is completed each time the register matrix is ​​filled. The interpolation calculation process is as follows: In RGB Taking the four pixels of 11, RGB12, RGB21, and RGB22 as examples, the red channel value of the RGB11 pixel is RGB11_R=R11, the green channel value is RGB11_G=(G01+G10+G21+G12) / 4, and the blue channel value is RGB11_B=(B00+B02+B20+B22) / 4; the red channel value of the RGB12 pixel is RGB12_R=(R11+R13) / 2, the green channel value is RGB12_G=G12, and the blue channel value is RGB12_B=(B02+B22) / 2; the red channel value of the RGB21 pixel is RGB21_R=(R11+R31) / 2, the green channel value is RGB12_G=G21, and the blue channel value is

[0043] RGB21_B=(B20+B22) / 2; red channel value of RGB22 pixel

[0044] RGB22_R=(R11+R13+R31+R33) / 4, green channel value RGB22_G=(G12+G21+G32G23) / 4, blue channel value RGB21_B=B22, where RGB11 represents pixels in odd rows and odd columns, RGB12 represents pixels in odd rows and even columns, RGB21 represents pixels in even rows and odd columns, and RGB22 represents pixels in even rows and even columns. The pixels of FIFO pipeline operation are judged by the last bit of the row and column counter to select the corresponding formula to complete the calculation. Finally, the values ​​of the three channels are combined in the order of R, G, and B to obtain the RGB888 format pixels for display.

[0045] The grayscale world algorithm module further processes the converted image to correct the color deviation. It is suitable for scenes with rich colors, and the module can be reconstructed according to the actual application scenario without affecting the overall pipeline structure. The logical implementation steps of the grayscale world algorithm are as follows: the pixel detection of the extended area is carried out through the line synchronization signal and the field synchronization signal to obtain the flag of a frame of image, and three accumulators are set and cleared. Each accumulator corresponds to a primary color channel. When the image data of the extended area can be used for effective image display, the three accumulators respectively accumulate the channel values ​​of the three primary colors of a frame of the image, and the accumulated data is divided by 1920x1080 to obtain the average value of each channel. The white balance processing of the new frame of image is performed, and the three channels are merged. Repeat the above operation until each frame of the image completes the white balance processing.

[0046] When calculating the average value of the three channels, considering that the divisor 1920x1080 is a fixed value, the fixed value can be approximated to about 2097152, that is, 2 to the 21st power. After the error calculation (2097152-1920x1080) / 1920x1080, the error result is only 1.13%. In this way, the lower 21 bits of the accumulated sum of each channel can be cut off, and the remaining high bits are the average value of the corresponding channel. This optimization method reduces the call of the divider, saves resources overall, and improves the processing speed of the algorithm.

[0047] The core formula of the grayscale world algorithm: Taking the red channel as an example, Rnew=red*K / Ravg, where Rnew is the red channel data in the processed pixel, red is the red channel data passed in by the image analysis and conversion module, K is the grayscale value, and Ravg is the average value of the red channel. The formula is implemented by calling the multiplier IP and the divider IP. There will be a delay of three clocks for the multiplier operation, so the data valid signal will also be delayed by three clocks. When the calculation result of the channel is greater than the threshold of 255, it is directly mapped to 255. This processing method is more effective in scenes with complex colors.

[0048] The data write memory module and the data read memory module are implemented based on ZYNQ's high-performance interface HP port, which is used to transmit high-speed data streams to meet the needs of real-time image display. The HP port follows the AXI protocol. When using the AXI protocol, these two modules set the data burst length to 256, the data bit width to 64 bits, and the synchronization clock to 250 MHz. Add a FIFO to the data read module to cache the data read from the AXI bus. When the image data cached in the FIFO is less than one line, start the AXI bus to read the data in DDR3 to fill the FIFO. When the VGA control module is started for the first time and determines that the image data in the FIFO is greater than one line, the VGA drive timing can be started, such as Figure 4 As shown. The clock of the FIFO write port is consistent with the synchronous clock of the AXI bus, and the clock of the read port is consistent with the driving clock 148.5mhz of the VGA control module, ensuring that the filling efficiency of the FIFO is greater than the reading efficiency of the VGA control module, so that the image can be displayed in real time. Add the self-designed write data in the data write module, open the DDR3 mapped memory in the software SDK to check whether the actual written data is consistent, so as to verify the function of the module. Similarly, write data to the DDR3 mapped memory through C language in the SDK software, and then read its data through the data read module for comparison to complete the verification of the data read module.

[0049] In the present invention, the frame rate of the COMS camera used to collect images is 30hz, and the frame rate of the display is 60hz. It is necessary to divide the memory space accessed by the data writing module and the data reading module. The memory space of area A and area B has the same size and can just store one frame of image. Area A and area B are continuous. In addition, a flag signal is introduced at the source end of the data. When the flag signal is at a high level, the number in area A is read and the image data just collected is written to area B. Similarly, when the flag signal is at a low level, the number in area B is read and the image data just collected is written to area A. This method solves the tearing problem of the image in real-time display, such as Figure 5 shown.

[0050] The VGA control module scans and synchronizes the image data read out from DDR3, and designs the code into a parameterized form. According to the 1920x1080@60hz mode, it changes the corresponding parameters and outputs the horizontal synchronization signal hsync, field synchronization signal vsync and 3-channel RGB image data required by the VGA interface, and outputs them to a display with a VGA interface.

[0051] After receiving the data from the VGA control module, the HDMI transmission module performs 8-bit to 10-bit encoding, DC balancing, and parallel-to-serial conversion. Each channel needs to perform 8-bit to 10-bit encoding and parallel-to-serial conversion. The encoding module can download the Encoder code from the Xilinx official website, and the parallel-to-serial module can be implemented by searching the Serializer source language in Vivado, and finally output to the display with an HDMI interface. The design of these two modules meets the actual display interface requirements, and can finally complete the real-time display of the image.

[0052] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time image display system based on ZYNQ, characterized in that: include: CMOS camera, SPI bus control module, block design system and display, wherein the block design system is externally connected to the CMOS camera and display; The SPI bus control module is connected to a CMOS camera; The block design system includes 6 IP modules: image analysis and conversion module, grayscale processing module, data writing memory module, data reading memory module, VGA control module, and HDMI sending module; The SPI bus control module initializes the CMOS camera and configures the built-in registers of the CMOS camera through the SPI bus, and obtains that the original image captured by the CMOS camera has a frame rate of 30 Hz, a size of 1920x1080, and a format of Bayer format; The image analysis and conversion module extracts each frame of the original image collected by the CMOS camera and analyzes the field synchronization signal and the line synchronization signal of the image, and converts the image into RGB888 format by using the interpolation method of the field synchronization signal and the line synchronization signal; The grayscale processing module performs white balance processing on the three primary colors of each frame of image to obtain a processed image; The data writing memory module writes the processed image data into the DDR3 SDRAM memory via the AXI4 bus, and temporarily caches the image via the asynchronous FIFO; The data read memory module reads the image temporarily cached in the DDR3 SDRAM memory through the AXI4 bus; The VGA control module scans and synchronizes the image read out by the data read memory module through the VGA timing, and then sends the synchronized data to the HDMI sending module; The HDMI sending module encodes, performs DC balance and serial-to-parallel conversion on the data synchronized by the VGA control module, and sends the processed image to the display to complete image display.

2. According to the ZYNQ-based real-time image display system of claim 1, it is characterized in that: The image analysis and conversion module includes: a state machine and a row and column counter, and the image analysis and conversion module is divided into two parts: image analysis and image conversion. The image analysis part uses the state machine to analyze the number of pixels effectively displayed by the original image collected by the sensor in the CMOS camera, obtains the row synchronization signal and the field synchronization signal, and accurately counts each pixel through the row and column counter to accurately control each pixel; the image conversion part converts the bayer format of the analyzed image into the RGB888 image format for display on the display.

3. The real-time image display system based on ZYNQ according to claim 2, characterized in that: The original image in Bayer format collected by the CMOS camera reserves 8 circles of edge pixels outside the effective display area of ​​1920x1080. The 8 circles of edge pixels outside the effective display area are used to perform image enhancement processing on the collected original image. The edge pixels outside the effective display area are extended by a circle of edge pixels using the 3x3 interpolation method to convert the original image in Bayer format into an image in RGB888 format for display on the display.

4. The real-time image display system based on ZYNQ according to claim 1, characterized in that: The grayscale processing module uses the grayscale world algorithm to further process the converted image, specifically: pixel detection is performed on the extended area through the line synchronization signal and the field synchronization signal, three accumulators are set and cleared, each accumulator corresponds to a primary color channel, and when the image data in the extended area can be used for effective image display, the three accumulators respectively accumulate the channel values ​​of the three primary colors of one frame of the image, and the accumulated data is divided by 1920x1080 to obtain the average value of each channel, and the white balance processing is performed on the new frame of the image, and the three channels are merged, and the above operation is repeated until the white balance processing is completed for each frame of the image.

5. The real-time image display system based on ZYNQ according to claim 1, characterized in that: The data write memory module and the data read memory module are built based on ZYNQ's high-performance interface HP. The HP port follows the AXI protocol. A FIFO is added to cache the data written by the data write memory module, and the data read memory module reads the data cached in the FIFO. When the image data cached in the FIFO is less than one line, the AXI bus is started to read the data in DDR3 to fill the FIFO.

6. The real-time image display system based on ZYNQ according to claim 1, characterized in that: The memory space accessed by the data write memory module and the data read memory module is divided into area A and area B of the same size, and area A and area B are continuous. A flag signal is introduced at the source end of the data. When the flag signal is at a high level, the image data in area A is read and the read image data is written to area B at the same time. When the flag signal is at a low level, the image data in area B is read and the read image data is written to area A at the same time.

7. The real-time image display system based on ZYNQ according to claim 1, characterized in that: The VGA control module scans and synchronizes the read image data, changes the image data parameters to 1920x1080@60hz, and outputs the image data to the HDMI sending module.

8. The real-time image display system based on ZYNQ according to claim 1, characterized in that: The HDMI sending module receives the image data output by the VGA control module, performs 8-bit to 10-bit encoding, DC balancing and parallel-to-serial conversion on the data, and outputs the data to a display with an HDMI interface.

9. The real-time image display system based on ZYNQ according to claim 1, characterized in that: Set the clock of the FIFO write port to be consistent with the synchronous clock of the AXI bus, and the clock of the read port to be consistent with the driving clock 148.5 MHz of the VGA control module, to ensure that the filling efficiency of the FIFO is greater than the reading efficiency of the VGA control module, so that the image can be displayed in real time.