Universal image operator acceleration system based on PYNQ and implementation method

Abstract

The present application relates to image processing acceleration application, and particularly relates to a general image operator acceleration system based on PYNQ and an implementation method. The present application designs a general image operator acceleration system based on a PYNQ architecture, and is divided into four categories of acceleration operators. Each category of acceleration operator directly communicates with memory, can be independently operated or combined, has rich operator resources for use, and simultaneously reserves a self-defined operator interface module, so that a user's own operator can be quickly added. The user of the system does not need to care about image input, data caching, display output and the like, and can quickly and flexibly achieve the acceleration purpose by calling the acceleration operator IP through Python.

Description

Technical Field

[0001] This invention relates to the fields of image processing and FPGA design technology, and to a general image operator acceleration system and working method based on PYNQ, which can be applied to security, medical, industrial and other fields that require accelerated image processing. Background Technology

[0002] With the continuous advancement of technology and rapid social development, the application scope of image processing technology is becoming increasingly widespread, including medical imaging, security monitoring, and smart homes. This has led to a growing demand for faster and more efficient image processing. Traditional CPUs are slow at processing images, limited by the real-time nature of software and the finite number of threads. The biggest advantage of FPGAs over CPUs lies in their speed. FPGAs control signal and register transmission through clocks, precisely controlling tasks. Furthermore, FPGAs execute in parallel, processing multiple data streams per clock cycle. This allows for the parallel execution of multiple image processing modules, thereby increasing processing speed and accelerating image processing.

[0003] Traditional pure FPGA image processing requires implementing the entire system flow using the hardware description language Verilog, which is technically challenging, complex, and inflexible. Under the PYNQ architecture, users need to build their own acceleration system, considering image acquisition and data storage, while also implementing the acceleration IP in Verilog, requiring a high level of technical expertise. Therefore, there is an urgent need for a general-purpose and flexible image processing acceleration system that eliminates the need to consider image acquisition, storage, output, and driving issues, and can quickly achieve image processing acceleration using existing acceleration operators or by adding custom operators. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a general-purpose image operator acceleration system and implementation method based on PYNQ. Based on the PYNQ architecture, the image processing leverages the PS and PL features of ZYNQ, implementing the acceleration operator module on the PL side. The PS side consists of an Ubuntu operating system and Python built on Ubuntu, implementing parts that do not require acceleration, such as image preprocessing and post-processing. This architecture provides a complete library for accessing FPGA resources, allowing developers to call the FPGA through Python programming.

[0005] The technical solution of this invention is as follows: A general-purpose image operator acceleration system based on PYNQ, configured at the PL end of the ZYNQ core processor, comprising:

[0006] The image input module is used to configure and drive the camera, and to acquire the data collected by the camera and perform mode conversion to obtain image data;

[0007] The image data storage module is used to cache image data in an on-chip FIFO, and after being packetized according to configuration parameters, send it to the off-chip storage in DDR3 on the PS side, and read the packetized data from DDR3 on the PS side and send it to the operator acceleration module to realize multi-channel parallel data caching.

[0008] The operator acceleration module contains at least one category of acceleration operators for image processing, interacts with DDR3 memory by category, and sends the processed image to the image display module, achieving parallel and pipelined processing.

[0009] The image display module is used to output and display images via an HDMI interface.

[0010] The image data storage module includes:

[0011] The parameter configuration module is used to receive control parameters from the PS end via the AXILite interface;

[0012] The parameter parsing module is used to calculate the configuration parameters of the DMA control module based on the control parameters of the parameter configuration module.

[0013] The burst transmission module is used to buffer the image data collected by the image input module in the on-chip FIFO according to the configuration parameters obtained by the parameter parsing module, and send it to the off-chip storage in PSDDR3 through the DMA control module after being packetized according to the configuration parameters. At the same time, it controls the DMA control module to read data from PSDDR3 and send it to the operator acceleration module.

[0014] The DMA control module, custom-implemented based on the AXI bus protocol, is used for reading and writing control of the AXI4-FULL bus, and converting the signals sent by the burst transfer module, which include address, data, length, and validity, into control signals that conform to the requirements of the AXI bus. At the same time, it automatically segments the address and data for bursts exceeding 256 in length, enabling the transmission of data of any size.

[0015] The control parameters include the row pixel value X_SIZE, column pixel value Y_SIZE, number of transmissions per row X_TIME, read / write channel data width DATA_WIDTH, number of buffered frames W_BUFSIZE, FIFO buffer depth W_BUFDEPTH, read / write channel set memory start address W_BASEADDR, and AXI data width AXI_DATA_WIDTH.

[0016] The configuration parameters of the DMA control module are calculated as follows:

[0017] DMA control module burst operation count = Y_SIZE * X_TIME;

[0018] The size of a single burst by the DMA control module = (DATA_WIDTH*X_SIZE / AXI_DATA_WIDTH) / X_TIME;

[0019] The address increment for a single burst in the DMA control module is (X_SIZE*(DATA_WIDTH / 8)) / X_TIME;

[0020] FIFO buffer width = DATA_WIDTH;

[0021] FIFO buffer address width = clog2(DATA_WIDTH) + 1;

[0022] Here, clog2 represents the logarithmic function.

[0023] The configuration parameters include the number of burst operations, the size of a single burst by the DMA control module, the address increment of a single burst by the DMA control module, and the address width of the FIFO buffer.

[0024] The state machine transitions for the burst transmission module are as follows:

[0025] In IDLE state, when the input image data frame synchronization signal is valid, the system enters RST state, generating a reset to synchronize the data frame and enter C_DATA state. When the DMA control module can receive data and the on-chip FIFO buffer is larger than the single burst size of the DMA control module, a write request is initiated to the DMA control module, and the system enters C_DATA1 state to start writing data. After one burst transmission, the system checks whether the number of transmissions has reached the burst operation limit of the DMA control module. If not, the system returns to C_DATA state to continue writing requests; if it has, the system returns to IDLE state.

[0026] The operator acceleration module includes acceleration operators for at least one category of image conversion, filtering or correction processing, feature extraction, and object detection. It is used to independently configure and use at least one category of acceleration operators as needed, and reserves interfaces to encapsulate custom acceleration operators under each category.

[0027] The implementation method of a general image operator acceleration system based on PYNQ includes the following steps:

[0028] The image input module configures and drives the camera, and acquires the data collected by the camera, performs mode conversion to obtain image data;

[0029] The image data storage module caches the image data in an on-chip FIFO, and then sends the data to the off-chip storage in DDR3 on the PS side after being packetized according to the configuration parameters. It also reads the packetized data from DDR3 on the PS side and sends it to the operator acceleration module to achieve multi-channel parallel data caching.

[0030] The operator acceleration module interacts with DDR3 memory according to category and sends the processed image to the image display module, realizing parallel and pipelined processing;

[0031] The image display module outputs and displays images via an HDMI interface.

[0032] The implementation method of a general image operator acceleration system based on PYNQ also includes the following steps:

[0033] 1) If you need to add a custom operator, implement the operator in HLS and output it in the form of an AXI interface; add the operator IP output by the AXI interface to the reserved operator in the operator acceleration module on the PL side, and compile to generate a .BIT file; otherwise, directly execute step 2).

[0034] 2) The operator acceleration module is updated by loading the .BIT file from the PL terminal onto the host computer.

[0035] The beneficial effects and advantages of this invention are:

[0036] 1. For basic image acceleration processing, there are abundant acceleration operator IPs available. There is no need to consider issues such as acceleration cores, image acquisition, and storage. The host computer uses Python to call the acceleration operator IPs to achieve the acceleration effect.

[0037] 2. Reserved interfaces allow for quick and easy addition of custom image processing acceleration operator IPs, providing strong expandability.

[0038] 3. Various image processing acceleration operator IPs are independently connected to memory, allowing for flexible selection and parallel execution as needed. Users can quickly build their own acceleration system using existing IPs.

[0039] 4. Based on the PYNQ architecture, this invention designs a general-purpose image operator acceleration system, divided into four categories of acceleration operators. Each category of acceleration operators communicates directly with memory and can run independently or in combination. It offers abundant operator resources and reserves a custom operator interface module, allowing users to quickly add their own operators. Users do not need to worry about image input, data caching, display output, etc., but can quickly and flexibly achieve acceleration by calling the acceleration operator IPs through Python. Attached Figure Description

[0040] Figure 1This is the state machine diagram of the write control of the burst transmission module of the present invention.

[0041] Figure 2 This is a block diagram showing the connection between the image acceleration operator, the image data storage module, and the ZYNQ PS in this invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and examples.

[0043] A general-purpose image operator acceleration system based on PYNQ and its working method are presented. The system consists of a host computer and execution hardware. The execution hardware mainly includes a ZYNQ core processor, an OV5640 camera, DDR3 storage, a USB interface, Ethernet, and a power management unit. The software layer of the execution hardware runs on the ZYNQ's PS (Power Processor), primarily consisting of an Ubuntu operating system and Python built on top of Ubuntu. The ZYNQ PL (Programmable Logic) part is designed as an overlay, using different bitstream files and can be called via software APIs to dynamically switch logic functions on the FPGA. The application layer mainly consists of Jupyter Notebook and IPython running on top of Python. The host computer primarily runs the Jupyter Notebook environment, a browser-based interactive computing environment based on a Python kernel and the WebSocket protocol. Through the application layer's API library, Python calls the acceleration cores of the FPGA-side processing unit to accelerate applications.

[0044] The ZYNQ PL terminal primarily handles image input acquisition, image data storage, image acceleration operators, image display output, and command interaction with the ZYNQ PS terminal within the acceleration system. It is the core component of the general-purpose image operator acceleration system based on PYNQ. It mainly consists of an image input module, an image data storage module, an operator acceleration module, and an image display module.

[0045] The image input module primarily configures and drives the OV5640 camera, converting the RGB565 data transmitted from the camera into RGB888 data. Register configuration for the OV5640 module is performed via the SCCB interface; the data format is shown in Table 1.

[0046] Table 1

[0047]

[0048] The configured registers mainly include the PLL clock register, image output format register, and reset register. Data acquisition and conversion are achieved by controlling the input frame synchronization signal and data validity signal to concatenate 8-bit data into 24-bit data, convert it into RGB888 data, and simultaneously output the frame synchronization signal.

[0049] The image data storage module implements high-speed data caching with multiple parallel channels. The data cache is divided into on-chip FIFO cache and off-chip DDR3 cache, and includes a parameter configuration module, a parameter calculation module, a burst transmission module, and a DMA control module.

[0050] The parameter configuration module receives control parameters from the PS, including the row pixel value X_SIZE, column pixel value Y_SIZE, number of transmissions per row X_TIME, read / write channel data width DATA_WIDTH, number of buffered frames W_BUFSIZE, FIFO buffer depth W_BUFDEPTH, read / write channel memory start address W_BASEADDR, and AXI data width AXI_DATA_WIDTH, etc. These configuration parameters determine the packet size relationship during transmission and are stored accordingly so that the stored image data can be read out when needed.

[0051] The parameter calculation module mainly calculates the number of burst operations of the DMA control module, the size of one burst of the DMA control module, the address increment of a single burst of the DMA control module, and the read / write address width of the FIFO buffer area based on the parameters of the parameter configuration module.

[0052] DMA control module burst operation count = Y_SIZE * X_TIME

[0053] Size of a single burst from the DMA control module = (DATA_WIDTH * X_SIZE / AXI_DATA_WIDTH) / X_TIME

[0054] The address increment for a single burst in the DMA control module is (X_SIZE*(DATA_WIDTH / 8)) / X_TIME;

[0055] FIFO buffer width = DATA_WIDTH;

[0056] FIFO buffer address width = clog2(DATA_WIDTH) + 1;

[0057] The burst transmission module, based on the configuration parameters obtained from the parameter calculation module, buffers and packets the image data acquired by the image input module, then sends it to PSDDR3 for storage via the DMA control module. Simultaneously, it controls the DMA control module to read data from PSDDR3 and send it to the operator acceleration module. The state control for the read and write processes is essentially the same. For the write state controller, the state machine transitions are as follows: Figure 1 As shown. In IDLE state, the frame synchronization signal of the video input data is valid, and the system enters RST state, generating a reset to synchronize data frames and enter C_DATA state. When the DMA control module can receive data and the buffer in the FIFO is larger than the single burst size of the DMA control module, a write request is initiated to the DMA control module, and the system enters C_DATA1 state to start writing data. After one burst transmission, it is determined whether the number of transmissions has reached the burst operation limit of the DMA control module. If not, the system returns to C_DATA state to continue the write request; if it has, the system returns to IDLE state.

[0058] The DMA control module is a custom-designed DMA controller based on the AXI bus protocol, primarily responsible for read and write control of the AXI4-FULL bus. The read and write processes are essentially the same. During AXI bus operation, over 20 signals, including channel address, data address, and response signals, need to be controlled simultaneously, with a maximum data burst of only 256. The DMA control module converts the simple signals (address, data, length, valid) sent by the burst transfer module into complex control signals conforming to AXI bus requirements. It also automatically segments the address and data for bursts exceeding 256, enabling the transfer of data of any size and providing greater flexibility and convenience.

[0059] The operator acceleration module integrates commonly used image processing operators, categorized into four main types according to the image processing workflow: image transformation, filtering or correction, feature extraction, and object detection operators. The image transformation module includes color space conversion (RGB-HSV, RGB-CMYK, RGB-YUV) and color-to-grayscale conversion; image filtering and correction operators include common filtering operators such as Gamma transform, mean filtering, median filtering, Gaussian filtering, and low-pass / high-pass filtering, and can also add custom filtering algorithms based on the AXI bus; feature extraction operators include Laplacian, Sobel, Roberts, and Prewitt operators, and a reserved interface allows for the addition of custom operators based on the AXI bus; object detection includes face detection and frame difference-based object detection, and a reserved interface allows for the addition of custom detection operators based on the AXI bus. Compared to software processing methods, these operators employ parallel, pipelined processing methods to achieve acceleration.

[0060] The image display module is mainly used for image output and display, with an HDMI output interface. It includes an HDMI driver and a data readout driver. The data readout driver primarily drives the read interface of the burst transmission module, reading the data stored in DDR and sending it to the HDMI driver for display.

[0061] The PYNQ-based general-purpose image operator acceleration system is characterized by its rich set of image acceleration operators, capable of meeting general needs. Furthermore, each image acceleration operator is categorized and connected to DDR3 memory, allowing for the independent use of a single operator from a specific category or the use of operators from all categories. Interfaces are reserved for encapsulating custom acceleration operators within each category, resulting in greater versatility and flexibility. Figure 2 As shown. There are multiple image data storage modules, equal to and corresponding to the number of operator acceleration modules. Each category can interact with the DDR in parallel, operating as an independent channel. In this embodiment, the PS end has four HP channels, with each of the four image data storage modules corresponding to one of the four operator acceleration modules. The ZYNQ PL acceleration core is called via Python in the Jupyter Notebook environment of the host computer, selecting the acceleration operator for each category through control address and offset address.

[0062] The working method of the general image operator acceleration system based on PYNQ is divided into two cases: adding a custom operator module and not adding a custom operator module.

[0063] (1) Without adding a custom operator module, use the existing operator modules in the general system: Taking target detection as an example, the operators used include color to grayscale operator, mean filtering operator, and frame difference method target detection operator. Since no custom operator module needs to be added, the ZYNQ PL end does not need to be changed.

[0064] 1) On the host computer, enter the target network port IP address in Jupyter Notebook to connect to the target board;

[0065] 2) Load the .bit file from the PL end;

[0066] 3) Configure the OV5640 camera parameters in the image input module and enable the acquisition function;

[0067] 4) Configure the parameters of the image data storage module by address offset;

[0068] 5) The selected acceleration operators include color-to-grayscale operator, mean filtering operator, and frame difference method target detection operator;

[0069] 6) Enable the selected operator to perform image processing and drive the image display module to output through the HDMI interface.

[0070] (2) A custom operator module needs to be added: Taking the target detection operator with background subtraction as an example, and using the color-to-grayscale operator and median filtering operator in the general system. The steps are as follows:

[0071] 1) Implement the background subtraction target detection operator in HLS, optimize parallelism and pipeline, and output it in the form of AXI interface;

[0072] 2) In the PL system, add the generated operator IP core to the custom operator module under the target detection operator category;

[0073] 3) Compile and generate the system's .BIT file;

[0074] 4) On the host computer, enter the target network port IP address in Jupyter Notebook to connect to the target board and load the .BIT file;

[0075] 5) Configure the OV5640 camera parameters in the image input module and enable the acquisition function;

[0076] 6) Configure the parameters of the image data storage module by address offset;

[0077] 7) The selected acceleration operators include color-to-grayscale operator, mean filtering operator, and a custom background subtraction target detection operator;

[0078] 8) Enable the selected operator to perform image processing and drive the image display module to output through the HDMI interface.

Claims

1. A general image operator acceleration system based on PYNQ, characterized in that, The PL terminal configured on the ZYNQ core processor includes: The image input module is used to configure and drive the camera, and to acquire the data collected by the camera and perform mode conversion to obtain image data; The image data storage module is used to cache image data in an on-chip FIFO, and after being packetized according to configuration parameters, send it to the off-chip storage in DDR3 on the PS side, and read the packetized data from DDR3 on the PS side and send it to the operator acceleration module to realize multi-channel parallel data caching. The operator acceleration module contains at least one category of acceleration operators for image processing, interacts with DDR3 memory by category, and sends the processed image to the image display module, achieving parallel and pipelined processing. The image display module is used to output and display images via an HDMI interface.

2. The general image operator acceleration system based on PYNQ according to claim 1, characterized in that, The image data storage module includes: The parameter configuration module is used to receive control parameters from the PS end via the AXILite interface; The parameter parsing module is used to calculate the configuration parameters of the DMA control module based on the control parameters of the parameter configuration module. The burst transmission module is used to buffer the image data collected by the image input module in the on-chip FIFO according to the configuration parameters obtained by the parameter parsing module, and send it to the off-chip storage in PSDDR3 through the DMA control module after being packetized according to the configuration parameters. At the same time, it controls the DMA control module to read data from PSDDR3 and send it to the operator acceleration module. The DMA control module, custom-implemented based on the AXI bus protocol, is used for reading and writing control of the AXI4-FULL bus, and converting the signals sent by the burst transfer module, which include address, data, length, and validity, into control signals that conform to the requirements of the AXI bus. At the same time, it automatically segments the address and data for bursts exceeding 256 in length, enabling the transmission of data of any size.

3. The general image operator acceleration system based on PYNQ according to claim 1, characterized in that, The control parameters include the row pixel value X_SIZE, column pixel value Y_SIZE, number of transmissions per row X_TIME, read / write channel data width DATA_WIDTH, number of buffered frames W_BUFSIZE, FIFO buffer depth W_BUFDEPTH, read / write channel set memory start address W_BASEADDR, and AXI data width AXI_DATA_WIDTH.

4. The general image operator acceleration system based on PYNQ according to claim 2 or 3, characterized in that, The configuration parameters of the DMA control module are calculated as follows: DMA control module burst operation count = Y_SIZE * X_TIME; The size of a single burst by the DMA control module = (DATA_WIDTH*X_SIZE / AXI_DATA_WIDTH) / X_TIME; The address increment for a single burst in the DMA control module is (X_SIZE*(DATA_WIDTH / 8)) / X_TIME; FIFO buffer width = DATA_WIDTH; FIFO buffer address width = clog2(DATA_WIDTH) + 1; Here, clog2 represents the logarithmic function.

5. The general image operator acceleration system based on PYNQ according to claim 2, characterized in that, The configuration parameters include the number of burst operations, the size of a single burst by the DMA control module, the address increment of a single burst by the DMA control module, and the address width of the FIFO buffer.

6. The general image operator acceleration system based on PYNQ according to claim 2, characterized in that, The state machine transitions for the burst transmission module are as follows: In IDLE state, when the input image data frame synchronization signal is valid, the system enters RST state, generating a reset to synchronize the data frame and enter C_DATA state. When the DMA control module can receive data and the on-chip FIFO buffer is larger than the single burst size of the DMA control module, a write request is initiated to the DMA control module, and the system enters C_DATA1 state to start writing data. After one burst transmission, the system checks whether the number of transmissions has reached the burst operation limit of the DMA control module. If not, the system returns to C_DATA state to continue writing requests; if it has, the system returns to IDLE state.

7. The general image operator acceleration system based on PYNQ according to claim 1, characterized in that, The operator acceleration module includes acceleration operators for at least one category of image conversion, filtering or correction processing, feature extraction, and object detection. It is used to independently configure and use at least one category of acceleration operators as needed, and reserves interfaces to encapsulate custom acceleration operators under each category.

8. A method for implementing a general image operator acceleration system based on PYNQ, characterized in that, Includes the following steps: The image input module configures and drives the camera, and acquires the data collected by the camera, performs mode conversion to obtain image data; The image data storage module caches the image data in an on-chip FIFO, and then sends the data to the off-chip storage in DDR3 on the PS side after being packetized according to the configuration parameters. It also reads the packetized data from DDR3 on the PS side and sends it to the operator acceleration module to achieve multi-channel parallel data caching. The operator acceleration module interacts with DDR3 memory according to category and sends the processed image to the image display module, realizing parallel and pipelined processing; The image display module outputs and displays images via an HDMI interface.

9. The implementation method of the general image operator acceleration system based on PYNQ according to claim 8, characterized in that, It also includes the following steps: 1) If you need to add a custom operator, implement the operator in HLS and output it in the form of an AXI interface; add the operator IP output by the AXI interface to the reserved operator in the operator acceleration module on the PL side, and compile to generate a .BIT file; otherwise, directly execute step 2). 2) The operator acceleration module is updated by loading the .BIT file from the PL terminal onto the host computer.

Images