Multi-protocol low-delay image acquisition system based on FPGA
By using an FPGA-based multi-protocol low-latency image acquisition system, high-definition, high-bitrate, and low-latency image acquisition from multiple cameras was achieved, solving the problems of protocol compatibility and high latency in traditional systems and improving the autonomous driving performance of unmanned vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING LES ELECTRONICS EQUIP CO LTD
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies struggle to achieve high-definition, high-bitrate, and low-latency multi-camera image acquisition, and the systems are difficult to adapt to various protocols, resulting in poor system versatility, high transmission latency, and difficulty in meeting the real-time environmental perception requirements of autonomous vehicles.
The system employs an FPGA-based multi-protocol low-latency image acquisition system, including adaptive GMSL and SDI protocol cameras, a MIPI interface conversion module, a frame buffer module, and a PCIE communication module. Through modular design and custom IP cores, it achieves synchronous acquisition and orderly writing of multiple video data streams. Combined with PCIE XDMA direct access to DDR3 memory, it constructs a low-latency transmission channel.
It achieves high synchronization, low latency acquisition, and stable transmission of multi-channel video data, ensuring the integrity of video frames and the real-time performance of the system. It supports multiple camera protocols, improves the scalability and applicability of the system, and meets the real-time environmental perception requirements of unmanned vehicles.
Smart Images
Figure CN122069423A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image data acquisition and processing, and specifically to a multi-protocol low-latency image acquisition system based on FPGA. Background Technology
[0002] To meet the autonomous driving requirements of unmanned vehicles, environmental perception information needs to be collected in real time and fed to the central processing module to improve autonomous driving performance. The images of the vehicle's surrounding environment collected by cameras are an important part of the environmental perception information and directly determine the autonomous driving logic decisions of the vision solution. The complex vehicle surrounding environment requires multiple cameras to collect multiple sets of image data for comprehensive analysis. Therefore, a high-definition, high-bitrate, and low-latency image acquisition system has become an urgent requirement. At the same time, the system needs to be compatible with cameras of various protocols to improve the system's versatility. Summary of the Invention
[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-protocol low-latency image acquisition system based on FPGA, which addresses the shortcomings of the existing technology.
[0004] To address the aforementioned technical problems, this invention discloses a multi-protocol low-latency image acquisition system based on FPGA (Field Programmable Gate Array). The system includes an adaptive GMSL (Gigabit Multimedia Serial Link) protocol camera and an SDI (Serial Digital Interface) protocol camera, a camera configuration module, a MIPI (Mobile Industry Processor Interface) interface conversion module, a frame buffer module, and a PCIE (Peripheral Component Interconnect Express) communication module.
[0005] After the camera configuration module of the FPGA system completes the configuration of 8 GMSL protocol cameras, it receives video image data from the cameras. The GMSL cameras are in MIPI format and require MIPI protocol conversion, while the SDI cameras are in parallel YUV422 format and do not require conversion. The parallel video data is written to the DDR3 (Double Data Rate) memory module through the frame buffer module. The PCIe module completes the communication between the FPGA and the host computer.
[0006] The adaptive GMSL and SDI protocol camera features high real-time performance, high bandwidth, and low latency, enabling concurrent acquisition of video images from multiple cameras.
[0007] The GMSL protocol camera has 8 channels; the SDI protocol camera has 3 channels.
[0008] The specific image acquisition process includes:
[0009] Step 1: After power-on, the camera configuration module writes configuration data to the configuration chip (MAX9296 chip) for GMSL protocol cameras, and completes the parameter configuration of the camera through the configuration chip.
[0010] Step 2: After successful configuration, the camera transmits video image data back via the configuration chip using the MIPI protocol; SDI protocol cameras do not require configuration and return video image data.
[0011] Step 3: The 8-channel IP core MIPI CSI-2 Rx Subsystem is internally called to convert MIPI protocol video image data into a parallel video stream that can be processed by the FPGA's internal logic, and output it through the AXI-Stream interface;
[0012] Step 4: The frame buffer module uses a custom IP core FDMA to convert YUV422 format video image data into AXI4-FULL standard protocol data format. The 11 data streams are arbitrated by the AXI-Interconnect IP core and then written to the DDR3 memory module by the MiGIP core.
[0013] Step 5: The PCIE communication module uses the IP core XDMA to connect to the axi-interconnect arbitration module of the frame buffer module to read DDR3 video data and achieve high-speed communication with the host computer software.
[0014] DDR3 read and write operations are handled by MIG, while PCIe is only responsible for transmitting the read data to the host computer.
[0015] The parameter configuration described in step 1 includes: ID number and MIPI rate.
[0016] The camera configuration module includes the IIC driver and camera configuration table data.
[0017] The IIC driver is written using the internal logic resources of the FPGA, with an SCL clock rate of 100k. The camera is configured using a configuration chip as an intermediate medium. The configuration data is written to the configuration chip through the IIC driver, and the configuration chip then configures the camera. The camera configuration table data consists of several 32-bit data, and each configuration data is written in four parts, byte by byte. After all the data is written, the camera video data acquired by the camera through the GMSL protocol is transmitted back to the configuration chip, and then the chip inputs the data into the FPGA through the MIPI protocol (and then executes step 3).
[0018] The MIPI protocol conversion module includes a MIPI to AXI4-Stream module and an AXI4-Stream interface to YUV422 module:
[0019] The MIPI to AXI4-Stream module is implemented through the IP core MIPI CSI-2 Rx Subsystem. The external video stream is input to the IP core through a 2-line MIPI interface. The IP core pixel format is set to YUV422 8bits, and the line rate (transmission rate of a single line) is set to 1000Mbps. These two parameter settings are consistent with the data in the camera configuration table. The output is a 32-bit AXI4-Stream data stream. Then, the IP core AXI4-Stream Data Width Converter converts the 32-bit video data to 16-bit video data, and at the same time, the WVS signal and WDE synchronization signal are obtained.
[0020] The frame buffer module includes a custom IP core FDMA, an AXI interconnect arbiter, and a MIG memory controller.
[0021] The IP core FDMA input interface has write data and read data fields; this system only uses the write data field. The write data input interface has WVS, WDE, and WData interfaces. Eight channels of GMSL camera image data require calls to the IP core FDMA. The first channel of video data is used as the reference source. After receiving a complete frame, the first channel generates an interrupt number. The subsequent seven channels of video data use the interrupt generated by the first channel as the reference, generating an offset from the base address (the image storage uses a three-frame buffer; this base address is the address of the first frame in DDR3 memory) and writing the data to DDR3 via the memory controller MIG (Memory Interface Generator). Simultaneously, the host computer reads image data based on the first interrupt signal. Since all eight channels of video data need to be stored in DDR3, it is necessary to determine at a certain time which channel should be written to DDR3. The IP core uses AXI interconnect. By providing an AXI4-FULL interface and configuring the number of AXI master and slave interface channels to achieve channel arbitration, this system is configured with 9 slave interfaces and one master interface. Eight of these interfaces correspond to 8 channels of video data. One master interface connects to the MIG memory controller via an IP core to achieve DDR3 read / write operations. The remaining slave interface connects to a PCIe XDMA IP core, thus enabling data interaction between the core and the host computer software. The image data from the 3 SDI protocol cameras is similar to that of the GMSL protocol cameras mentioned above, using the first interrupt as a reference and also employing the AXI interconnect arbiter for arbitration.
[0022] The FPGA system stores multiple configuration table data through internal logic, enabling adaptive configuration of different cameras upon power-up; the three SDI protocol cameras do not require configuration, and the returned video data format is directly yuv422 8bits.
[0023] The entire image acquisition system is implemented using FPGA, which ensures the simultaneous acquisition of multiple video images with high synchronization and good real-time performance.
[0024] The FPGA employs a block design approach, resulting in a high degree of modularity. The modules communicate with each other using the AXI4 (Advanced eXtensible Interface 4) protocol, ensuring high versatility and transmission efficiency.
[0025] Using a custom IP core FDMA simplifies the AXI4-FULL interface, allowing for customization of data bit width and burst length, improving bus utilization and transmission efficiency, and ensuring low latency in data transmission.
[0026] The use of the axi-interconnect arbitration module ensures that multiple video data streams are written to DDR3 in an orderly manner, thus improving transmission efficiency.
[0027] Using XDMA to implement PCIe protocol communication and directly access DDR3 significantly reduces the latency of data interaction.
[0028] Beneficial effects:
[0029] 1. Simultaneous acquisition and ordered writing of multiple video streams:
[0030] To address issues such as timing discrepancies and DDR write contention that can easily occur when acquiring multiple video streams simultaneously, this invention adopts a modular design based on the AXI4 protocol, combined with a custom FDMA IP core and an axi-interconnect arbitration mechanism, to achieve conflict-free, high-bandwidth, ordered writing of multiple video streams in DDR3, ensuring strict synchronization and integrity of video frames.
[0031] 2. Low-latency data transmission channel:
[0032] In traditional systems, video data is transmitted to the host after passing through multiple relays, resulting in significant latency. This invention implements PCIe direct memory access (DMA) via XDMA, constructing a "short path" transmission channel from the FPGA to the host memory, significantly reducing end-to-end data transmission latency and improving system real-time performance.
[0033] 3. Scalable system architecture:
[0034] With its modular design and standardized AXI4 interface, this system supports flexible addition and removal of video acquisition channels and can adapt to video sources with different resolutions and frame rates, improving the system's reconfigurability and applicability while ensuring performance. Attached Figure Description
[0035] Figure 1 This is a diagram illustrating the composition of a multi-protocol low-latency image acquisition system based on FPGA disclosed in this invention.
[0036] Figure 2 Block diagram of camera configuration module;
[0037] Figure 3 This is a block diagram of the MIPI protocol conversion module.
[0038] Figure 4 This is a block diagram of the frame buffer module.
[0039] Figure 5 This is a physical image of the image acquisition system.
[0040] Figure 6Design diagram for the MIPI protocol conversion module block_design;
[0041] Figure 7a This is footage captured by four of the cameras in the system.
[0042] Figure 7b This is footage captured by the remaining four cameras in the system. Detailed Implementation
[0043] This invention provides a multi-channel video acquisition and transmission system based on FPGA. Its innovation lies not in the use of a single IP core or interface protocol, but in the realization of high synchronization, low latency acquisition and stable transmission of multiple video channels through a specific architecture combination and collaborative processing mechanism. This solves the problems of multi-channel video data writing conflicts, insufficient real-time transmission and poor system scalability in traditional solutions.
[0044] After configuring 8 GMSL protocol cameras, the camera configuration module of the FPGA system receives video image data from the cameras. The GMSL cameras are in MIPI format and require MIPI protocol conversion, while the SDI cameras use parallel YUV422 format data and do not require conversion. The parallel video data is written to the DDR3 memory module through the frame buffer module. The PCIe module completes the communication between the FPGA and the host computer.
[0045] The adaptive GMSL and SDI protocol camera features high real-time performance, high bandwidth, and low latency, enabling concurrent acquisition of video images from multiple cameras.
[0046] The GMSL protocol camera has 8 channels; the SDI protocol camera has 3 channels. The number of cameras of each type can be flexibly adjusted according to actual needs. The types of cameras include wide-angle (front view), fisheye (surround view), rear view / side view and stereo cameras, which can meet the 360-degree all-round field of view.
[0047] The specific image acquisition process includes:
[0048] Step 1: After power-on, the camera configuration module writes configuration data to the configuration chip (MAX9296 chip) for GMSL protocol cameras, and completes the parameter configuration of the camera through the configuration chip.
[0049] Step 2: After successful configuration, the camera transmits video image data back via the configuration chip using the MIPI protocol; SDI protocol cameras do not require configuration and return video image data.
[0050] Step 3: Internally call the 8-channel MIPI CSI-2 Rx Subsystem IP core to convert MIPI protocol video image data into a parallel video stream that can be processed by the FPGA's internal logic, and output it through the AXI-Stream interface;
[0051] Step 4: The frame buffer module uses a custom IP core FDMA to convert YUV422 format video image data into AXI4-FULL standard protocol data format. The 11 data streams are arbitrated by the AXI-Interconnect IP core and then written to the DDR3 memory module by the MiGIP core.
[0052] Step 5: The PCIE communication module uses the IP core XDMA to connect to the axi-interconnect arbitration module of the frame buffer module to read DDR3 video data and achieve high-speed communication with the host computer software.
[0053] In this embodiment, the specific image acquisition results are as follows: Figure 7a and Figure 7b As shown: Each row displays three buffered frames of images from a single camera. The camera is placed there still, so the three frames show the same image.
[0054] DDR3 read and write operations are handled by MIG, while PCIe is only responsible for transmitting the read data to the host computer.
[0055] The parameter configuration described in step 1 includes: ID number and MIPI rate.
[0056] The camera configuration module includes the IIC driver and camera configuration table data.
[0057] The IIC driver is written using the internal logic resources of the FPGA, with an SCL clock rate of 100k. The camera is configured using a configuration chip as an intermediate medium. The configuration data is written to the configuration chip through the IIC driver, and the configuration chip then configures the camera. The camera configuration table data consists of several 32-bit data, and each configuration data is written in four parts, byte by byte. After all the data is written, the camera video data acquired by the camera through the GMSL protocol is transmitted back to the configuration chip, and then the chip inputs the data into the FPGA through the MIPI protocol (and then executes step 3).
[0058] The MIPI protocol conversion module includes a MIPI to AXI4-Stream module and an AXI4-Stream interface to YUV422 module:
[0059] The MIPI to AXI4-Stream module is implemented through the IP core MIPI CSI-2 Rx Subsystem. The external video stream is input to the IP core through a 2-line MIPI interface. The IP core pixel format is set to YUV422 8bits and the line rate is set to 1000Mbps. These two parameter settings are consistent with the data in the camera configuration table. The output is a 32-bit AXI4-Stream data stream. Then, the IP core AXI4-Stream Data Width Converter converts the 32-bit video data to 16-bit video data, and at the same time, the WVS signal and WDE synchronization signal are obtained.
[0060] The frame buffer module includes a custom IP core FDMA, an AXI interconnect arbiter, and a MIG memory controller.
[0061] The IP core FDMA input interface has write data and read data fields; this system only uses the write data field. The write data input interface has WVS, WDE, and WData interfaces. Eight channels of GMSL camera image data require calls to the IP core FDMA. The first channel of video data is used as the reference source. After receiving a complete frame, the first channel generates an interrupt number. The subsequent seven channels of video data use the interrupt generated by the first channel as the reference, generating an offset from the base address (the image storage uses a three-frame buffer; this base address is the address of the first frame in DDR3 memory) and writing the data to DDR3 via the memory controller MIG. Simultaneously, the host computer reads image data based on the first interrupt signal. Since all eight channels of video data need to be stored in DDR3, it is necessary to determine at a certain time which channel should be written to DDR3. The IP core AXIinterconnect... By providing an AXI4-FULL interface and configuring the number of AXI master and slave interface channels to achieve channel arbitration, this system is configured with 9 slave interfaces and one master interface. Eight of these interfaces correspond to 8 channels of video data. One master interface connects to the MIG memory controller via an IP core to achieve DDR3 read / write operations. The remaining slave interface connects to a PCIe XDMA IP core, thus enabling data interaction between the core and the host computer software. The image data from the 3 SDI protocol cameras is similar to that of the GMSL protocol cameras mentioned above, using the first interrupt as a reference and also employing the AXI interconnect arbiter for arbitration.
[0062] The FPGA system stores multiple configuration table data through internal logic, enabling adaptive configuration of different cameras upon power-up; the three SDI protocol cameras do not require configuration, and the returned video data format is directly yuv422 8bits.
[0063] This method implements an adaptive acquisition and ordered frame buffering mechanism for multi-channel heterogeneous cameras based on the first-channel video interrupt signal as the synchronization reference. The method proposes an adaptive acquisition, timing alignment, and ordered writing of video data from multiple heterogeneous protocol (GMSL and SDI) cameras to DDR3, using the first-channel camera video frame completion signal (interrupt signal) as the global synchronization reference. This mechanism is implemented in the following way:
[0064] Establishment of synchronization reference: After power-on, the system automatically selects the first GMSL camera as the timing reference source. After it completes the acquisition of one frame of image, it generates a hardware interrupt signal. This signal is not only used to generate the storage address of the data of this channel, but also serves as the basis for the synchronization trigger of the remaining 7 GMSL cameras and 3 SDI cameras.
[0065] Offset address generation and frame buffer management: Subsequently, each camera generates its own storage offset address based on the first interrupt signal and a unified base address (using a three-frame buffer mechanism). This ensures that multiple video data streams are written to DDR3 in frame order without conflict, avoiding bandwidth contention and data misalignment during concurrent multi-channel writing.
[0066] Heterogeneous protocol adaptive processing: The system supports video streams of both MIPI (converted) and parallel YUV422 protocols through the same interrupt synchronization mechanism, realizing unified timing management for both GMSL and SDI cameras, and improving the system's protocol compatibility and scalability.
[0067] Low-latency transmission guarantee: By combining a custom FDMA IP core with an AXI interconnect arbiter, the system ensures strict synchronization of multiple video streams while directly accessing DDR3 via PCIE XDMA, thus building a "short path" transmission channel from the FPGA to the host computer and significantly reducing end-to-end data transmission latency.
[0068] The specific problems solved by this invention include:
[0069] The challenge of timing synchronization in multi-channel video acquisition, ensuring consistency in the acquisition of multiple video feeds at the same time;
[0070] Bandwidth contention and data loss issues when writing high-speed video streams to external memory;
[0071] High system transmission latency leads to limitations in real-time interactive applications;
[0072] Traditional video capture systems have rigid architectures that make it difficult to support dynamic adjustments to the number and format of channels.
[0073] The invention is characterized by multiple cameras and low latency; the simultaneous display of multiple cameras eliminates blind spots and overcomes the limitation of a single viewpoint in understanding complex road conditions. Furthermore, the multiple cameras can form a redundant system, preventing system failure due to the malfunction of a single camera.
[0074] Low latency enables rapid and accurate prediction and response to unexpected situations. When the vehicle is traveling at high speed, low latency can minimize the loop time of "perception-decision-control" and effectively increase the maximum speed of the vehicle in autonomous driving mode.
[0075] This invention provides a multi-channel video acquisition and transmission system based on FPGA. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. A multi-protocol low-latency image acquisition system based on FPGA, characterized in that, The system includes an adaptive GMSL protocol camera, an SDI protocol camera, a camera configuration module, a MIPI interface conversion module, a frame buffer module, and a PCIe communication module; After completing the GMSL protocol camera configuration, the camera configuration module of the FPGA system receives video image data from the camera. The GMSL camera uses MIPI format data, which requires MIPI protocol conversion, while the SDI camera uses parallel YUV422 format data, which does not require conversion. The parallel video data is written to the DDR3 memory module through the frame buffer module. The PCIe module enables communication between the FPGA and the host computer.
2. The FPGA-based multi-protocol low-latency image acquisition system according to claim 1, characterized in that, The specific image acquisition process includes: Step 1: After power-on, the camera configuration module writes configuration data into the configuration chip for GMSL protocol cameras, and completes the parameter configuration of the camera through the configuration chip. Step 2: After successful configuration, the camera transmits video image data back via the configuration chip using the MIPI protocol; SDI protocol cameras do not require configuration and directly return video image data. Step 3: Internally call the MIPI CSI-2 Rx Subsystem IP core to convert MIPI protocol video image data into a parallel video stream that can be processed by the FPGA's internal logic, and output it through the AXI-Stream interface; Step 4: The frame buffer module uses a custom IP core FDMA to convert YUV422 format video image data into AXI4-FULL standard protocol data format. The data is arbitrated by the AXI-Interconnect IP core and then written to the DDR3 memory module by the MIG IP core. Step 5: The PCIE communication module uses the IP core XDMA to connect to the axi-interconnect arbitration module of the frame buffer module to read DDR3 video data and achieve communication with the host computer software.
3. The FPGA-based multi-protocol low-latency image acquisition system according to claim 2, characterized in that, The parameter configuration described in step 1 includes: ID number and MIPI rate.
4. The FPGA-based multi-protocol low-latency image acquisition system according to claim 1, characterized in that, The camera configuration module includes the IIC driver and camera configuration table data. The IIC driver is written using the internal logic resources of the FPGA. It uses the configuration chip as an intermediate medium to configure the camera. The configuration data is written to the configuration chip through the IIC driver, and the configuration chip then configures the camera. After all the data is written, the camera video data acquired by the camera through the GMSL protocol is transmitted back to the configuration chip, and then the chip inputs the data into the FPGA through the MIPI protocol.
5. The FPGA-based multi-protocol low-latency image acquisition system according to claim 1, characterized in that, The MIPI protocol conversion module includes a MIPI to AXI4-Stream module and an AXI4-Stream interface to YUV422 module: The MIPI to AXI4-Stream module is implemented through the IP core MIPI CSI-2 Rx Subsystem. The external video stream is input to the IP core through a 2-line MIPI interface. The IP core pixel format is set to YUV422 8bits and the line rate is set to 1000Mbps. These two parameter settings are consistent with the data in the camera configuration table. The output is a 32-bit AXI4-Stream data stream. Then, the IP core AXI4-Stream Data Width Converter converts the 32-bit video data to 16-bit video data, and at the same time, the WVS signal and WDE synchronization signal are obtained.
6. The FPGA-based multi-protocol low-latency image acquisition system according to claim 2, characterized in that, The frame buffer module includes a custom IP core FDMA, an AXI interconnect arbiter, and a MIG memory controller. The IP core FDMA input interface has a write data field; the write data field input interface has a wvs interface, a wde interface, and a wdata interface. The n channels of GMSL camera image data need to call the n channels of IP core FDMA. The n channels of video data take the first channel as the reference source. After the first channel of video data receives a complete frame, it generates an interrupt number. The subsequent n-1 channels of video data all take the interrupt generated by the first channel as the reference, generate an offset based on the base address, and write the data to DDR3 through the memory controller MIG; at the same time, the host computer reads the image data based on the first interrupt signal, and the n channels of video data are stored in DDR3.
7. The FPGA-based multi-protocol low-latency image acquisition system according to claim 6, characterized in that, The video data is stored in DDR3 specifically by the AXI interconnect arbitrator providing an AXI4-FULL interface to configure the number of AXI master and slave interface channels to achieve channel arbitration.
8. The FPGA-based multi-protocol low-latency image acquisition system according to claim 1, characterized in that, The FPGA system stores multiple configuration table data through internal logic, enabling adaptive configuration of different cameras upon power-up; SDI protocol cameras do not require configuration, and the returned video data format is directly yuv422 8bits.
9. The FPGA-based multi-protocol low-latency image acquisition system according to claim 1, characterized in that, The GMSL protocol camera has 8 channels; the SDI protocol camera has 3 channels.
10. The FPGA-based multi-protocol low-latency image acquisition system according to claim 3, characterized in that, The configured chip is the MAX9296 chip.