FPGA-PCIe-based multi-channel image sensor data transmission system and optoelectronic system

Abstract

This invention provides a multi-channel image sensor data transmission system and optoelectronic system based on FPGA-PCIe, belonging to the technical field of video processing and acquisition systems. The multi-channel image sensor data transmission system includes an FPGA circuit board, external memory, and a back-end image processing platform. The FPGA circuit board includes an integrated image processing module, a bus control module, and a PCIe interface control module. Utilizing an FPGA to process and acquire multiple sensor video streams saves hardware costs and circuit board area. It also ports various operation operators to the FPGA, enabling real-time processing and acquisition of the currently transmitted video stream, improving the real-time performance of the operation. Simultaneously, it achieves control and transmission of multiple video streams, meeting the requirements for data transmission agility and high speed. This solves the problem in existing methods where inconsistent video resolutions and frame rates from different image sensors lead to insufficient data bandwidth to meet protocol requirements, resulting in the inability of PCIe devices at both ends of the FPGA to communicate normally.

Description

Technical Field

[0001] This invention belongs to the technical field of video processing and acquisition systems, specifically relating to a multi-channel image sensor data transmission system based on FPGA-PCIe, and an optoelectronic system using this system for data transmission processing. Background Technology

[0002] In the application of unmanned aerial vehicles and multi-platform collaborative systems, the onboard optoelectronic system, as the core component of the system's visual perception, plays a crucial role in "detection, intervention, communication, and guidance." To meet the requirements of platform adaptability and consumability, and to keep pace with technological development trends, existing optoelectronic pods need to be characterized by serialization, modularity, low cost, and rapid iteration to support flexible switching between different mission scenarios. The image acquisition module of the optoelectronic pod typically integrates multiple sensors to acquire images of the external scene, including but not limited to visible light, low light, infrared long-wave, mid-wave, and short-wave sensors. Each sensor has its own video data stream channel, and its video data stream is usually transmitted in an independent mode.

[0003] Existing image acquisition modules generally adopt a siloed architecture, with each image sensor having its own independent video output channel and corresponding receiving and processing unit. While each sensor has its own video data stream channel, operating independently without interference, this also prevents system resource sharing, resulting in significant waste of hardware resources. When multiple sensors are needed to acquire images of the external scene, the system cost increases dramatically with the number of sensors, and the multi-channel video acquisition circuitry also significantly occupies hardware board space, hindering the miniaturization and integration of the device. Furthermore, image acquisition modules on small UAVs need to quickly capture targets and perform stable tracking. Because existing image parameter settings and processing operations (such as exposure area selection, noise reduction, image enhancement, and color space conversion) cannot be performed efficiently and in real-time during acquisition, the system's flexibility and responsiveness are limited. In tasks with high real-time requirements, insufficient image processing speed directly impacts the overall system performance; simultaneously, inconsistent video resolution and frame rates from different sensors lead to insufficient data bandwidth to meet protocol requirements, affecting normal communication, and inconsistent exposure times between different sensors also cause asynchronous output videos. Summary of the Invention

[0004] The purpose of this invention is to solve the problems of existing image acquisition modules being unable to process images at high speed and in real time, as well as the inconsistency in video resolution and frame rate, video asynchrony, and excessive circuit board area of ​​different sensors. The invention provides a multi-channel image sensor data transmission system based on FPGA-PCIe, and an optoelectronic system using this system for data transmission processing.

[0005] To achieve the above objectives, the technical solution provided by this invention is:

[0006] A multi-channel image sensor data transmission system based on FPGA-PCIe is provided, including an FPGA circuit board, external memory, and a back-end image processing platform. The FPGA circuit board is connected to multiple image sensors. The FPGA circuit board includes an integrated image processing module, a bus control module, and a PCIe interface control module. The number of integrated image processing modules is consistent with the number of image sensors. The integrated image processing modules are used to independently drive and configure multiple image sensors, simultaneously acquire video signals from the image sensors, and preprocess the acquired video signals. The bus control module is connected to both the integrated image processing modules and the external memory. It is used to write the preprocessed video image data from the integrated image processing modules to the external memory and to arbitrate and schedule data read / write requests to the external memory. The PCIe interface control module is connected to the bus control module. Data read / write requests from the back-end image processing platform are received by the PCIe interface control module and then arbitrated and scheduled by the bus control module. The PCIe interface control module realizes bidirectional conversion between the FPGA internal bus protocol and the PCIe bus protocol. The back-end image processing platform is connected to the PCIe interface control module of the FPGA circuit board via the PCIe bus to receive the preprocessed multi-channel video image data and perform post-processing.

[0007] Furthermore, the integrated image processing module includes an image acquisition module, an image preprocessing module, and an image buffer module. The image acquisition module is connected to the driver board of an external image sensor via the FPGA circuit board's I / O interface, and is used for parameter configuration, signal driving, exposure control, and video image data acquisition of the image sensor. The image preprocessing module is connected to the image acquisition module and is used for preprocessing the acquired video signals. The image buffer module is connected to the image preprocessing module and is used for temporarily storing the preprocessed video image data.

[0008] Furthermore, preprocessing includes one or more of the following: image tap transformation, fixed noise reduction, RAW to RGB conversion, automatic white balance, color correction, automatic gain control, automatic brightness control, Gamma correction, and image enhancement.

[0009] Furthermore, the bus control module accesses the external memory through the external storage control module. The external storage control module is used to control the data read and write of the external memory and converts the interface bus protocol of the bus control module into the interface bus protocol of the external memory. The PCIe interface control module directly accesses the external memory through the bus control module and the external storage control module.

[0010] Furthermore, the external memory is SDRAM, and the data read and write of the SDRAM uses a burst transfer mode.

[0011] Furthermore, the read / write data transfer rate of SDRAM memory is higher than the data output rate of image sensors.

[0012] Furthermore, the integrated image processing module also includes a FIFO memory. The receiving end of the FIFO memory is connected to the image buffer module, and the reading end is connected to the bus control module. The image buffer module writes video image data into the FIFO memory through the clock and horizontal and vertical synchronization signals of the video data stream. The video image data written into the FIFO memory awaits reading by the bus control module. The video data stream is the carrier form for transmitting video image data on the physical link.

[0013] Furthermore, the read clock and read data width of the FIFO memory are greater than the write clock and write data width of the FIFO memory; where the write clock is the pixel clock of the sensor, the write data width is the video signal data width of the image sensor, the read clock is the data read / write clock of the external memory, and the read data width is the data width in the external memory.

[0014] It also provides an optoelectronic system mounted on an unmanned aerial vehicle and a multi-platform collaborative system, which uses the aforementioned multi-channel image sensor data transmission system for data transmission processing.

[0015] The advantages of this invention are:

[0016] 1. This invention provides a multi-channel image sensor data transmission system based on FPGA-PCIe. It utilizes an FPGA circuit board to process and acquire multiple sensor video streams, saving hardware costs and circuit board area. Furthermore, it allows for the porting of various video operation operators to the FPGA circuit board, enabling real-time processing and acquisition of the currently transmitted video stream, thus improving real-time performance. Simultaneously, the PCIe interface control module controls and transmits the multiple video streams, satisfying the requirements for data transmission agility and high speed. This solves the problem in existing methods where inconsistent video resolutions and frame rates from different image sensors lead to insufficient data bandwidth to meet protocol requirements, preventing normal communication between the PCIe devices at both ends of the FPGA. Since the FPGA circuit board inputs multiple image sensor data streams, while the PCIe interface control module and external memory can only input one data stream, scheduling is achieved through a bus control module. This pipelined design of the FPGA circuit board allows for real-time data processing, resolving the problem of video data asynchrony caused by inconsistent exposure times of different image sensors in existing methods.

[0017] 2. The photoelectric system provided by this invention, which is carried by a multi-platform collaborative system for unmanned aerial vehicles, utilizes a multi-channel image sensor data transmission system for data transmission processing. This enables the photoelectric system to process the acquired image data in real time and efficiently, thereby achieving stable tracking of the captured target. Attached Figure Description

[0018] The features and advantages of the invention will become more readily understood from the following description with reference to the accompanying drawings, in which:

[0019] Figure 1 This is a schematic diagram of the framework of a multi-channel image sensor data transmission system based on FPGA-PCIe according to the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments thereof. It should be noted that the following detailed description of the present invention is for illustrative purposes only and is not intended to limit the scope of the invention.

[0021] This invention provides a multi-channel image sensor data transmission system based on FPGA-PCIe, such as... Figure 1 As shown, the system includes an FPGA circuit board, external memory, and a back-end image processing platform. The FPGA circuit board is connected to multiple image sensors. The FPGA circuit board includes an integrated image processing module, a bus control module, and a PCIe interface control module. The multi-channel image sensor data transmission system is used for parameter configuration, image acquisition, and image preprocessing of multiple image sensors, ultimately sending the data to the back-end image processing platform via the PCIe interface control module. The data transmission system provided by this invention improves the resource utilization of the FPGA, reduces the circuit board area, and saves hardware costs.

[0022] The number of integrated image processing modules matches the number of image sensors. These modules independently drive and configure multiple image sensors, simultaneously acquiring various video signals from the image acquisition system and preprocessing the acquired signals. Different image sensors have different parameter configurations, signal drives, exposure controls, and video image acquisition methods. Image acquisition from multiple image sensors can run in parallel without interfering with each other.

[0023] The bus control module is connected to the integrated image processing module and the external memory respectively. It is used to write the pre-processed video image data of the integrated image processing module to the external memory, and to arbitrate and schedule data read and write requests to the external memory, so that the external memory can read data from multiple image sensors.

[0024] The PCIe interface control module is connected to the bus control module. Data read / write requests from the backend image processing platform are received by the PCIe interface control module and then arbitrated and scheduled by the bus control module. The PCIe interface control module enables bidirectional conversion between the FPGA internal bus protocol and the PCIe bus protocol. In this embodiment, the FPGA internal bus protocol is preferably the AXI4 bus protocol.

[0025] The read / write speed of the bus control module is much higher than the data transmission speed of the image sensor. It is necessary to allocate a whole frame of storage space for each image sensor. However, the cache resources inside the FPGA chip are limited, so an external memory needs to be installed outside the FPGA chip for large-scale image caching.

[0026] The PCIe interface control module primarily performs the conversion between the FPGA's internal bus protocol and the PCIe bus protocol, enabling high-speed serial data transmission between the FPGA board and the back-end image processing platform. The PCIe interface control module can directly access external memory via the bus control module without CPU intervention, making it suitable for large-data-volume scenarios such as image processing.

[0027] The back-end image processing platform is connected to the PCIe interface control module of the FPGA board via the PCIe bus, and is used to receive pre-processed multi-channel video image data and perform post-processing.

[0028] In this embodiment, as Figure 1 As shown, three image sensors are configured, each with an image resolution of 640*512 pixels and a frame rate of 30Hz. The three image sensors are mounted on an image sensor driver board and connected to the I / O interfaces on the FPGA circuit board via cables. The integrated image processing module on the FPGA circuit board can directly configure the image sensor parameters and perform signal driving, image reading, and image preprocessing operations. The image sensor data output interface can be LVDS, MIPI, or other interfaces, while the image sensor parameter configuration interface can be GPIO, I2C, SPI, or other interfaces. The integrated image processing module includes an image acquisition module, an image preprocessing module, and an image buffer module.

[0029] The image acquisition module is connected to the external image sensor driver board through the FPGA circuit board's IO interface. It is used for parameter configuration, signal driving, exposure control, and video image data acquisition of the image sensor.

[0030] The image preprocessing module is connected to the image acquisition module and transmits the acquired video signals to the image preprocessing module. The image preprocessing module is used to preprocess the acquired video signals. In this embodiment, the preprocessing includes, but is not limited to, image tap transformation, fixed noise elimination, RAW to RGB conversion, automatic white balance, color correction, automatic gain control, automatic brightness control, Gamma correction, and image enhancement.

[0031] The image caching module is connected to the image preprocessing module and is used to temporarily store the video image data preprocessed by the image preprocessing module.

[0032] In this embodiment, the external memory is SDRAM, and the SDRAM uses burst mode for data read and write. The data clock of SDRAM can reach hundreds of megahertz or even gigahertz, reading 32 bits or 64 bits of data in one clock cycle. However, the data transfer rate of an image sensor is typically tens of megahertz, and the data width for each read / write operation is generally 8 to 14 bits. Because the external SDRAM uses burst mode, its read / write data transfer rate is higher than the data output rate of the image sensor, therefore image buffering is required on the FPGA circuit board.

[0033] The integrated image processing module also includes a FIFO memory. Since external memory, such as DDR3, uses burst data transmission, the width of the burst data is much larger than the width of the image data output by the image sensor. Therefore, a FIFO memory is needed to buffer the image data. The receiving end of the FIFO memory is connected to the image buffer module, and the reading end is connected to the bus control module. The image buffer module writes video image data into the FIFO memory using the clock and horizontal / vertical synchronization signals of the video data stream. The video image data written into the FIFO memory awaits reading by the bus control module. The video data stream is the carrier form for transmitting video image data over the physical link.

[0034] The read clock and read data width of a FIFO memory are greater than the write clock and write data width of a FIFO memory; where the write clock is the pixel clock of the sensor, the write data width is the video signal data width of the image sensor, the read clock is the data read / write clock of the external memory, and the read data width is the data width in the external memory.

[0035] In this embodiment, the bus control module accesses the external memory through the external storage control module. The external storage control module controls data read and write operations on the external memory and converts the interface bus protocol of the bus control module into the interface bus protocol of the external memory. In this embodiment, the interface bus protocol of the bus control module is preferably the DDR bus protocol, and the external memory is preferably the AXI4 bus protocol. The PCIe interface control module directly accesses the external memory through the bus control module and the external storage control module.

[0036] The external storage control module is connected to the bus control module inside the FPGA and to the external SDRAM memory via the FPGA board's I / O interface. The external storage control module receives data write commands from the bus control module, writes video data from the three image sensors into three specified address segments of the external SDRAM memory, and then reads the buffered video data from the SDRAM memory according to data read commands. The external storage control module parses the serial address and data into parallel data to operate on the SDRAM memory.

[0037] The FPGA circuit board uses an external storage control module to perform high-speed burst read and write operations on the SDRAM memory to cache multiple video image data, thus solving the rate matching problem between the output rate of the image sensor and the access rate of the SDRAM memory.

[0038] Each of the three image sensors corresponds to an integrated image processing module. The three image sensors share a bus control module, a PCIe interface control module, and an external memory. The bus control module sequentially acquires video image data from the three image sensors and sends it to the external storage control module. At the same time, it receives data read commands from the PCIe interface control module and forwards the commands to the external storage control module, thereby realizing the scheduling between the data streams.

[0039] This invention provides a multi-channel image sensor data transmission system based on FPGA-PCIe. It utilizes an FPGA circuit board to process and acquire multiple sensor video streams, saving hardware costs and circuit board area. Furthermore, it allows for the porting of various video operation operators to the FPGA circuit board, enabling real-time processing and acquisition of the currently transmitted video stream, thus improving real-time performance. Simultaneously, a PCIe interface control module controls and transmits the multiple video streams, satisfying the requirements for agile and high-speed data transmission. This solves the problem in existing methods where inconsistent video resolutions and frame rates from different image sensors lead to insufficient data bandwidth to meet protocol requirements, preventing normal communication between the PCIe devices at both ends of the FPGA. The system achieves data processing and acquisition of multiple image sensor interface video streams via the PCIe interface control module, and can process video data in real-time during transmission. The pipelined design of the FPGA circuit board enables real-time data processing, resolving the problem of video data asynchrony caused by inconsistent exposure times of different image sensors in existing methods.

[0040] This invention also provides an optoelectronic system mounted on an unmanned aerial vehicle and a multi-platform collaborative system, utilizing the aforementioned multi-channel image sensor data transmission system for data transmission processing. By using the multi-channel image sensor data transmission system for data transmission processing, the optoelectronic system can perform real-time and efficient processing of the acquired image data, thereby enabling stable tracking of the captured target.

[0041] Finally, it should be noted that the features mentioned and / or shown in the above description of exemplary embodiments of the present invention can be combined in the same or similar manner with one or more other embodiments, combined with features in other embodiments, or substituted for corresponding features in other embodiments. These combined or substituted technical solutions should also be considered to be included within the scope of protection of the present invention.

Claims

1. A multi-channel image sensor data transmission system based on FPGA-PCIe, characterized in that, It includes an FPGA circuit board, external memory, and a back-end image processing platform; the FPGA circuit board is connected to multiple image sensors; The FPGA circuit board includes an integrated image processing module, a bus control module, and a PCIe interface control module. The number of integrated image processing modules is the same as the number of image sensors. The integrated image processing modules are used to independently drive and configure multiple image sensors, simultaneously acquire various video signals obtained by the image sensors, and preprocess the acquired video signals. The bus control module is connected to the integrated image processing module and the external memory respectively, and is used to write the preprocessed video image data of the integrated image processing module into the external memory, and to arbitrate and schedule data read and write requests to the external memory; The PCIe interface control module is connected to the bus control module. Data read / write requests issued by the back-end image processing platform are received by the PCIe interface control module and then arbitrated and scheduled by the bus control module. The PCIe interface control module realizes bidirectional conversion between the FPGA internal bus protocol and the PCIe bus protocol. The back-end image processing platform is connected to the PCIe interface control module of the FPGA circuit board via the PCIe bus, and is used to receive pre-processed multi-channel video image data and perform post-processing.

2. The multi-channel image sensor data transmission system according to claim 1, characterized in that, The integrated image processing module includes an image acquisition module, an image preprocessing module, and an image caching module; The image acquisition module is connected to the external image sensor driver board through the IO interface of the FPGA circuit board, and is used for parameter configuration, signal driving, exposure control and video image data acquisition of the image sensor; The image preprocessing module is connected to the image acquisition module and is used to preprocess the acquired video signals. The image caching module is connected to the image preprocessing module and is used to temporarily store the video image data preprocessed by the image preprocessing module.

3. The multi-channel image sensor data transmission system according to claim 2, characterized in that, The preprocessing includes one or more of the following: image tap transformation, fixed noise reduction, RAW to RGB conversion, automatic white balance, color correction, automatic gain control, automatic brightness control, Gamma correction, and image enhancement.

4. The multi-channel image sensor data transmission system according to claim 2, characterized in that, The bus control module accesses the external memory through the external storage control module. The external storage control module is used to control the data read and write of the external memory and convert the interface bus protocol of the bus control module into the interface bus protocol of the external memory. The PCIe interface control module directly accesses the external memory through the bus control module and the external storage control module.

5. The multi-channel image sensor data transmission system according to claim 4, characterized in that, The external memory is an SDRAM memory, and the data read and write of the SDRAM memory adopts a burst transfer mode.

6. The multi-channel image sensor data transmission system according to claim 5, characterized in that, The read / write data transfer rate of the SDRAM memory is higher than the data output rate of the image sensor.

7. The multi-channel image sensor data transmission system according to claim 5, characterized in that, The integrated image processing module also includes a FIFO memory, the receiving end of which is connected to the image buffer module, and the reading end of which is connected to the bus control module; The image caching module writes video image data into the FIFO memory using the clock and horizontal / vertical synchronization signals of the video data stream. The video image data written into the FIFO memory awaits reading by the bus control module. The video data stream is a carrier form for transmitting video image data over a physical link.

8. The multi-channel image sensor data transmission system according to claim 7, characterized in that, The read clock and read data width of the FIFO memory are greater than the write clock and write data width of the FIFO memory; The write clock is the pixel clock of the sensor, the write data width is the video signal data width of the image sensor, the read clock is the data read / write clock of the external memory, and the read data width is the data width in the external memory.

9. An optoelectronic system mounted on an unmanned aerial vehicle and a multi-platform collaborative system, characterized in that, Data transmission processing is performed using the multi-channel image sensor data transmission system according to any one of claims 1 to 8.

Images