High-bandwidth camera based on FPGA and USB3.2

Abstract

The invention discloses a high-bandwidth camera based on an FPGA (Field Programmable Gate Array) and a USB3.2 (Universal Serial Bus 3.2). The high-bandwidth camera comprises three parts, namely an image sensor driving part, an FPGA data sending part and a USB3.2 data transmission part, firstly, an image sensor is driven to obtain original image data, and the data of the image sensor are analyzed through an FPGA; secondly, image sensor data obtained in the FPGA are sent to the USB3.2 chip through the LVDS interface; and finally, the data is sent to an upper computer for display through the dual-channel DMA of the USB3.2. The system can meet application scenes with high real-time requirements, the transmission bandwidth can reach 8.5 Gbps, and the system can be suitable for high-resolution, high-bandwidth and high-frame-rate data acquisition and transmission.

Description

Technical Field

[0001] The present invention belongs to the field of image acquisition, and in particular relates to a high-bandwidth camera based on FPGA and USB3.2. Background Art

[0002] In today's camera system landscape, the transmission interface plays a crucial role in determining the performance of the system. Currently, high-bandwidth transmission protocols such as Camera Link and CXP exist that can meet the needs of some applications with extremely high data transfer rates. However, their requirement for specific PCIE card slots and transmission cables limits the system's versatility and portability. USB interfaces are widely used in computers, and their flexibility and versatility have made them a mainstream transmission interface for camera systems. However, the current maximum theoretical transmission bandwidth of USB 3.0 interfaces is limited to 5Gbps, which cannot meet the transmission requirements of high resolution, high bandwidth, and high frame rates, thus limiting the development of camera systems in some high-end industrial applications. Therefore, to achieve high-speed image data transmission and improve the transmission frame rate of camera systems, LVDS transmit interfaces and USB 3.2 transmission have become important solutions to address this transmission bottleneck.

[0003] USB camera interfaces are relatively widely used due to their portability and flexibility. However, the theoretical bandwidth of the fastest USB camera currently available can only reach 5Gbps. The transmission interface uses a single-ended input and output port for data transmission, with a maximum speed of 320Mbps, which is far from meeting the requirements of high bandwidth, high frame rate, high resolution, and real-time performance.

[0004] In view of the above problems of the current method, it is necessary to invent a high-bandwidth camera based on FPGA and USB3.2, combining the parallel computing of FPGA and the fast transmission rate of USB3.2 chip to speed up the transmission bandwidth and accuracy of image data, and at the same time use dual-channel data transmission to reduce the gap in data transmission. Summary of the Invention

[0005] A high-bandwidth camera based on FPGA and USB3.2, the high-bandwidth transmission system includes the following steps:

[0006] The interface used by the high-bandwidth camera is the LVDS interface, which is 16 channels and has a theoretical bandwidth of up to 10Gbps, which is about 192% higher than the transmission bandwidth of USB3.0.

[0007] After the camera is powered on, the LVDS interface must first be initialized and configured with parameters. The USB3.2 chip sends the relevant configuration parameters, such as bit depth, sync word, resolution, and amount of data transmitted, through the IIC bus.

[0008] According to the synchronization word transmitted by the USB3.2 chip, at the beginning of training, USB3.2 will send a physical layer training start signal through a specific pin. The FPGA will detect the signal in real time. Once the training start is detected, it will start training. During the training, the FPGA will continuously send synchronization words through the LVDS interface for at least 50us. After the training is completed, a physical layer training end flag signal will be generated.

[0009] After the physical layer training is completed, it will enter the next stage of training, the link layer training. During the training process, synchronization words will be continuously sent. After the training is completed, a training end flag will be generated, and preparations will be made for entering the next stage.

[0010] After the training phase, the device polling phase begins. During this phase, the device is constantly checked to see if it is active. Since the chip supports simultaneous data transmission from multiple devices, polling is performed only between the idle state and these device numbers. When the PC software sends a request to collect data, this phase ends and the data transmission phase begins.

[0011] When the host computer software initiates a data request, the state machine switches from the device polling phase to the data transmission phase. During this phase, the device continuously sends image data to the host via the LVDS interface until the host computer's request signal ends. Throughout the data transmission process, the device notifies the DMA buffer status through a specific pin. When the pin signal is 0, it indicates that the DMA buffer is empty, and image data can be written to the buffer via the LVDS interface. When the pin signal is 1, it indicates that the DMA buffer is full, and the LVDS interface needs to suspend data transmission to prevent data overflow.

[0012] When the buffer of the USB3.2 chip receives data, DMA starts to move the data. When the buffer is empty, DMA will not move the data. When data is detected, DMA will start to transmit the data to the host computer through dual channels.

[0013] Compared with existing technologies:

[0014] 1. The present invention proposes to use an LVDS interface to transmit image data, which increases the efficiency of transmitting data by 192% compared to the single-ended input and output port of USB3.0, greatly improving the transmission efficiency and transmission speed.

[0015] 2. The system of the present invention can ensure stable operation in complex scenarios and provides a solution for stable and high-speed transmission of industrial cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the system of the present invention;

[0017] Figure 2 This is a schematic diagram of initialization parameter configuration of the present invention;

[0018] Figure 3 This is a schematic diagram of the physical layer training of the present invention;

[0019] Figure 4 This is a schematic diagram of link layer training of the present invention;

[0020] Figure 5 A schematic diagram of successful training of the present invention;

[0021] Figure 6 Schematic diagram of data transmission;

[0022] Figure 7 This is a schematic diagram of the host computer bandwidth test. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] like Figure 1 As shown in the figure, the block diagram of the high-bandwidth camera system is shown in the figure. The image data collected by the image sensor is transmitted to the FPGA chip for processing. Secondly, the processed image data is transmitted to the USB3.2 chip via LVDS. Finally, the received image data is transmitted to the host computer for real-time display.

[0025] like Figure 2 As shown in FIG, the parameter configuration required when the LVDS interface is powered on, such as the bit depth, the width and height of the image resolution, the total image data volume, the device number, and the synchronization word required for training.

[0026] like Figure 3 As shown in the timing diagram of physical layer training, the start_phy_ts_counter signal indicates the flag for starting training; ds_phy_counter is the training counter; P0_CTL_TX_DATA, P1_CTL_TX_DATA, P0_D0_TX_DATA~P0_D7_TX_DATA and P1_D0_TX_DATA~P1_D7_TX_DATA ports are data transmission ports respectively, and 0x39 indicates the synchronization word during physical layer training.

[0027] like Figure 4As shown in the figure, a schematic diagram of link layer training is shown, where P0_CTL_TX_DATA and P1_CTL_TX_DATA and LVDS data channels P0_D0_TX_DATA~P0_D7_TX_DATA and P1_D0_TX_DATA~P1_D7_TX_DATA represent the data channel and control channel of LVDS; 0xAA55AA55 is the synchronization word during link layer training.

[0028] like Figure 5 As shown, it is a schematic diagram of successful physical layer and link layer training, wherein the area marked by the rectangular box is an indication of successful physical layer and link layer training.

[0029] like Figure 6 Figure 1 shows a data transmission diagram. Socket 0 and socket 1 are different threads. When image data comes in, sockets 0 and 1 move the data to the corresponding buffers 1 to 4.

[0030] like Figure 7 As shown in the figure, the bandwidth and frame rate of the host computer are displayed. The frame rate of the USB3.2 interface transmitting a 3840×2160 resolution image can reach 64.17fps, and the bandwidth can reach 8.5Gbps.

[0031] In order to illustrate the technical solution of the present invention, some terms are explained below to facilitate understanding by those skilled in the art.

[0032] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0033] As a possible embodiment of the present invention, a high-bandwidth camera system is implemented.

[0034] S1. Drive the image sensor to obtain image data

[0035] The register is configured via the IIC bus to output image data.

[0036] S2, LVDS initialization process

[0037] When powered on, the USB3.2 chip configures the LVD initialization parameters through the IIC bus, such as bit depth, width and height of image resolution, total image data volume, device number, and synchronization word required for training.

[0038] S3, LVDS physical layer training

[0039] Reference Figure 3After the start_phy_ts_counter signal is pulled high, the FPGA begins physical layer training. The ds_phy_counter signal is a counter for physical layer training. When the counter reaches 50us, the start_phy_ts_counter signal is pulled low. While start_phy_ts_counter is high, channels such as P0_CTL_TX_DATA, P1_CTL_TX_DATA, P0_D0_TX_DATA to P0_D7_TX_DATA, and P1_D0_TX_DATA to P1_D7_TX_DATA continuously transmit the synchronization word 0x39 for physical layer training. When training is complete, data transmission stops.

[0040] S4, LVDS link layer training

[0041] Reference Figure 4 After the physical layer training is completed, the link layer training will begin. During the link layer training process, the synchronization word 0xAA55AA55 will be sent during the control channel P0_CTL_TX_DATA, P1_CTL_TX_DATA and the data channel P0_D0_TX_DATA~P0_D7_TX_DATA, P1_D0_TX_DATA~P1_D7_TX_DATA, and the transmission will stop after the transmission is completed.

[0042] S5. Data transmission

[0043] Reference Figure 6 ,Step 1: During socket initialization, socket 0 and socket 1 load DMA descriptor 1 and DMA descriptor 2 respectively.

[0044] Step 2: Once the data is valid, socket 0 transfers the data to DMA buffer 1 (Step 1a). The transfer length is L. At the end of this transfer, go to step 3.

[0045] Step 3: When new data arrives, socket 1 begins transferring the data. Socket 1 begins transferring data to DMA buffer 2 (Step 1b). Simultaneously, socket 0 loads DMA descriptor 3. When socket 1 completes transferring L amount of data, socket 0 is ready to transfer data to DMA buffer 3.

[0046] Step 4: When new data arrives again, the child socket 0 is switched. Socket 0 now transfers data of length L to DMA buffer 3 (Step 3a). Simultaneously, socket 1 loads DMA descriptor 4, preparing it to transfer data to DMA buffer 4. After socket 0 completes transferring data of length L, it proceeds to step 5.

[0047] Step 5: When new data comes in, GPIFIII transfers the valid data of socket 1 to DMA buffer 4 (Step 4a). At the same time, socket 0 loads DMA descriptor 1 and prepares to transfer data to DMA buffer 1.

[0048] Step 6: GPIF III switches sockets again, and socket 0 starts transferring data of length L to DMA buffer 1. And so on, the two threads of GPIF III transfer data alternately.

[0049] In summary, the LVDS interface of the present invention transmits high-bandwidth data and uses dual channels to transmit image data, which can achieve a transmission bandwidth of up to 8.5 Gbps, which is 1920% higher than the transmission bandwidth of USB 3.0.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-bandwidth camera based on FPGA and USB3.2, characterized in that: The high-bandwidth camera system comprises the following steps: (1) Drive the CMOS image sensor to output image data. The image sensor data is first obtained through the MIPI parsing module of the FPGA chip; (2) transmitting the obtained image data through the LVDS interface, and transmitting the image data through the LVDS interface when the buffer of the USB3.2 chip is empty; (3) When the USB3.2 chip receives data, it transmits the data to the host computer through dual-channel DMA for display.

2. The high-bandwidth camera based on FPGA and USB3.2 according to claim 1, characterized in that: In step (1), the MIPI parsing module is implemented by an IP core in the FPGA. The IP core is used to parse the image sensor of the MIPI interface, and the RAW12 and RAW16 image data can be parsed.

3. The high-bandwidth camera based on FPGA and USB3.2 according to claim 2, characterized in that: In step (2), LVDS interface data transmission can ensure that the data is not affected by these interferences during the transmission process, thereby ensuring accurate signal transmission. At the same time, the LVDS interface transmission rate is higher than that of the single-ended interface transmission rate, up to 1.9Gbps, which can meet the needs of high-resolution and high-bandwidth video transmission.

4. The high-bandwidth camera based on FPGA and USB3.2 according to claim 3, characterized in that: In step (2), the LVDS interface data transmission first needs to be initialized and configured with parameters when powered on. The USB3.2 chip sends the relevant parameter configurations, such as bit depth, synchronization word, resolution, and amount of data transmitted, through the IIC bus.

5. The high-bandwidth camera based on FPGA and USB3.2 according to claim 4, characterized in that: In step (2), according to the synchronization word transmitted by the USB3.2 chip, at the beginning of training, USB3.2 will send a physical layer training start signal through a specific pin, and the FPGA will detect the signal in real time. Once the training start is detected, it will start training. During the training, the FPGA will continuously send synchronization words through the LVDS interface for at least 50us, and after the end, a physical layer training end flag signal will be generated.

6. The high-bandwidth camera based on FPGA and USB3.2 according to claim 5, characterized in that: In step (2), after the physical layer training is completed, the training process of the next stage will be entered, the link layer training. During the training process, synchronization words will be continuously sent. After the training is completed, a training end mark will be generated, and preparations will be made to enter the next stage.

7. The high-bandwidth camera based on FPGA and USB3.2 according to claim 6, characterized in that: In step (2), after the training phase is completed, the device polling phase will begin. In this phase, the device will be constantly checked to see if it is in an active state. Since the chip supports multiple devices transmitting data simultaneously, it will only continuously poll between the idle state and these device numbers. When the host computer software sends a request to collect data, this phase will end and the data transmission phase will then begin.

8. The high-bandwidth camera based on FPGA and USB3.2 according to claim 7, characterized in that: In step (2), when the host computer software initiates a data request, the state machine switches from the device polling phase to the data transmission phase. In this phase, the device continuously transmits image data through the LVDS interface until the host computer's request signal ends. During the entire data transmission process, the device notifies the DMA buffer status through a specific pin. When the pin signal is 0, it means that the DMA buffer is empty, and the image data can be written to the buffer through the LVDS interface. When the pin signal is 1, it means that the DMA buffer is full, and the LVDS needs to suspend data transmission to avoid data overflow.

9. The high-bandwidth camera based on FPGA and USB3.2 according to claim 8, characterized in that: In step (3), when the buffer of the USB3.2 chip receives data, DMA starts to move the data. When the buffer is empty, DMA does not move the data. When data is detected, DMA starts to transmit the data to the host computer through dual channels.

Images