Video data transmission method and system based on FPGA

Abstract

The invention discloses a video data transmission method and system based on an FPGA, and relates to data transmission. The method comprises the steps that a video data stream is acquired, and a time sequence signal of the video data stream is analyzed; writing the parallel data into an external memory to form a frame buffer area; packaging the read video data into a data packet meeting a USB3.2 transmission protocol; packaging the data packet based on a USB3.2 link layer protocol; and converting the USB3.2 protocol transmission frame into a differential signal, and transmitting video data in the USB3.2 protocol transmission frame according to the differential signal. In view of data loss caused by mismatching of a video input rate and a USB output rate in the prior art, by establishing an annular buffer structure and adopting a dynamic pointer management and waterline threshold monitoring mechanism, rate adaptive matching of a video data stream is realized, and continuous and stable transmission of video data is ensured.

Description

Technical Field

[0001] This application relates to the field of data transmission, and more specifically, to a video data transmission method and system based on FPGA. Background Technology

[0002] As video technology advances towards higher resolution (4K / 8K), higher frame rate (HFR), and higher dynamic range (HDR), the amount of video data is increasing dramatically, and the data generation rate of video sources is becoming increasingly diverse and dynamic. For example, a 4K@60fps video source can generate data at a rate of up to 12Gbps, while the output rate of video capture devices varies significantly across different application scenarios. Meanwhile, the USB interface, due to its universality and plug-and-play nature, is widely used to connect external video devices. However, the actual transmission rate of the USB interface is significantly affected by factors such as protocol overhead, host scheduling, and bus contention. This mismatch between video input rate and USB output rate has become a key technical challenge restricting high-quality video transmission.

[0003] Existing technologies mainly employ the following solutions for video transmission: Software-based UVC solution: This solution relies on the host CPU to run a UVC driver and transmit video streams via the standard UVC protocol. This solution lacks an effective rate matching mechanism; when the video input rate momentarily exceeds the available USB bandwidth, it can only adapt by dropping frames or reducing resolution, resulting in video data loss. Furthermore, software buffer management has poor real-time performance and struggles to handle sudden rate changes.

[0004] Dedicated USB video bridging chip solution: This solution uses a fixed ASIC or dedicated chip to convert USB to a video interface. These chips typically employ a fixed-size FIFO buffer, lacking dynamic adjustment capabilities. When there is a significant difference between the input and output rates, buffer overflow or underload can easily occur, resulting in lost video frames or transmission interruptions.

[0005] The main problems with existing technologies in rate matching are: Insufficient buffering mechanism: Existing solutions mostly use simple FIFO or fixed buffers, which cannot effectively absorb the dynamic differences between video input rate and USB output rate. Especially in the case of long-term rate mismatch, it is easy to cause buffer overflow and data loss.

[0006] Lack of rate monitoring and adjustment mechanisms: Existing technologies lack the ability to monitor and dynamically adjust input and output rates in real time, and cannot adjust data flow control strategies according to actual transmission conditions, resulting in frequent frame drops when the network is congested or the host load changes.

[0007] Inefficient protocol: The traditional UVC protocol is designed based on USB 2.0 or earlier USB 3.0, with large protocol overhead and low actual effective bandwidth utilization, which further exacerbates the difficulty of rate matching. Even under the high bandwidth conditions of USB 3.2, it still cannot effectively solve the problem of reliable transmission of video data.

[0008] Insufficient real-time guarantee: The latency uncertainty of the software processing solution and the lack of flexibility of the hardware solution cannot provide predictable rate matching guarantee for video data transmission, resulting in poor performance in applications with strict requirements for data integrity, such as industrial inspection and medical imaging.

[0009] Therefore, there is an urgent need for a video data transmission solution to solve the problems of data loss and unstable transmission caused by the dynamic mismatch between video input rate and USB output rate. Summary of the Invention

[0010] To address the data loss caused by the mismatch between video input rate and USB output rate in existing technologies, this application provides a video data transmission method and system based on FPGA. By establishing a circular buffer structure and adopting dynamic pointer management and waterline threshold monitoring mechanisms, the rate of the video data stream is adaptively matched, ensuring continuous and stable transmission of video data.

[0011] One aspect of this application provides a video data transmission method based on FPGA, comprising: S1, acquiring a video data stream and parsing the timing signal of the video data stream, extracting pixel data from the timing signal, and converting the pixel data into parallel data; S2, writing the parallel data into an external memory to form a frame buffer; S3, reading video data from the frame buffer according to a preset reading strategy, encapsulating the read video data into a data packet that conforms to the USB 3.2 transmission protocol, the data packet containing video data payload and USB control protocol; S4, encapsulating the data packet based on the USB 3.2 link layer protocol, adding a link control header, sequence number, and error detection code to the data packet, and performing device enumeration response, endpoint configuration, and flow control according to the USB protocol to form a USB 3.2 protocol transmission frame; S5, converting the USB 3.2 protocol transmission frame into a differential signal, and transmitting the video data in the USB 3.2 protocol transmission frame according to the differential signal.

[0012] Further, S1, converting pixel data into parallel data includes: identifying the interface type of the video data stream, the interface type including at least one of MIPI, LVDS, or parallel digital video interface; determining timing parsing parameters according to the interface type; extracting line synchronization signals, field synchronization signals, and pixel clock signals from the video data stream according to the timing parsing parameters; extracting pixel data from the video data stream according to the line synchronization signals, field synchronization signals, and pixel clock signals; and converting the pixel data into parallel data.

[0013] Furthermore, the timing analysis parameters include: clock frequency, synchronization signal polarity, and data valid window parameters.

[0014] Further, in step S2, parallel data is written to external memory to form a frame buffer, including: calculating the storage capacity of a single frame of video data based on the resolution and frame rate of the video data stream, and allocating storage space of the corresponding size in the external memory according to the calculated storage capacity; dividing the allocated storage space into multiple frame buffer units to establish a circular buffer structure, with each frame buffer unit storing one frame of video data; determining the initial interval between the write pointer and the read pointer of the frame buffer according to the input rate of the parallel data in step S1 and the reading rate of the video data in step S3, and setting the waterline threshold of the buffer; writing the parallel data to the frame buffer unit currently pointed to by the write pointer in the circular buffer in a burst write manner; monitoring the relative position of the write pointer and the read pointer, and generating a buffer status signal when the interval between the write pointer and the read pointer reaches the set waterline threshold, and adjusting the data acquisition rate in step S1 or the data reading rate in step S3 according to the buffer status signal.

[0015] Furthermore, S3 reads video data from the frame buffer according to a preset reading strategy, including: determining the video data reading rate and data block size based on the bandwidth limitations of the USB 3.2 protocol and the transmission mode requested by the host; reading video pixel data line by line from the frame buffer unit pointed to by the current reading pointer according to the position of the reading pointer in the circular buffer, and determining the amount of data read each time according to the data block size; adding timestamps and frame sequence numbers to the read video pixel data to form a video data payload; adding a packet header to the video data payload according to the USB 3.2 protocol, the packet header containing endpoint address, data type identifier, and payload length information to form the data field of the USB data packet; adding USB control protocol information to the data field, the control protocol information including data packet sequence number, flow control information, and CRC checksum, to generate a data packet that meets the USB 3.2 transmission protocol.

[0016] Furthermore, S4 encapsulates the data packets based on the USB 3.2 link layer protocol, including: assigning a sequence number to each data packet; generating a link control header based on the current transmission status, the link control header containing a link command field and a data packet type identifier; adding the link control header to the data packet containing the sequence number; calculating the CRC-32 error detection code of the data packet with the link control header, adding the error detection code to the data packet, and forming a USB 3.2 protocol transmission frame.

[0017] Further, in step S5, the USB 3.2 protocol transmission frame is converted into a differential signal, and the video data in the USB 3.2 protocol transmission frame is transmitted according to the differential signal. This includes: encoding the USB 3.2 protocol transmission frame to obtain an encoded data stream; performing parallel-to-serial conversion on the encoded data stream, converting the parallel data into a serial bit stream according to the USB 3.2 transmission rate; converting the serial bit stream into a differential signal; and sending the USB 3.2 protocol transmission frame containing video data to the host through the differential signal.

[0018] Another aspect of this application provides an FPGA-based video data transmission system, comprising: a data acquisition module for acquiring a video data stream, parsing the timing signal of the video data stream, extracting pixel data from the timing signal, and converting the pixel data into parallel data; a frame buffer management module for writing the parallel data into an external memory to form a frame buffer; a data encapsulation module for reading video data from the frame buffer according to a preset reading strategy, encapsulating the read video data into a data packet that conforms to the USB 3.2 transmission protocol, wherein the data packet contains a video data payload and a USB control protocol; a link layer encapsulation module for encapsulating the data packet based on the USB 3.2 link layer protocol, adding a link control header, a sequence number, and an error detection code to the data packet, and performing device enumeration response, endpoint configuration, and flow control according to the USB protocol to form a USB 3.2 protocol transmission frame; and a physical layer conversion module for converting the USB 3.2 protocol transmission frame into a differential signal and transmitting the video data in the USB 3.2 protocol transmission frame according to the differential signal.

[0019] Furthermore, it also includes: an external memory connected to the memory interface of the FPGA chip to store video frame data; and a USB connector connected to the differential signal output terminal of the physical layer conversion module to transmit the differential signal to the host.

[0020] Compared to existing technologies, the advantages of this application are: (1) By establishing a multi-frame capacity circular buffer in external memory, and with independent control of the write pointer and read pointer, the instantaneous and continuous differences between the video input rate and the USB output rate can be effectively absorbed. When the input rate is higher than the output rate, the data is temporarily stored in the buffer; when the output rate recovers, the buffered data is released, avoiding the overflow and frame loss problem of the traditional fixed FIFO scheme, and realizing lossless transmission of video data; (2) The buffer usage status is monitored in real time through a waterline threshold monitoring mechanism. When the buffer occupancy reaches the preset threshold, a status signal is generated and fed back to the data acquisition and reading module to dynamically adjust the input and output rates. This closed-loop control mechanism can make preventive adjustments in the early stage of rate mismatch, avoid the buffer from reaching the limit state, and ensure the continuity and stability of video transmission. It is particularly suitable for video acquisition applications that run for a long time. Attached Figure Description

[0021] Figure 1 This is an exemplary flowchart of a video data transmission method based on FPGA according to this application; Figure 2 This is a block diagram of the overall structure of the FPGA-based USB32 video transmission solution of this application; Figure 3 This is a schematic diagram of the ring structure of the frame buffer and the management of read / write pointers in this application. Detailed Implementation

[0022] The present application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 like Figure 1 As shown, the process involves acquiring a video data stream, parsing its timing signals, extracting pixel data from the timing signals, and converting the pixel data into parallel data. The parallel data is then written to external memory to form a frame buffer. Video data is read from the frame buffer according to a preset reading strategy, and the read video data is encapsulated into data packets that conform to the USB 3.2 transmission protocol. These data packets contain video data payloads and USB control protocols. The data packets are encapsulated based on the USB 3.2 link layer protocol, with a link control header, sequence number, and error detection code added. Device enumeration responses, endpoint configurations, and flow control are executed according to the USB protocol to form USB 3.2 protocol transmission frames. These USB 3.2 protocol transmission frames are then converted into differential signals, and the video data within the USB 3.2 protocol transmission frames is transmitted based on these differential signals.

[0024] like Figure 2 This embodiment provides a high-bandwidth, low-latency, rate-adaptive video data transmission method based on FPGA, suitable for real-time video capture and USB 3.2 transmission scenarios at resolutions of 4K@60fps and above. The entire system uses the FPGA as the core controller, implementing video signal parsing, frame buffer management, protocol encapsulation, and physical layer driving through hardware logic, ensuring stable and lossless transmission of the video stream even under conditions of fluctuating input rates and limited USB output bandwidth.

[0025] I. Video signal acquisition and parallel processing (S1): The FPGA first receives raw video data streams from image sensors or video sources via its high-speed I / O pins. This data stream may use one of the following interfaces: MIPI CSI-2, LVDS, or a parallel digital video interface. The FPGA integrates a multi-protocol compatible timing analysis engine that automatically identifies the interface type and loads the corresponding timing analysis parameters, including: pixel clock frequency, horizontal sync signal (HSYNC) and vertical sync signal (VSYNC), and valid data.

[0026] For example, in MIPI CSI-2 mode, the FPGA unpacks the high-speed serial differential signal into parallel pixel data through the D-PHY receiver and extracts the embedded sync codes to reconstruct HSYNC / VSYNC; while in parallel interface mode, the external synchronization signal is sampled directly.

[0027] Subsequently, the FPGA performs bit-width alignment and format standardization on the extracted pixel data (e.g., YUV422 to RGB888), and converts it into a fixed-width parallel data bus (e.g., 64-bit or 128-bit) through an internal register set for efficient writing to external memory.

[0028] II. Ring Frame Buffer Construction and Dynamic Write Control (S2): like Figure 3 As shown, the FPGA connects to external SDRAM via its high-speed memory interface (DDR4 PHY) to construct a multi-frame capacity circular buffer. The buffer is divided into N independent frame buffer units, the size of which is dynamically calculated based on the resolution and color depth of the input video. For example, for a 4K (3840×2160)@60fps RGB888 format video, the data size of a single frame is approximately 3840×2160×3 bytes ≈ 24.9 MB; accordingly, the FPGA pre-allocates at least 3~5 frames of contiguous storage space in the DDR and establishes write pointers (WP) and read pointers (RP).

[0029] The writing process adopts a burst write strategy: when a whole row or a whole frame of pixel data is ready, the FPGA initiates a write request to the frame buffer unit pointed to by the current WP with the maximum burst length (such as 256 bytes), which greatly improves the DDR bandwidth utilization.

[0030] Simultaneously, the FPGA monitors the distance between WP and RP (i.e., buffer occupancy depth) in real time. The system presets high and low watermark thresholds (e.g., high watermark = 80%, low watermark = 20%). When WP approaches RP (the buffer is about to overflow), the FPGA generates a buffer full flag, which is fed back to the front-end data acquisition module, triggering any of the following adjustment mechanisms: pausing or reducing the output frame rate of the image sensor, inserting virtual line / field blanking extension to extend the effective data interval, or activating the backpressure mechanism to notify the upstream device to slow down. Conversely, when RP catches up with WP (the buffer is about to empty), the read rate can be appropriately increased or the USB endpoint configuration can be adjusted to improve throughput.

[0031] III. USB 3.2 Protocol Data Packet Encapsulation (S3): The FPGA's data encapsulation module reads video data from the circular buffer according to the transmission mode requested by the host. The reading strategy is to read in line order and block alignment, with the size of each data block read matching the maximum packet length of USB32 SuperSpeed ​​(typically 1024 bytes).

[0032] The read pixel data is appended with a timestamp and frame number to form a structured video payload. Subsequently, the FPGA adds a standard USB header to the payload according to the USB 3.2 protocol specification, which includes: endpoint address, data type identifier, payload length, and then appends USB control protocol fields, including: packet sequence number, flow control token, and CRC-16 checksum, to finally form a complete data packet that conforms to the USB 3.2 requirements.

[0033] IV. Link Layer Encapsulation and Error Control (S4): At the link layer, the FPGA assigns a unique link sequence number to each data packet and generates a link control header containing link commands and packet type identifiers. Subsequently, the FPGA calculates a CRC-32 error detection code for the entire link layer frame and appends it to the frame trailer. This mechanism can trigger retransmission or discarding when physical layer errors occur, ensuring data reliability. Furthermore, the FPGA has a built-in USB device state machine that automatically completes the following upon power-up: device enumeration response, endpoint configuration, flow control negotiation, and ensures the host can correctly identify the device as a UVC (USB Video Class) compatible device.

[0034] V. Physical Layer Differential Signal Generation and Transmission (S5): Finally, the FPGA's physical layer (PHY) module encodes the link layer frame into 8b / 10b or 128b / 132b values ​​(depending on USB 3.2 Gen1 / Gen2), generating an encoded parallel data stream. This data stream is then converted into a high-speed serial bit stream (e.g., 5 Gbps or 10 Gbps) by a serializer, and finally output as a pair of high-speed differential signals (TX+ / TX) by a differential driver. ).

[0035] Differential signals are transmitted to the host via a USB Type-C or Micro-B connector, completing end-to-end video data delivery.

[0036] Example 2 After the system powers on, the FPGA first initializes the USB 3.2 PHY and waits for the host to initiate an enumeration request. During the enumeration process, the FPGA returns a preset device descriptor, configuration descriptor, and UVC class-specific descriptor, enabling the host to recognize it as a standard USB video device.

[0037] Meanwhile, the video input interface module begins listening for data streams from the image sensor. Once a valid synchronization signal (HSYNC / VSYNC) is detected, the data acquisition module immediately initiates the pixel extraction process and sends the pixel data to the frame buffer management module in 64-bit parallel format.

[0038] The frame buffer management module dynamically allocates storage space in DDR according to the current video format (such as 1080p@30fps or 4K@60fps) and maintains the write pointer WP. When a frame is completely written, WP automatically jumps to the next free frame buffer unit and sets the "frame ready" flag.

[0039] The data encapsulation module polls this flag. Once a new frame is found to be ready, it reads data in blocks from the corresponding frame buffer unit according to the maximum packet size and transmission interval previously set by the host via the SET_CUR request. Each block read is encapsulated into a USB 3.2 isochronous transfer packet and handed over to the link layer encapsulation module for processing.

[0040] The link layer module adds a unique sequence number and link control header to each packet and calculates CRC-32. After encapsulation, the data enters the transmit FIFO of the physical layer conversion module.

[0041] The physical layer module continuously monitors the USB link status. When the link is in a "configured" and "no flow control blocking" state, the data in the FIFO is encoded at 128b / 132b at the USB3.2 Gen1 (5 Gbps) or Gen2 (10 Gbps) rate and the differential pair is driven out through a high-speed serializer.

[0042] Throughout the process, the frame buffer management module continuously monitors the distance between the WP and RP. If the buffer occupancy exceeds the high watermark (e.g., 85%), a "deceleration request" is sent to the data acquisition module; if it falls below the low watermark (e.g., 15%), the host is notified to increase the requested bandwidth (via in-band signaling or UVC control request). This closed-loop feedback mechanism significantly improves the system's robustness under non-steady-state input conditions.

[0043] In addition, if the physical layer detects consecutive CRC errors or the link retransmission limit is exceeded, the system can trigger a frame drop protection mechanism: skip the current damaged frame and directly read the next complete frame to avoid erroneous data from polluting the display screen, while recording error logs for debugging purposes.

[0044] The foregoing illustrative description of the present application and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. The accompanying drawings are only one embodiment of the present application, and the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present application, such designs should fall within the scope of protection of this application. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A video data transmission method based on FPGA, characterized in that, include: S1: Acquire video data stream, parse the timing signal of video data stream, extract pixel data from timing signal, and convert pixel data into parallel data; S2 writes parallel data to external memory to form a frame buffer; S3 reads video data from the frame buffer according to a preset reading strategy, and encapsulates the read video data into a data packet that meets the USB 3.2 transmission protocol. The data packet contains the video data payload and the USB control protocol. S4 encapsulates data packets based on the USB 3.2 link layer protocol, adds a link control header, serial number, and error detection code to the data packets, and performs device enumeration response, endpoint configuration, and flow control according to the USB protocol to form a USB 3.2 protocol transmission frame. S5 converts the USB 3.2 protocol transmission frame into a differential signal and transmits the video data in the USB 3.2 protocol transmission frame according to the differential signal.

2. The FPGA-based video data transmission method according to claim 1, characterized in that: S1 converts pixel data into parallel data, including: Identify the interface type of the video data stream, including at least one of MIPI, LVDS, or parallel digital video interface; Determine the timing parsing parameters based on the interface type, and extract the line synchronization signal, field synchronization signal, and pixel clock signal from the video data stream based on the timing parsing parameters; Pixel data is extracted from the video data stream based on the horizontal synchronization signal, the vertical synchronization signal, and the pixel clock signal. Convert pixel data into parallel data.

3. The FPGA-based video data transmission method according to claim 2, characterized in that: Timing analysis parameters include: clock frequency, synchronization signal polarity, and data valid window parameters.

4. The FPGA-based video data transmission method according to claim 2, characterized in that: S2, writes parallel data to external memory to form a frame buffer, including: Based on the resolution and frame rate of the video data stream, calculate the storage capacity of a single frame of video data, and allocate storage space of the corresponding size in external memory according to the calculated storage capacity. The allocated storage space is divided into multiple frame buffer units, and a circular buffer structure is established, with each frame buffer unit storing one frame of video data. Based on the input rate of parallel data in step S1 and the reading rate of video data in step S3, determine the initial interval between the write pointer and the read pointer of the frame buffer, and set the waterline threshold of the buffer. Parallel data is written to the frame buffer unit pointed to by the current write pointer in the circular buffer in a burst write manner; Monitor the relative positions of the write pointer and the read pointer. When the interval between the write pointer and the read pointer reaches the set waterline threshold, generate a buffer status signal. Adjust the data acquisition rate in step S1 or the data reading rate in step S3 according to the buffer status signal.

5. The FPGA-based video data transmission method according to claim 4, characterized in that: S3 encapsulates the read video data into data packets that conform to the USB 3.2 transmission protocol, including: Based on the bandwidth limitations of the USB 3.2 protocol and the transmission mode requested by the host, determine the video data read rate and data block size; Based on the position of the read pointer in the circular buffer, video pixel data is read sequentially from the frame buffer unit pointed to by the current read pointer, and the amount of data read each time is determined according to the data block size. Add timestamps and frame numbers to the read video pixel data to form the video data payload; According to the USB 3.2 protocol, a header is added to the video data payload. The header contains the endpoint address, data type identifier, and payload length information, forming the data field of the USB data packet. Add USB control protocol information to the data field. The control protocol information includes the data packet sequence number, flow control information and CRC check code, and generate a data packet that meets the USB 3.2 transmission protocol.

6. The FPGA-based video data transmission method according to claim 4, characterized in that: S4 encapsulates data packets based on the USB 3.2 link layer protocol, including: Assign a sequence number to each data packet; A link control header is generated based on the current transmission status. The link control header contains a link command field and a packet type identifier. Add the link control header to the data packet containing the sequence number; Calculate the CRC-32 error detection code of the data packet with the link control header, add the error detection code to the data packet, and form a USB 3.2 protocol transmission frame.

7. The FPGA-based video data transmission method according to claim 4, characterized in that: S5 converts the USB 3.2 protocol transmission frame into a differential signal, including: The USB 3.2 protocol transmission frames are encoded to obtain an encoded data stream; The encoded data stream is converted from parallel to serial, and the parallel data is converted into a serial bit stream according to the USB3.0 transmission rate. Convert a serial bit stream into a differential signal; The USB 3.2 protocol transmission frame containing video data is sent to the host via differential signaling.

8. A video data transmission system based on FPGA, characterized in that, include: The data acquisition module acquires the video data stream, parses the timing signal of the video data stream, extracts pixel data from the timing signal, and converts the pixel data into parallel data. The frame buffer management module writes the parallel data into an external memory to form a frame buffer; The data encapsulation module reads video data from the frame buffer according to a preset reading strategy, and encapsulates the read video data into a data packet that meets the USB 3.2 transmission protocol. The data packet contains the video data payload and the USB control protocol. The link layer encapsulation module encapsulates data packets based on the USB 3.2 link layer protocol, adds a link control header, serial number, and error detection code to the data packets, and performs device enumeration response, endpoint configuration, and flow control according to the USB protocol to form a USB 3.2 protocol transmission frame. The physical layer conversion module converts USB 3.2 protocol transmission frames into differential signals, and transmits the video data in the USB 3.2 protocol transmission frames according to the differential signals.

9. The FPGA-based video data transmission system according to claim 8, characterized in that: Also includes: An external memory, connected to the memory interface of the FPGA chip, stores video frame data; The USB connector is connected to the differential signal output terminal of the physical layer conversion module to transmit the differential signal to the host.

Images