Low-latency UDP video transmission system based on FPGA and implementation method of low-latency UDP video transmission system

Abstract

The invention belongs to the field of video transmission, and particularly relates to a low-delay UDP (User Datagram Protocol) video transmission system based on an FPGA (Field Programmable Gate Array) and an implementation method thereof. The system comprises a video data interface module, a dynamic UDP (User Datagram Protocol) subpackage logic module, a network interface controller interface module and a network state monitoring module which are carried on an FPGA (Field Programmable Gate Array), the input end of the video data interface module is connected with an external video encoder, and the output end of the video data interface module is connected with the data input end of the dynamic UDP subpackage logic module; the data output end of the dynamic UDP subpackage logic module is connected with the data input end of the network interface controller interface module; the output end of the interface module of the network interface controller is connected with an external physical layer chip; the monitoring end of the network state monitoring module monitors the network state between the physical layer chip and the receiving end, and the output end of the network state monitoring module is connected with the network input end of the dynamic UDP subpackage logic module. According to the invention, the effective load size of the next UDP data packet can be adaptively determined in real time according to the monitored network parameters so as to balance delay and throughput.

Description

Technical Field

[0001] The present invention relates to a video transmission system, and in particular to a low-delay UDP video transmission system based on FPGA and an implementation method thereof. Background Art

[0002] A software-based UDP video transmission system based on a general-purpose processor (CPU): This is the most common implementation method. In this type of system, an application or operating system kernel running on the CPU of a server or embedded device is responsible for receiving the data stream from the video encoder. The software logic segments the video data and fills it into a memory buffer based on a certain strategy (usually configuring a fixed target packet size, such as 1400 bytes, slightly smaller than the maximum transmission unit (MTU) of the network path to avoid IP layer fragmentation and maximize single-packet efficiency). When the data in the buffer reaches the preset size, or when there is insufficient data but the waiting timeout expires, the software calls the operating system's network protocol stack interface (such as the Socket API). The operating system then completes the addition of the UDP / IP header, the checksum calculation (either performed by the CPU or partially offloaded by the network card), and the packet transmission. Although some complex software solutions may allow the application layer to adjust sending behavior based on rough network feedback (such as receiving-end statistics) or video content (such as frame-by-frame packaging), its real-time nature is limited by software execution efficiency and operating system scheduling delays, and the decision on packet size is usually not based on fine-grained, real-time network transmission parameters (such as millisecond-level RTT changes and instantaneous packet loss rate) for dynamic adjustment on a packet-by-packet basis.

[0003] FPGA network acceleration solutions using fixed packetization strategies: To improve performance, some advanced network interface cards (NICs) or dedicated hardware systems utilize FPGAs to accelerate network packet processing. In these solutions, the FPGA may directly receive the video data stream and perform UDP / IP header encapsulation and checksum calculation in hardware before handing the packet over to the MAC / PHY layer for transmission. This can significantly reduce protocol processing latency. However, in known FPGA solutions of this type, the packetization strategy for UDP data is typically still based on fixed logic. For example, the FPGA internally implements a FIFO buffer. The generation and transmission of a UDP packet is triggered only when the accumulated data in the buffer reaches a fixed hardware-preset threshold (e.g., 1KB or 1.4KB). Although this approach operates at the hardware level, its packetization strategy lacks the ability to perceive and adapt to real-time, dynamically changing network conditions. It does not leverage real-time network feedback to dynamically optimize the size of the packets to be sent.

[0004] Introducing unnecessary packetization delays: Whether using a fixed-size packetization strategy implemented in software or hardware, when the video data generation rate is low or unstable, or when data is interrupted before a packet is fully loaded, the sender must wait for subsequent data to arrive to fill the preset packet size, or wait for a preset timeout. This "packetization" wait directly increases the time between data generation and actual transmission, resulting in significant end-to-end latency, particularly affecting real-time interactive applications that are extremely sensitive to latency.

[0005] Lack of adaptability to network conditions: The latency, packet loss rate, and available bandwidth along the network transmission path change dynamically. Fixed packet size policies cannot be optimized based on these real-time changes. When network conditions are good (low latency, low packet loss), sticking to a smaller fixed packet size may not fully utilize network capacity, limiting throughput. However, when network conditions deteriorate (high latency, high packet loss), continuing to send larger fixed data packets increases the probability and impact of single packet loss. Once lost, recovery takes longer or the retransmission cost is higher (if retransmission is performed at the application layer), further deteriorating the experience. This "one-size-fits-all" policy results in poor system performance in a changing network environment.

[0006] Inherent latency and resource consumption of software implementations: For pure software implementations, even without considering the latency of the packetization strategy itself, software execution (including interrupt response, task scheduling, memory copying, and traversing multiple layers of the protocol stack) inherently introduces significant processing latency. During high-bitrate video transmission, the CPU's packetization process consumes significant computing resources, potentially becoming a system bottleneck. Summary of the Invention

[0007] The purpose of the present invention is to solve the technical problems in the existing technology that UDP video transmission solutions generally adopt a fixed packet grouping strategy, which introduces unnecessary delays, cannot adaptively adjust the transmission efficiency according to real-time network conditions, and have inherent delays and high CPU loads caused by software implementation. The present invention provides a low-latency UDP video transmission system based on FPGA and its implementation method.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] A low-latency UDP video transmission system based on FPGA, which is special in the following aspects:

[0010] It includes a video data interface module, a dynamic UDP packet subpacketization logic module, a network interface controller interface module and a network status monitoring module mounted on the FPGA;

[0011] The input end of the video data interface module is connected to the external video encoder, and the output end is connected to the data input end of the dynamic UDP subpacketization logic module, which is used to receive the external video data to be transmitted, decode and buffer it to obtain the video data stream, and output it to the dynamic UDP subpacketization logic module;

[0012] The data output end of the dynamic UDP packet subdivision logic module is connected to the data input end of the network interface controller interface module, and is used to calculate the size of the UDP data packet to be sent in real time according to the current network parameters, and obtain the video data of the corresponding data volume according to the calculated UDP data packet size to generate a UDP data packet, and then output it to the network interface controller interface module;

[0013] The output end of the network interface controller interface module is connected to the external physical layer chip, and is used to send the UDP data packet to the external physical layer chip, so as to be sent to the external receiving end through the physical layer chip using the network;

[0014] The monitoring end of the network status monitoring module monitors the network status between the physical layer chip and the receiving end, and the output end is connected to the network input end of the dynamic UDP packet subpacketization logic module, which is used to obtain network parameters according to the network status and output them to the dynamic UDP packet subpacketization logic module.

[0015] Furthermore, the dynamic UDP packet subpacketization logic module includes a decision engine, a data assembly unit, and a UDP header generation and encapsulation unit;

[0016] The input end of the decision engine is connected to the output end of the video data interface module and the output end of the network parameter monitoring module respectively, and the output end is connected to the first input end of the data assembly unit and the UDP header generation and encapsulation unit respectively; the decision engine is used to calculate the size of the UDP data packet to be sent in real time according to the current network parameters, and output it to the data assembly unit and the UDP header generation and encapsulation unit respectively;

[0017] The second input end of the data assembly unit is connected to the output end of the video data interface module, and the output end is connected to the second input end of the UDP header generation and encapsulation unit; the data assembly unit is used to obtain the video data of the corresponding data volume according to the calculated UDP data packet size, and output it to the UDP header generation and encapsulation unit;

[0018] The output end of the UDP header generation and encapsulation unit is connected to the data input end of the network interface controller interface module; the UDP header generation and encapsulation unit is used to generate a standard UDP header of corresponding field length according to the calculated UDP data packet size, and assemble it with the acquired video data to generate a UDP data packet, and then output it to the network interface controller interface module.

[0019] Further, the dynamic UDP packet splitting logic module is used to determine whether the current network parameters meet the set threshold conditions. If they are met, the large packet adjustment formula is used to calculate and adjust the size of the UDP packet to be sent in real time; if not, the small packet adjustment formula is used to calculate and adjust the size of the UDP packet to be sent in real time; video data with the corresponding data volume is obtained according to the calculated UDP packet size to generate a UDP packet, and then it is output to the network interface controller interface module;

[0020] The large packet adjustment formula is: UDP packet size = min(maximum UDP packet size, current UDP packet size × growth factor);

[0021] The small packet adjustment formula is: UDP packet size = max(minimum UDP packet size, current UDP packet size × decay factor);

[0022] In the formula: the growth factor and the decay factor are set using the AIMD strategy in TCP congestion control; the UDP packet is smaller than the maximum transmission unit of the network.

[0023] Further, the network parameters include network round-trip time, packet loss rate, and network jitter;

[0024] The threshold conditions are: network round-trip time < RTT threshold, packet loss rate < RLR threshold, and network jitter < Jitter threshold.

[0025] Further, the dynamic UDP packet splitting logic module is used to smoothly adjust the size of the UDP packet to be sent using a sliding window or exponential smoothing algorithm.

[0026] Further, the network status monitoring module is used to monitor the network status between the physical layer chip and the receiving end through passive monitoring or active monitoring or passive + active monitoring;

[0027] The passive monitoring is to analyze the network status feedback information returned from the receiving end; the active monitoring is to actively send a probe packet to request the receiving end to reply and analyze the response.

[0028] Meanwhile, the present invention also provides an implementation method of a low-latency UDP video transmission system based on FPGA, which is characterized in that it includes the following steps:

[0029] Step 1, receive the external video data to be transmitted through the video data interface module, decode and buffer it to obtain a video data stream, and output it to the dynamic UDP packet splitting logic module;

[0030] Step 2: The network status monitoring module monitors the network status between the physical layer chip and the receiving end in real time, obtains the current network parameters, and outputs them to the dynamic UDP packet subpacketization logic module;

[0031] Step 3: The dynamic UDP packet subassembly logic module calculates the size of the UDP data packet to be sent in real time according to the current network parameters, obtains the corresponding amount of data from the video data stream according to the calculated UDP data packet size, generates a UDP data packet, and then outputs it to the network interface controller interface module;

[0032] Step 4: Send the UDP data packet to the external physical layer chip through the network interface controller interface module, so as to be transmitted to the external receiving end through the physical layer chip using the network.

[0033] The beneficial effects of the present invention are:

[0034] 1. Dynamic UDP packet size adjustment mechanism based on real-time network status feedback: The payload size of the next UDP packet is adaptively determined in real time based on monitored network parameters (such as latency and packet loss rate) to balance latency and throughput, which is different from all fixed packet size solutions.

[0035] 2. Implement the dynamic adjustment and packet assembly logic in FPGA hardware: Implement the dynamic decision-making and UDP packet construction process entirely or primarily through hardware logic within the FPGA (rather than software), leveraging the parallelism, pipelining, and low-latency characteristics of the hardware to ensure real-time performance.

[0036] 3. FPGA system architecture optimized for low-latency video transmission: This combines the aforementioned dynamic packetization mechanism with optimizations for video streaming (such as frame awareness) and other FPGA hardware acceleration technologies to form a complete, end-to-end low-latency UDP video transmission hardware acceleration system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a structural diagram of an embodiment of a low-latency UDP video transmission system based on FPGA of the present invention;

[0038] Figure 2 This is a structural diagram of a dynamic UDP packet subpacketization logic module in an embodiment of a low-latency UDP video transmission system based on FPGA of the present invention;

[0039] Figure 3 The present invention is a flowchart of an embodiment of a method for implementing a low-latency UDP video transmission system based on FPGA. DETAILED DESCRIPTION

[0040] To further clarify the objectives, advantages, and features of the present invention, the following describes in further detail an FPGA-based low-latency UDP video transmission system and its implementation method, in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent based on the following specific implementations.

[0041] Reference Figure 1 This embodiment provides a low-latency UDP video transmission system based on FPGA, whose core processing logic is deployed inside an FPGA chip. The system includes:

[0042] Video Data Interface Module: As the system's video data entry point, it receives raw or encoded video data streams from external video sources (e.g., camera sensors or video encoders). This module typically includes an input buffer (e.g., FIFO) and receives data according to a predefined interface standard (e.g., MIPI CSI-2, AXI-Stream, etc.), then passes it to the dynamic UDP packetization logic module.

[0043] Network Status Monitoring Module: This module is a key input source for dynamic adjustments. It is configured to monitor one or more parameters reflecting the quality of the current network transmission path in real time or near real time. These parameters may include, but are not limited to, round-trip time (RTT), packet loss rate (PLR), jitter, or estimated available bandwidth. Monitoring can be achieved by parsing feedback information returned from the receiver (such as RTCP reports), actively sending probe packets to request a reply from the receiver, analyzing the response, or a combination of both. After processing, this module outputs quantified network parameters to the dynamic UDP packetization logic module.

[0044] Dynamic UDP packet sub-module: This is the core innovation unit of the present invention, which is completely implemented by FPGA hardware logic. Figure 2 , the module can be divided into:

[0045] The decision engine receives real-time network parameters from the network status monitoring module and buffer status information from the video data interface module. Its core function is to calculate and determine the optimal payload size for the next UDP packet to be sent in real time based on internally preset or configurable dynamic packetization strategies / algorithms. This strategy is key to distinguishing it from the fixed strategies of existing technologies. For example, it can adopt adaptive logic such as increasing the UDP packet size to improve efficiency when the network is good (low latency, low packet loss), and reducing the UDP packet size to reduce risk and latency when the network is poor.

[0046] The steps to calculate the optimal payload size are as follows:

[0047] 1. Monitor network status

[0048] i. Real-time parameter collection: Obtain data such as RTT, packet loss rate, and jitter through the network interface. Utilize receiver feedback (such as RTCP reports) to further adjust the UDP packet size strategy.

[0049] ii. Statistical analysis: Use the sliding window method to smooth the statistics of these parameters to obtain a reliable network status assessment.

[0050] 2. Evaluate buffer status

[0051] i. Monitor fill level: Check the current amount of data in the video data buffer.

[0052] ii. Dynamic adjustment: Adjust the UDP packet size according to the buffer status to avoid buffer overflow (too much data) or idle (not enough data).

[0053] 3. Calculate the optimal UDP packet size

[0054] A) Adaptive strategy:

[0055] i. When the network status is good (low RTT, low packet loss rate, and small jitter), increase the UDP packet size to improve transmission efficiency.

[0056] ii. When the network status is poor (high RTT, high packet loss rate, and high jitter), reduce the UDP packet size to reduce latency and packet loss risks.

[0057] B) Algorithm implementation:

[0058] i. Set Thresholds: Define thresholds for parameters such as RTT and packet loss rate, such as RTT_Threshold and Packet Loss Rate_Threshold. Threshold settings depend on the actual application scenario. RTT_Threshold: This is related to the video frame rate and encoding latency. For example, a 60fps video has a frame interval of 16.7ms; the RTT_Threshold should be significantly lower than this to ensure real-time performance. PLR_Threshold: This is related to the packet loss resilience of the video encoding. A high packet loss rate can result in degraded image quality.

[0059] C) UDP packet size adjustment formula:

[0060] i. If RTT < RTT_threshold and packet loss rate < packet loss rate_threshold, that is, the network condition is good, then: UDP packet size = min(max UDP packet size, current UDP packet size × growth factor). Otherwise, that is, the network condition is poor: UDP packet size = max(min UDP packet size, current UDP packet size × decay factor). The settings of the growth factor and decay factor refer to the AIMD (Additive Increase Multiplicative Decrease) strategy in TCP congestion control to ensure that the system can quickly adapt and remain stable when the network condition changes.

[0061] ii. Smoothing process: Use a sliding window or exponential smoothing algorithm to avoid drastic fluctuations in packet size.

[0062] D) MTU limit: Ensure that the UDP packet size does not exceed the maximum transmission unit (MTU) of the network to avoid IP fragmentation.

[0063] 4. Data assembly and encapsulation

[0064] i. Read data: According to the calculated UDP packet size, read the corresponding amount of data from the video buffer.

[0065] ii. Generate UDP packets: Add the UDP header (including source port, destination port, length, and checksum) to encapsulate into complete packets. Use FPGA hardware acceleration for UDP header generation and checksum calculation to improve processing efficiency.

[0066] 5. Send packets

[0067] Transmit the encapsulated UDP packets to the network interface to complete the sending.

[0068] Example process

[0069] Assume that the initial UDP packet size is 1000 bytes, the maximum UDP packet size is 1400 bytes, the minimum UDP packet size is 500 bytes, the growth factor is 1.2, the decay factor is 0.8, the RTT_threshold is 50 ms, and the packet loss rate_threshold is 2%;

[0070] Current RTT = 30 ms, packet loss rate = %, (good network status):

[0071] UDP packet size = min(1400, 1000 × 1.2) = 1200 bytes.

[0072] Current RTT = 80 ms, packet loss rate = 5% (poor network status):

[0053] 77>

[0073] UDP packet size = max(500, 1000 × 0.8) = 800 bytes.

[0074] Data assembly unit: Based on the precise payload size determined by the decision engine, it reads the corresponding number of bytes of video data from the buffer of the video data interface module.

[0075] UDP Header Generation and Encapsulation Unit: This unit is responsible for generating a standard UDP header. Crucially, its "Length" field is dynamically filled based on the actual payload size calculated by the decision engine. Furthermore, this unit preferably calculates the UDP checksum quickly in hardware. Finally, it assembles the generated UDP header with the payload received from the data assembly unit to form a complete UDP packet. The entire module utilizes a deeply pipelined design, ensuring highly parallelized decision-making, data retrieval, and encapsulation steps, resulting in extremely low processing latency.

[0076] Network Interface Controller (NIC): This module is responsible for sending UDP packets, constructed by the module, to the physical layer chip (PHY) outside the FPGA according to standard network interface protocols for transmission over the network medium. This module typically includes the necessary send buffer management and scheduling logic.

[0077] System dynamic operation process: refer to Figure 3 During system operation, video data continuously flows into the video data interface module and is buffered. Simultaneously, the network status monitoring module continuously monitors network status and updates real-time parameters to the dynamic UDP packetization logic module. The decision engine within the dynamic UDP packetization logic module calculates the optimal data size for the next packet almost instantly, or based on very small time windows or data blocks, based on the latest network parameters and internal policies. Once the decision is made, the data assembly unit immediately retrieves the corresponding data from the video data interface module, while the header generation unit prepares a UDP header with the correct length field. These two are combined to form a UDP packet, which is then queued for transmission at the network interface controller module. Crucially, this process is continuous, and the UDP packet size changes dynamically based on real-time calculations, with virtually no delay caused by waiting for a fixed size. The entire process is efficiently executed within the FPGA hardware pipeline.

Claims

1. A low-latency UDP video transmission system based on FPGA, characterized by: It includes a video data interface module, a dynamic UDP packet subpacketization logic module, a network interface controller interface module and a network status monitoring module mounted on the FPGA; The input end of the video data interface module is connected to the external video encoder, and the output end is connected to the data input end of the dynamic UDP packet sub-packetization logic module, which is used to receive the external video data to be transmitted, decode and buffer it to obtain the video data stream, and output it to the dynamic UDP packet sub-packetization logic module; The data output end of the dynamic UDP packet subdivision logic module is connected to the data input end of the network interface controller interface module, and is used to calculate the size of the UDP data packet to be sent in real time according to the current network parameters, and obtain the video data of the corresponding data volume according to the calculated UDP data packet size to generate a UDP data packet, and then output it to the network interface controller interface module; The output end of the network interface controller interface module is connected to the external physical layer chip, and is used to send the UDP data packet to the external physical layer chip, so as to be sent to the external receiving end through the physical layer chip using the network; The monitoring end of the network status monitoring module monitors the network status between the physical layer chip and the receiving end, and the output end is connected to the network input end of the dynamic UDP packet subpacketization logic module, which is used to obtain network parameters according to the network status and output them to the dynamic UDP packet subpacketization logic module.

2. The low-latency UDP video transmission system based on FPGA according to claim 1, characterized in that: The dynamic UDP packet subpacketization logic module includes a decision engine, a data assembly unit, and a UDP header generation and encapsulation unit; The input end of the decision engine is connected to the output end of the video data interface module and the output end of the network parameter monitoring module respectively, and the output end is connected to the first input end of the data assembly unit and the UDP header generation and encapsulation unit respectively; the decision engine is used to calculate the size of the UDP data packet to be sent in real time according to the current network parameters, and output it to the data assembly unit and the UDP header generation and encapsulation unit respectively; The second input end of the data assembly unit is connected to the output end of the video data interface module, and the output end is connected to the second input end of the UDP header generation and encapsulation unit; the data assembly unit is used to obtain the video data of the corresponding data volume according to the calculated UDP data packet size, and output it to the UDP header generation and encapsulation unit; The output end of the UDP header generation and encapsulation unit is connected to the data input end of the network interface controller interface module; the UDP header generation and encapsulation unit is used to generate a standard UDP header of corresponding field length according to the calculated UDP data packet size, and assemble it with the data obtained from the video data stream to generate a UDP data packet, and then output it to the network interface controller interface module.

3. The FPGA-based low-latency UDP video transmission system according to claim 1 or 2, characterized in that: The dynamic UDP packet splitting logic module is used to determine whether the current network parameters meet the set threshold conditions. If they are met, the large packet adjustment formula is used to calculate and adjust the size of the UDP data packet to be sent in real time; If not, the small packet adjustment formula is used to calculate and adjust the size of the UDP data packet to be sent in real time; the video data of the corresponding data volume is obtained according to the calculated UDP data packet size to generate a UDP data packet, and then output to the network interface controller interface module; The large packet adjustment formula is: UDP data packet size = min(maximum UDP data packet size, current UDP data packet size × growth factor); The small packet adjustment formula is: UDP data packet size = max(minimum UDP data packet size, current UDP data packet size × decay factor); In the formula: the growth factor and the decay factor are set using the AIMD strategy in TCP congestion control; the UDP data packet is smaller than the maximum transmission unit of the network.

4. The low-latency UDP video transmission system based on FPGA according to claim 3, wherein: The network parameters include network round-trip time, packet loss rate, and network jitter; The threshold conditions are: network round-trip time < RTT threshold, and packet loss rate < RLR threshold, and network jitter < Jitter threshold.

5. The low-latency UDP video transmission system based on FPGA according to claim 4, wherein: The dynamic UDP packet splitting logic module is used to smoothly adjust the size of the UDP data packet to be sent using a sliding window or exponential smoothing algorithm.

6. The low-latency UDP video transmission system based on FPGA according to claim 5, wherein: The network status monitoring module is used to monitor the network status between the physical layer chip and the receiving end through passive monitoring or active monitoring or passive + active monitoring; The passive monitoring is to analyze the network status feedback information returned from the receiving end; The active monitoring is to actively send a probe packet to request the receiving end to reply and analyze the response.

7. A method for implementing a low-latency UDP video transmission system based on FPGA according to any one of claims 1 to 6, characterized in that: It includes the following steps: Step 1, receive the external video data to be transmitted through the video data interface module, decode and buffer it to obtain the video data stream, and output it to the dynamic UDP packet splitting logic module; Step 2, monitor the network status between the physical layer chip and the receiving end in real time through the network status monitoring module, obtain the current network parameters, and output them to the dynamic UDP packet splitting logic module; Step 3, calculate the size of the UDP data packet to be sent in real time through the dynamic UDP packet splitting logic module according to the current network parameters, and obtain the video data of the corresponding data volume according to the calculated UDP data packet size to generate a UDP data packet, and then output it to the network interface controller interface module; Step 4, send the UDP data packet to the external physical layer chip through the network interface controller interface module, so as to be transmitted to the external receiving end through the physical layer chip using the network.

Images