UDP video transmission optimization system and method based on dynamic QoS

Abstract

The invention belongs to the field of video transmission optimization, and particularly relates to a UDP (User Datagram Protocol) video transmission optimization system and method based on dynamic QoS (Quality of Service). The system comprises a video stream classification and marking module, a UDP (User Datagram Protocol) data packet encapsulation module, a dynamic QoS (Quality of Service) control module, a priority queue management module and a receiving end, the input end of the video stream classification and marking module is connected with an external video encoder, and the output end of the video stream classification and marking module is connected with the input end of the UDP data packet packaging module; the output end of the UDP data packet packaging module is connected with the data input end of the priority queue management module; the data output end of the priority queue management module is connected with the input end of the receiving end through a network interface, and the control end is connected with the output end of the dynamic QoS control module; the output end of the receiving end is connected with a display or decoding device. According to the method, priority marking can be performed on the video frame data according to the criticality of the video frame data, the video frame data with different priorities can be distinguished, and the transmission strategy can be dynamically adjusted in real time according to the network state.

Description

Technical Field

[0001] The present invention relates to a video transmission optimization system and method, and in particular to a UDP video transmission optimization system and method based on dynamic QoS. Background Art

[0002] UDP (User Datagram Protocol) is a connectionless transport layer protocol widely used in scenarios such as video streaming due to its low overhead and high real-time performance. A video stream typically consists of I-frames (Intra-frames), P-frames (Predicted-frames), and B-frames (Bi-directional predicted-frames). I-frames contain complete image information and are the basis for video decoding. P-frames and B-frames rely on I-frames or other frames, have smaller data volumes, and are sensitive to packet loss. In environments with strong interference (such as electromagnetic pulses), UDP packet loss rates increase significantly, resulting in degraded video quality.

[0003] Current video transmission solutions often use static QoS policies, which pre-set fixed priorities before transmission. For example, this involves marking packet priorities at the IP layer (e.g., DiffServ). However, this approach cannot adapt policies based on real-time network conditions. Furthermore, the traditional UDP protocol lacks the ability to distinguish between important packets. In noisy environments, packets of all frame types are equally likely to be lost, severely impacting video stream quality. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems that the existing UDP protocol has a high packet loss rate in a strong interference environment, which cannot guarantee the reliable transmission of key data (I frames) in the video stream; lacks the ability to respond to network conditions in real time and cannot adapt to dynamically changing interference environments; cannot effectively distinguish and prioritize the transmission of key frames, resulting in resource waste and quality degradation, and to provide a UDP video transmission optimization system and method based on dynamic QoS.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A UDP video transmission optimization system based on dynamic QoS, which is special in the following aspects:

[0007] Including the sending end and the receiving end;

[0008] The transmitting end includes a video stream classification and marking module, a UDP data packet encapsulation module, a dynamic QoS control module and a priority queue management module;

[0009] The input end of the video stream classification and marking module is connected to the external video encoder, and the output end is connected to the input end of the UDP data packet encapsulation module; the video stream classification and marking module is used to receive the video data stream to be optimized for transmission; classify the video data stream into frame categories, including I frames, P frames and B frames; mark I frames as high priority, P frames as medium priority, and B frames as low priority; and output the video frame data with priority marks to the UDP data packet encapsulation module;

[0010] The output end of the UDP data packet encapsulation module is connected to the data input end of the priority queue management module. The UDP data packet encapsulation module is used to add a custom QoS field to the video frame data with the priority mark, and generate a complete UDP data packet based on the video frame data, and output it to the priority queue management module;

[0011] The data output end of the priority queue management module is connected to the input end of the receiving end through a network interface, and the control end is connected to the output end of the dynamic QoS control module; the priority queue management module is used to store UDP data packets according to priority, and selectively output UDP data packets to the receiving end according to priority sorting based on the QoS control signal;

[0012] The monitoring end of the dynamic QoS control module is used to monitor the network status between the priority queue management module and the receiving end, dynamically adjust the data transmission strategy according to the network status, generate a QoS control signal, and output it to the priority queue management module;

[0013] The output end of the receiving end is connected to an external display device or decoding device. The receiving end is used to identify the priority of the video frame data according to the QoS field of the received UDP data packet, reorganize the video frame data according to the priority, and compensate when the video frame data is lost, generate a complete video frame, and output it to the external display device or decoding device.

[0014] Furthermore, the video stream classification and marking module includes an input data buffer processing unit, a frame parsing and marking state machine, a frame header parsing logic unit, a priority generation logic unit, a priority latch and a data path processing unit;

[0015] The input end of the input data buffer processing unit is connected to the external video encoder, and the output end is respectively connected to the input ends of the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit, so as to receive the video data stream to be optimized for transmission and output the video frame data to the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit;

[0016] The output end of the frame parsing and marking state machine is respectively connected to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit, and is used to obtain a control signal, a latch enable signal and a data transmission control signal in combination with the video frame data according to the set control logic, and output them to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit respectively;

[0017] The frame header parsing logic unit, the priority generation logic unit, the priority latch and the data path processing unit are connected in sequence, and the output end of the data path processing unit is connected to the input end of the UDP data packet encapsulation module;

[0018] The frame header parsing logic unit is used to classify the video frame data into frame categories according to the control signal, including I frames, P frames and B frames, and detect frame boundaries and type fields, and output the frame categories to the priority generation logic unit; the priority generation logic unit obtains the corresponding priority tag to be added according to the frame category, and outputs it to the priority latch; the priority latch is used to obtain the priority tag of the current frame category according to the priority tag to be added, and output it to the data path processing unit; the data path processing unit is used to add the corresponding priority tag to the header or additional field of the current video frame data according to the frame category under the control of the data transmission control signal, mark the I frame as high priority, the P frame as medium priority, and the B frame as low priority, and output the video frame data with the priority tag to the UDP data packet encapsulation module.

[0019] Furthermore, the UDP data packet encapsulation module includes an input buffer shift register, a fragmentation logic unit, a packing state machine, a QoS field adder, a UDP header generator, a UDP checksum calculator, and an output buffer unit;

[0020] The input end of the input buffer shift register is connected to the output end of the data path processing unit, and the output end is respectively connected to the input ends of the slicing logic unit and the QoS field adder, and is used to slice the video frame data with the priority mark according to the UDP maximum transmission unit, obtain the sliced ​​data, retain the priority mark of the video frame data in each UDP data packet, and output it to the input ends of the slicing logic unit and the QoS field adder respectively;

[0021] The output end of the sharding logic unit is connected to the input end of the packaging state machine, and is used to obtain a control signal for packaging start / end according to the sharding data and output it to the packaging state machine;

[0022] The output end of the packaging state machine is connected to the control end of the QoS field adder, the UDP header generator, and the UDP checksum calculator respectively, for controlling the working states of the QoS field adder, the UDP header generator, and the UDP checksum calculator;

[0023] The QoS field adder, UDP header generator, UDP checksum calculator and output buffer unit are connected in sequence; the QoS field adder is used to add a custom QoS field to the header of each fragmented data and output it to the UDP header generator; the UDP header generator is used to add a standard UDP header to the fragmented data with the QoS field added; the UDP checksum calculator is used to generate a complete UDP data packet by performing checksum calculation on the fragmented data after the UDP header is added, and output it to the output buffer unit;

[0024] The output end of the output buffer unit is connected to the data input end of the priority queue management module.

[0025] Furthermore, the priority queue management module includes an input demultiplexer, a priority queue storage unit and a dequeue arbitration logic unit connected in sequence;

[0026] The data input end of the input demultiplexer is connected to the output end of the output buffer unit, which is used to receive complete UDP data packets and divide the UDP data packets according to the priority according to the QoS field in the UDP data packets, including three data transmission channels of high, medium and low;

[0027] The priority queue storage unit includes three storage queues: high, medium and low, which are respectively used to store UDP data packets of corresponding priorities;

[0028] The control end of the dequeue arbitration logic unit is connected to the output end of the dynamic QoS control module, and the output end is connected to the receiving end, and is used to selectively output UDP data packets to the receiving end according to the priority sorting based on the QoS control signal.

[0029] Furthermore, the dynamic QoS control module includes a network monitoring unit, a QoS decision unit and a control signal output unit connected in sequence;

[0030] The network monitoring unit is used to monitor the network status between the priority queue management module and the receiving end, including normal status, high packet loss status and medium congestion status;

[0031] The QoS decision unit is used to dynamically adjust the data transmission strategy according to the network status and generate a QoS control signal; the data transmission strategy includes: transmitting all UDP data packets in a normal state, transmitting only high-priority UDP data packets in a high packet loss state, and transmitting high-priority and medium-priority UDP data packets in a medium congestion state;

[0032] The output end of the control signal output unit is connected to the control end of the dequeue arbitration logic unit, and is used to output the QoS control signal to the dequeue arbitration logic unit.

[0033] Furthermore, the receiving end includes a UDP packet parser, a packet reordering buffer, a frame state management and control logic unit, a packet loss compensation logic and a frame reassembler;

[0034] The input end of the UDP packet parser is connected to the output end of the dequeue arbitration logic unit, and the output end is connected to the packet reordering buffer and the first input end of the frame status management and control logic unit respectively, and is used to obtain packet meta information according to the UDP data packet, and identify the priority of the video frame data according to the QoS field in the UDP data packet, obtain UDP payload data, and output it to the packet reordering buffer and the frame status management and control logic unit respectively;

[0035] The second input end of the packet reordering buffer is connected to the first output end of the frame status management and control logic unit, and the output end is respectively connected to the second input end of the frame status management and control logic unit and the first input end of the frame reassembler, and is used to obtain storage status data and read data in sequence according to UDP payload data and a buffer read address control signal, and output the data to the frame status management and control logic unit and the frame reassembler respectively;

[0036] The output end of the frame state management and control logic unit is connected to the input end of the packet loss compensation logic and the second input end of the frame reassembler respectively, and is used to obtain a buffer read address control signal, a compensation / discarding signal and a reassembly signal according to the packet metadata and the storage state data, and output them to the packet reordering buffer, the packet loss compensation logic and the frame reassembler respectively;

[0037] The output end of the packet loss compensation logic is connected to the third input end of the frame reassembler, and is used to obtain a compensation data signal according to the compensation / discarding signal and output it to the frame reassembler;

[0038] The output end of the frame reassembler is connected to an external display device or decoding device, and is used to generate a reassembled video frame based on the reassembled signal and the sequentially read data, and output it to the external display device or decoding device; or when receiving the compensation data, compensate the sequentially read data with the compensation data, and then reassemble it to generate a reassembled compensated video frame, and output it to the external display device or decoding device.

[0039] At the same time, the present invention also provides a UDP video transmission optimization method based on dynamic QoS, which is based on the aforementioned UDP video transmission optimization system based on dynamic QoS, and is special in that it includes the following steps:

[0040] Step 1: Receive the video data stream to be optimized for transmission through the video stream classification and marking module; classify the video data stream into frame categories, including I frames, P frames, and B frames; mark I frames as high priority, P frames as medium priority, and B frames as low priority; and output the video frame data with priority markings to the UDP data packet encapsulation module;

[0041] Step 2: Add a custom QoS field to the video frame data with the priority mark through the UDP data packet encapsulation module, generate a complete UDP data packet based on the video frame data, and output it to the priority queue management module;

[0042] Step 3: The monitoring end of the dynamic QoS control module monitors the network status between the priority queue management module and the receiving end, dynamically adjusts the data transmission strategy according to the network status, generates a QoS control signal, and outputs it to the priority queue management module;

[0043] Step 4: The priority queue management module stores the UDP data packets according to the priority, and selectively outputs the UDP data packets to the receiving end according to the priority sorting according to the QoS control signal;

[0044] Step 5. The receiving end identifies the priority of the video frame data according to the QoS field of the received UDP data packet, reassembles the video frame data according to the priority, compensates when the video frame data is lost, generates the reassembled and compensated video frame, and outputs it to an external display device or decoding device.

[0045] The beneficial effects of the present invention are:

[0046] The present invention prioritizes video frame data according to its criticality, distinguishes video frame data of different priorities by adding a QoS field, and dynamically adjusts the transmission strategy in real time according to the network status, thereby ensuring the priority transmission of I frames and the reliability of video transmission, thereby avoiding decoding failure or severe image distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of step 1 in an embodiment of a UDP video transmission optimization method based on dynamic QoS of the present invention;

[0048] Figure 2 2 is a flow chart of step 2 in an embodiment of a UDP video transmission optimization method based on dynamic QoS of the present invention;

[0049] Figure 3 4 is a flow chart of step 4 in an embodiment of a UDP video transmission optimization method based on dynamic QoS of the present invention;

[0050] Figure 44 is a flow chart of step 5 in an embodiment of a UDP video transmission optimization method based on dynamic QoS of the present invention;

[0051] Figure 5 This is a structural diagram of an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0052] Figure 6 This is a data flow diagram of a video stream classification and marking module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0053] Figure 7 It is a state transition diagram of a video stream classification and marking module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0054] Figure 8 This is a data flow diagram of a UDP data packet encapsulation module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0055] Figure 9 It is a decision state diagram of a dynamic QoS control module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0056] Figure 10 This is a data flow diagram of a priority queue management module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0057] Figure 11 It is a state transition diagram of a priority queue management module in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention;

[0058] Figure 12 This is a data flow diagram of a receiving end in an embodiment of a UDP video transmission optimization system based on dynamic QoS of the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, advantages, and features of the present invention more apparent, the following describes in further detail a system and method for optimizing UDP video transmission based on dynamic QoS, in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent based on the following specific embodiments.

[0060] This embodiment provides a UDP video transmission optimization method based on dynamic QoS, which specifically includes the following steps:

[0061] Step 1: See Figure 1 ,Video Stream Classification and Marking ,Video stream classification and marking are the basis for achieving differentiated ,services, which aims to assign different priority tags to video frames ,for subsequent processing and transmission.

[0062] The received video stream to be optimized for transmission is classified into three types: I-frame (Intra-frame), P-frame (Predicted frame) and B-frame (Bi-directional predicted frame). I-frame is a key frame that contains complete image information and usually has a large data volume. P-frame is predictively encoded by referring to the previous frame (I-frame or P-frame) and has a medium data volume. B-frame is bi-directionally predicted by referring to the previous and next frames and has a small data volume.

[0063] Assume that the video frame sequence is F={F1,F2,…,F x}, x represents the number of frames, where F i Indicates the (i)th frame, the frame type can be defined as:

[0064] TF i =0, if F i For I frame;

[0065] TF i =1, if F i It is a P frame;

[0066] TF i =2, if F i For B frame.

[0067] Frame type marking

[0068] Assign an identifier to each frame to distinguish its type and priority. I frames are marked as high priority, P frames are marked as medium priority, and B frames are marked as low priority. The identifier can be embedded in the metadata of the video stream or the packet header and is represented by an integer. For example, I frames are marked as 0, P frames are marked as 1, and B frames are marked as 2. This marking can be obtained by the function M(F i )=TF i , realize, where M(F i ) represents frame F i The tag value of .

[0069] Step 2, see Figure 2 ,UDP packet encapsulation,UDP packet encapsulation is designed to encapsulate video data stream packets into UDP format and add priority information to support QoS control.

[0070] Encapsulate the video frame data with priority tags into UDP packets.

[0071] The encoded video frame data is divided into data packets suitable for network transmission. Let frame F i is divided into (k) packets, denoted as P i,j(j=1,2,…,k), each data packet carries a frame type tag M(F i The packet size must meet the network maximum transmission unit (MTU) requirements, usually less than 1500 bytes. The packetization process can be expressed as: F i →{P i,1 ,P i,2 ,…,P i,k}, where P i,k .tag=M(F i ).

[0072] Add a custom QoS field to the payload header of a UDP data packet to generate a UDP data packet with the added field.

[0073] Custom QoS field design

[0074] A 1-byte QoS field is added to the payload header of the UDP packet to identify the priority of the packet. Let the QoS field value be (Q), which is defined as follows:

[0075] Q=0, indicating I-frame data packets, which have the highest priority;

[0076] Q=1, indicating P frame data packet, medium priority;

[0077] Q=2, indicating B-frame data packets, which have the lowest priority.

[0078] This field is 8 bits long and can support 2 8 = 256 priority levels, only states 0-2 are used here.

[0079] UDP packet structure

[0080] UDP data packet consists of a standard header and a custom payload, and its structure can be expressed as: P = {H UDP ,Q,D}, where H UDP (Q) is the UDP header (8 bytes, including source port, destination port, length and checksum), (D) is the video data payload.

[0081] Packaging process

[0082] For the video stream data packet P i,j , mark the frame type M(F i ) is assigned to the QoS field, i.e. Q = M(F i ), and then adds the UDP header to complete the encapsulation.

[0083] Step 3: Dynamic QoS control. The dynamic QoS control mechanism monitors the network status in real time and dynamically adjusts the data packet transmission strategy to optimize the video stream transmission quality.

[0084] Network status monitoring

[0085] Regularly collect network parameters, including packet loss rate (PLR), delay (Latency) and jitter (Jitter). Assume that the monitoring period is (T) (unit: seconds), and the packet loss rate calculation formula for the (n)th period is: Among them, L n is the number of packets lost in a period, S n is the total number of packets sent. Delay and jitter can be calculated from the timestamp difference.

[0086] Methods for calculating network parameters

[0087] Packet Loss Rate (PLR)

[0088] The receiving end records the sequence number S marked by the sending end i (Each data packet carries a continuously increasing sequence number), count the number of data packets received N received and the expected number of receivers N expected (Based on the sequence number range). The packet loss rate is: In the case of packet loss, N received <N expected .

[0089] Round Trip Time (RTT)

[0090] The sender embeds time T in the data packet send , the receiving end records the arrival time T arrive , and returns T in the feedback package send or T arrive The RTT calculation depends on the arrival time T of the feedback packet. received :RTT=T received -T send .

[0091] If the feedback packet is lost, the RTT cannot be calculated and needs to be processed through a subsequent mechanism (retransmission request after timeout or use the last data for processing).

[0092] Jitter

[0093] The receiving end records the arrival times T1, T2, ..., T of consecutive data packets. y , calculate the interval D i =T i+1 -T i , and the jitter is: Packet loss will cause interval D i Discontinuous, need to skip the interval of missing packets. D is the standard interval.

[0094] QoS Decision

[0095] Formulate a transmission strategy based on network status parameters. Assume that the packet loss rate thresholds are θ1 and θ2 (for example, θ1 = 5%, θ2 = 20%). The decision rule is as follows:

[0096] If PLR n <θ1, transmit all frame type packets;

[0097] If θ1≤PLR n ≤θ2, only I-frame and P-frame data packets are transmitted;

[0098] If PLR n >θ2, only I frame data packets are transmitted.

[0099] The decision can be expressed as a function S(PLR n ), outputs the set of frame types allowed to be transmitted.

[0100] Strategy Execution

[0101] According to the decision result, adjust the sending end behavior. Let the sending queue be Q Send , only packets that meet the policy are added to the queue: Q Send ={P i,j |T(F i ∈S(PLR n ))}.

[0102] Step 4, see Figure 3 ,Priority queue management, through multi-queue design and scheduling strategy, ensures that high-priority data packets are transmitted first.

[0103] Cohort Design

[0104] Set up three priority queues:

[0105] High priority queue Q H : Store I frame data packet (Q=0);

[0106] Medium priority queue Q M : Store P frame data packets (Q=1);

[0107] Low priority queue Q L : Store B frame data packets (Q=2).

[0108] The capacity of each queue can be dynamically adjusted according to system resources.

[0109] Entry Rules

[0110] According to the QoS field (Q) of the data packet, it is assigned to the corresponding queue:

[0111]

[0112] Departure strategy

[0113] Prioritize Q H Out of the team, if Q H If it is empty, then from Q M Out of the team, finally from Q L Leave the team.

[0114] Queue Management

[0115] Regularly check the timeliness of the data packets in the queue and remove the timed-out data packets. The timeout threshold is set to T exp .

[0116] How to set the timeout threshold

[0117] Timeout threshold (T exp ) is set to ensure that data packets are not discarded due to expiration in the queue, thereby ensuring the real-time performance of the video stream. The setting method takes into account the characteristics of the video stream, network latency, and queue processing time. The specific steps are as follows:

[0118] Video streaming features

[0119] The frame rate (FPS) of the video stream is the core parameter. Assuming the frame rate is (f) (for example, 30FPS), the frame interval time is The timeout threshold must be at least greater than the interframe interval to prevent frames from expiring before they are fully transmitted.

[0120] Network latency

[0121] The average delay of network transmission d avg and dither d jitter It will affect the arrival time of the data packet. The maximum tolerable delay can be defined as d max =d avg +h·d jitter , where (h) is a safety factor (usually 2 or 3) to cover delay fluctuations.

[0122] Queue processing time

[0123] The processing and sending time of the data packets in the queue must also be included in the calculation. Assume that the average processing time of sending a data packet is t proc , there are (n) packets in the queue, then the maximum processing delay is n·t proc .

[0124] Timeout threshold calculation formula

[0125] Taking the above factors into consideration, the timeout threshold T exp Can be set to: T exp =Δt+d max+n·t proc

[0126] Example: For a 30FPS video stream, Δt = 33.3ms; if d avg =50ms,d jitter =10ms, k=2, then d max =50+2×10=70ms; if t proc =1ms, n=10, then: T exp =33.3+70+10×1=113.3ms.

[0127] Dynamic Adjustment

[0128] In practical applications, T can be dynamically adjusted according to real-time network conditions (such as increased delay) or queue load. exp , to balance real-time and data integrity. For example, when network congestion is detected, T exp To reduce packet loss.

[0129] Receiver processing: The receiver identifies the packet priority based on the QoS field, preferentially decodes I frames, and compensates for lost P frames or B frames using interpolation or frame replication.

[0130] Step 5, see Figure 4 ,The receiving end processing is responsible for parsing data packets, decoding video frames and handling packet loss.

[0131] Packet parsing

[0132] After receiving the UDP data packet, extract the QoS field (Q) and video data payload (D).

[0133] Decoding process

[0134] Decode according to frame type:

[0135] If Q=0 (I frame), directly decode and output the image;

[0136] If Q = 1 (P frame) or Q = 2 (B frame), check whether the dependent frame is received successfully. If the dependent frame exists, decode it; otherwise, discard the packet. The dependency relationship can be determined by the frame sequence number or timestamp.

[0137] In a video stream, P frames and B frames depend on other frames (I frames or P frames). The receiver needs to determine whether the dependent frames have been received. The dependency relationship can be determined by the frame sequence number or timestamp. The specific method is as follows:

[0138] Judging by frame sequence number

[0139] Frame sequence number definition

[0140] Each video frame is assigned a unique sequence number S(F i ), usually an increasing integer. I-frames do not depend on other frames, P-frames depend on the previous I-frame or P-frame, and B-frames depend on the previous and next I-frames or P-frames.

[0141] Dependencies

[0142] If F i It is a P frame, which depends on the F frame. d =F i-1 (Assuming only dependencies on the previous frame).

[0143] If F i It is a B frame, dependent on frame F d ={F i-1 ,F i+1}(assuming it depends on the previous and next frames).

[0144] Judgment Process

[0145] For the current frame F i :

[0146] If F i It is an I-frame and does not require any dependencies.

[0147] If F i For P frame, check if T(F) exists in (B) d ) <T(F i ).

[0148] If F i For B frame, check if F exists in (B) d1 and F d2 And T(F d1 ) <T(F i ) <T(F d2 ).

[0149] Processing Rules

[0150] If all dependent frames exist, then F i Decodable; otherwise, discard F i Data packets are waiting for retransmission or compensation.

[0151] Compensation Mechanism

[0152] When a P frame or B frame is lost, compensation techniques (such as frame duplication or interpolation) are used. Let the current frame be F i , the previous successfully decoded frame is F i-1 , the compensation frame can be approximated as: F i ′=F i-1 (Frame Copy).

[0153] See also Figure 5In order to implement the above optimization method, the system can be divided into a sending end and a receiving end; among them, the sending end can be divided into the following four modules: video stream classification and marking module; UDP data packet encapsulation module; dynamic QoS control module; priority queue management module.

[0154] These modules will achieve efficient video transmission optimization through hardware pipeline design and parallel processing. The following is the detailed design of each module:

[0155] Video stream classification and marking module

[0156] Function

[0157] 1. Classify the frame type of the input video data stream.

[0158] 2. Add a priority tag for each type of video frame data.

[0159] Implementation steps

[0160] 1. Input interface design

[0161] Receive compressed video stream data through an external video encoder (such as H.264 / H.265).

[0162] Data format: input frame by frame, with frame header information (such as frame type identification).

[0163] 2. Frame type analysis

[0164] Design a frame header parser to extract frame type information from the video stream.

[0165] Use a state machine to detect frame boundaries and type fields.

[0166] 3. Priority Marking

[0167] Add a 1-byte priority tag to each frame:

[0168] I-frame: marked as 0 (high priority).

[0169] P frame: marked as 1 (medium priority).

[0170] B frame: marked as 2 (low priority).

[0171] Write the marker to the header or additional field of the frame data.

[0172] 4. Output

[0173] Output video frame data with priority mark and pass it to the UDP packet encapsulation module.

[0174] Data Flow

[0175] Input: original video stream (with frame type information).

[0176] Output: Marked video frame data (frame data + priority tag).

[0177] For details, see Figure 6 and Figure 7 The video stream classification and marking module includes an input data buffer processing unit, a frame parsing and marking state machine, a frame header parsing logic unit, a priority generation logic unit, a priority latch and a data path processing unit; the input end of the input data buffer processing unit is connected to the external video encoder, and the output end is respectively connected to the input end of the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit, for receiving the video stream to be optimized for transmission, and outputting the video frame data to the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit; the output end of the frame parsing and marking state machine is respectively connected to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit, for obtaining a control signal, a latch enable signal and a data transmission control signal in combination with the video frame data according to the set control logic, and outputting them to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit respectively; the frame header parsing logic unit, the priority generation logic unit The editing unit, the priority latch and the data path processing unit are connected in sequence, and the output end of the data path processing unit is connected to the input end of the UDP data packet encapsulation module; the frame header parsing logic unit is used to classify the video frame data into frame categories according to the control signal, including I frames, P frames and B frames, and detect the frame boundaries and type fields, and output the frame categories to the priority generation logic unit; the priority generation logic unit obtains the corresponding priority tag to be added according to the frame category and outputs it to the priority latch; the priority latch is used to obtain the priority tag of the current frame category according to the priority tag to be added, and output it to the data path processing unit; the data path processing unit is used to add the corresponding priority tag to the header or additional field of the current video frame data according to the frame category under the control of the data transmission control signal, mark the I frame as high priority, the P frame as medium priority, and the B frame as low priority, and output the video frame data with the priority tag to the UDP data packet encapsulation module.

[0178] UDP packet encapsulation module

[0179] Function

[0180] The marked video frame data is encapsulated into a UDP data packet and a QoS field is added to the payload header.

[0181] Implementation steps

[0182] 1. Data Sharding

[0183] The video frame data is fragmented according to the UDP maximum transmission unit.

[0184] Each fragment retains the frame's priority tag.

[0185] 2. Add QoS field

[0186] Add a 1-byte QoS field to the payload header of each UDP packet:

[0187] 0: I-frame data packet.

[0188] 1: P-frame data packet.

[0189] 2: B-frame data packet.

[0190] 3. UDP header encapsulation

[0191] Add the standard UDP header (source port, destination port, length, checksum).

[0192] Use hardware parallelism to calculate checksums and speed up the packaging process.

[0193] 4. Output

[0194] Generate a complete UDP data packet and pass it to the priority queue management module.

[0195] Data Flow

[0196] Input: Labeled video frame data.

[0197] Output: Encapsulated UDP data packet (UDP header + QoS field + fragment data).

[0198] For details, see Figure 8, the UDP data packet encapsulation module input buffer shift register, fragmentation logic unit, packaging state machine, QoS field adder, UDP header generator, UDP checksum calculator and output buffer unit; the input end of the input buffer shift register is connected to the output end of the data path processing unit, and the output end is respectively connected to the input end of the fragmentation logic unit and the QoS field adder, which is used to fragment the video frame data with priority marks according to the UDP maximum transmission unit, obtain fragmented data, and retain the priority mark of the video frame data in each UDP data packet, and output them to the input end of the fragmentation logic unit and the QoS field adder respectively; the output end of the fragmentation logic unit is connected to the input end of the packaging state machine, which is used to obtain the control signal of packaging start / end according to the fragmented data and output it to the packaging state machine; the input of the packaging state machine The output end is respectively connected to the control end of the QoS field adder, UDP header generator, and UDP checksum calculator, which is used to control the start / end of the QoS field adder, UDP header generator, and UDP checksum calculator; the QoS field adder, UDP header generator, UDP checksum calculator and output buffer unit are connected in sequence; the QoS field adder is used to add a custom QoS field to the header of each fragmented data and output it to the UDP header generator; the UDP header generator is used to add a standard UDP header to the fragmented data with the QoS field added; the UDP checksum calculator is used to calculate and check the fragmented data after the UDP header is added, generate a complete UDP data packet, and output it to the output buffer unit; the output end of the output buffer unit is connected to the data input end of the priority queue management module.

[0199] Dynamic QoS control module

[0200] Function

[0201] Monitor network status in real time and adjust transmission strategies dynamically.

[0202] Implementation steps

[0203] 1. Network monitoring submodule

[0204] Input: Obtain the network parameters calculated by the receiving end through the external network interface.

[0205] Parameter collection:

[0206] Packet loss rate: Statistics on the ratio of sent packets to received confirmation packets.

[0207] Latency: Calculate the round-trip time (RTT) using timestamps.

[0208] Jitter: Calculates the variance of the time intervals between consecutive data packets.

[0209] Implementation: Use registers and counters to implement parameter statistics, with a configurable update frequency (such as every second).

[0210] 2. QoS decision submodule

[0211] Threshold configuration: preset thresholds for packet loss rate, delay, and jitter (e.g., packet loss rate > 5%).

[0212] Strategy Adjustment:

[0213] Normal state: All frames (I, P, B) are transmitted.

[0214] High packet loss state: Only I frames are transmitted, and P frames and B frames are discarded.

[0215] Moderate congestion: I frames and P frames are transmitted, and B frames are discarded.

[0216] Implementation: Use comparators and logic circuits to output control signals based on monitoring data.

[0217] 3. Control signal output

[0218] The decision result (such as "only transmit I frame") is sent to the priority queue management module.

[0219] Data Flow

[0220] Input: Network status feedback data.

[0221] Output: QoS control signal (indicates which priority packets are transmitted).

[0222] For details, see Figure 9 The dynamic QoS control module includes a network monitoring unit, a QoS decision unit and a control signal output unit connected in sequence; the network monitoring unit is used to monitor the network status between the priority queue management module and the receiving end, including normal status, high packet loss status and medium congestion status; the QoS decision unit is used to dynamically adjust the data transmission strategy according to the network status and generate a QoS control signal; the data transmission strategy includes: transmitting all UDP data packets in the normal status, transmitting only high-priority UDP data packets in the high packet loss status, and transmitting high-priority and medium-priority UDP data packets in the medium congestion status; the output end of the control signal output unit is connected to the control end of the dequeue arbitration logic unit, and is used to output the QoS control signal to the dequeue arbitration logic unit.

[0223] Priority queue management module

[0224] Function

[0225] Store and manage UDP packets according to priority, and transmit high-priority data first.

[0226] Implementation steps

[0227] 1. Queue design

[0228] Create three FIFO queues:

[0229] High priority queue (I frame, QoS=0).

[0230] Medium priority queue (P frames, QoS=1).

[0231] Low priority queue (B frames, QoS=2).

[0232] Use FPGA internal BRAM to implement queue storage.

[0233] 2. Enqueue logic

[0234] According to the QoS field in the UDP data packet, the data packet is written to the corresponding queue.

[0235] Use a multiplexer (MUX) to implement data splitting.

[0236] 3. Dequeue logic

[0237] Priority Scheduling:

[0238] If the high priority queue is not empty, I frames are transmitted first.

[0239] If the high priority queue is empty, the medium priority queue (P frame) is transmitted.

[0240] The low priority queue (B frames) are transmitted last.

[0241] Based on the signal from the dynamic QoS control module, low priority data packets (such as B frames) are discarded.

[0242] 4. Output

[0243] Send the data packet to the network interface (such as Ethernet PHY).

[0244] Data Flow

[0245] Input: Encapsulated UDP data packet + QoS control signal.

[0246] Output: Prioritized packet stream.

[0247] See also Figure 10 and Figure 11The priority queue management module includes an input demultiplexer, a priority queue storage unit and a dequeue arbitration logic unit connected in sequence; the data input end of the input demultiplexer is connected to the output end of the output buffer unit, which is used to receive complete UDP data packets and divert the UDP data packets according to the priority according to the QoS field in the UDP data packets, including three data transmission channels of high, medium and low; the priority queue storage unit includes three storage queues of high, medium and low, which are respectively used to store UDP data packets of corresponding priorities; the control end of the dequeue arbitration logic unit is connected to the output end of the dynamic QoS control module, and the output end is connected to the receiving end through the AXI-Stream network, which is used to selectively output UDP data packets to the receiving end according to the priority sorting based on the QoS control signal.

[0248] AXI-Stream is a high-speed data stream transmission protocol widely used in FPGA and SoC designs. It defines a set of signals to transmit continuous data streams between source (Master) and target (Slave) modules. The main signals include:

[0249] TDATA: transmits the actual data payload.

[0250] TVALID: The source module uses it to indicate that the data on TDATA (and other accompanying signals such as TLAST, TUSER, etc.) is valid.

[0251] TREADY: The target module uses this to indicate that it is ready to receive data. Data will be transmitted on the clock edge only when both TVALID and TREADY are high.

[0252] TLAST: The source module uses it to indicate that the currently transmitted data is the last data unit of a packet or frame.

[0253] TUSER: Provides an additional, user-defined channel for transmitting additional information (metadata or control information) along with the main data (TDATA).

[0254] Receiver

[0255] Function

[0256] Identify packet priorities based on the QoS field, prioritize decoding of I-frames, and compensate for lost frames.

[0257] Calculate parameters such as packet loss rate, delay, network jitter, and feed them back to the sender.

[0258] Implementation steps

[0259] 1. Data packet analysis

[0260] Extract the QoS field from the received UDP packet.

[0261] Use a state machine to identify packet boundaries and priorities.

[0262] Record the sequence number marked by the sender and count the number of received data packets to calculate the packet loss rate.

[0263] Parse the timestamp embedded in the data packet by the sender and calculate the round-trip delay and jitter.

[0264] 2. Frame Reassembly

[0265] Reassemble packets into video frames based on QoS fields:

[0266] QoS=0: I frame.

[0267] QoS=1: P frame.

[0268] QoS=2: B frame.

[0269] Use a buffer to temporarily store reassembled data.

[0270] 3. Frame loss compensation

[0271] If P-frame or B-frame loss is detected:

[0272] Interpolation: Linear interpolation based on the previous and next frame data.

[0273] Frame copy: Copy the previous frame data to fill the missing part.

[0274] Implementation: Use hardware multipliers and adders to implement interpolation calculations.

[0275] 4. Output

[0276] The reassembled and compensated video frame data is passed to the decoder or display module.

[0277] Regularly generate feedback data packets containing network parameters such as packet loss rate, delay, and network jitter and send them back to the sender.

[0278] Data Flow

[0279] Input: Received UDP data packet.

[0280] Output: reassembled and compensated video frame data.

[0281] See also Figure 12The receiving end includes a UDP packet parser, a packet reordering buffer, a frame status management and control logic unit, a packet loss compensation logic and a frame reassembler; the input end of the UDP packet parser is connected to the output end of the dequeue arbitration logic unit, and the output end is respectively connected to the packet reordering buffer and the first input end of the frame status management and control logic unit, for obtaining packet metadata according to the UDP data packet, and identifying the priority of the video frame data according to the QoS field in the UDP data packet, obtaining UDP payload data, and outputting them to the packet reordering buffer and the frame status management and control logic unit respectively; the second input end of the packet reordering buffer is connected to the first output end of the frame status management and control logic unit, and the output end is respectively connected to the second input end of the frame status management and control logic unit and the first input end of the frame reassembler, for obtaining storage status data and sequentially reading data according to the UDP payload data and the buffer read address control signal, and outputting them to the frame status The output end of the frame state management and control logic unit is connected to the input end of the packet loss compensation logic and the second input end of the frame reassembler respectively, and is used to obtain a buffer read address control signal, a compensation / discarding signal and a reassembly signal according to the packet meta information and the storage state data, and output them to the packet reordering buffer, the packet loss compensation logic and the frame reassembler respectively; the output end of the packet loss compensation logic is connected to the third input end of the frame reassembler, and is used to obtain a compensation data signal according to the compensation / discarding signal and output it to the frame reassembler; the output end of the frame reassembler is connected to an external display device or decoding device, and is used to generate a reassembled video frame according to the reassembly signal and the sequentially read data, and output it to the external display device or decoding device; or when receiving the compensation data, the sequentially read data is compensated by the compensation data, and then reassembled to generate a reassembled compensated video frame, and output it to the external display device or decoding device.

Claims

1. A UDP video transmission optimization system based on dynamic QoS, characterized by: Including the sending end and the receiving end; The transmitting end includes a video stream classification and marking module, a UDP data packet encapsulation module, a dynamic QoS control module and a priority queue management module; The input end of the video stream classification and marking module is connected to the external video encoder, and the output end is connected to the input end of the UDP data packet encapsulation module; the video stream classification and marking module is used to receive the video data stream to be optimized for transmission; Classify the video data stream into frame categories, including I-frames, P-frames and B-frames; Mark I frames as high priority, P frames as medium priority, and B frames as low priority; Output the video frame data with priority mark to the UDP data packet encapsulation module; The output end of the UDP data packet encapsulation module is connected to the data input end of the priority queue management module. The UDP data packet encapsulation module is used to add a custom QoS field to the video frame data with the priority mark, and generate a complete UDP data packet based on the video frame data, and output it to the priority queue management module; The data output end of the priority queue management module is connected to the input end of the receiving end through a network interface, and the control end is connected to the output end of the dynamic QoS control module; the priority queue management module is used to store UDP data packets according to priority, and selectively output UDP data packets to the receiving end according to priority sorting based on the QoS control signal output by the QoS control module; The monitoring end of the dynamic QoS control module is used to monitor the network status between the priority queue management module and the receiving end, dynamically adjust the data transmission strategy according to the network status, generate a QoS control signal, and output it to the priority queue management module; The output end of the receiving end is connected to an external display device or decoding device. The receiving end is used to identify the priority of the video frame data according to the QoS field of the received UDP data packet, reorganize the video frame data according to the priority, and compensate when the video frame data is lost, generate a complete video frame, and output it to the external display device or decoding device.

2. The UDP video transmission optimization system based on dynamic QoS according to claim 1, characterized in that: The video stream classification and marking module includes an input data buffer processing unit, a frame parsing and marking state machine, a frame header parsing logic unit, a priority generation logic unit, a priority latch and a data path processing unit; The input end of the input data buffer processing unit is connected to the external video encoder, and the output end is respectively connected to the input ends of the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit, so as to receive the video data stream to be optimized for transmission and output the video frame data to the frame parsing and marking state machine, the frame header parsing logic unit and the data path processing unit; The output end of the frame parsing and marking state machine is respectively connected to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit, and is used to obtain a control signal, a latch enable signal and a data transmission control signal in combination with the video frame data according to the set control logic, and output them to the control end of the frame header parsing logic unit, the priority latch and the data path processing unit respectively; The frame header parsing logic unit, the priority generation logic unit, the priority latch and the data path processing unit are connected in sequence, and the output end of the data path processing unit is connected to the input end of the UDP data packet encapsulation module; The frame header parsing logic unit is used to classify the video frame data into frame categories, including I-frames, P-frames and B-frames, according to the control signal, and detect the frame boundary and type field, and output the frame category to the priority generation logic unit; The priority generation logic unit obtains the corresponding priority tag to be added according to the frame category and outputs it to the priority latch; The priority latch is used to obtain the priority tag of the current frame category based on the priority tag to be added, and output it to the data path processing unit; the data path processing unit is used to add the corresponding priority tag to the header or additional field of the current video frame data according to the frame category under the control of the data transmission control signal, mark the I frame as high priority, the P frame as medium priority, and the B frame as low priority, and output the video frame data with the priority tag to the UDP data packet encapsulation module.

3. The UDP video transmission optimization system based on dynamic QoS according to claim 2, characterized in that: The UDP data packet encapsulation module includes an input buffer shift register, a fragmentation logic unit, a packing state machine, a QoS field adder, a UDP header generator, a UDP checksum calculator, and an output buffer unit; The input end of the input buffer shift register is connected to the output end of the data path processing unit, and the output end is respectively connected to the input ends of the slicing logic unit and the QoS field adder, and is used to slice the video frame data with the priority mark according to the UDP maximum transmission unit, obtain the sliced ​​data, retain the priority mark of the video frame data in each UDP data packet, and output it to the input ends of the slicing logic unit and the QoS field adder respectively; The output end of the sharding logic unit is connected to the input end of the packaging state machine, and is used to obtain a control signal for packaging start / end according to the sharding data and output it to the packaging state machine; The output end of the packaging state machine is connected to the control end of the QoS field adder, the UDP header generator, and the UDP checksum calculator respectively, for controlling the working states of the QoS field adder, the UDP header generator, and the UDP checksum calculator; The QoS field adder, UDP header generator, UDP checksum calculator and output buffer unit are connected in sequence; the QoS field adder is used to add a custom QoS field to the header of each fragment data and output it to the UDP header generator; The UDP header generator is used to add a standard UDP header to the fragmented data with the QoS field added; the UDP checksum calculator is used to calculate the fragmented data after the UDP header is added by checksum, generate a complete UDP data packet, and output it to the output buffer unit; The output end of the output buffer unit is connected to the data input end of the priority queue management module.

4. The UDP video transmission optimization system based on dynamic QoS according to claim 3, characterized in that: The priority queue management module includes an input demultiplexer, a priority queue storage unit and a dequeue arbitration logic unit connected in sequence; The data input end of the input demultiplexer is connected to the output end of the output buffer unit, which is used to receive complete UDP data packets and divide the UDP data packets according to the priority according to the QoS field in the UDP data packets, including three data transmission channels of high, medium and low; The priority queue storage unit includes three storage queues: high, medium and low, which are respectively used to store UDP data packets of corresponding priorities; The control end of the dequeue arbitration logic unit is connected to the output end of the dynamic QoS control module, and the output end is connected to the receiving end, and is used to selectively output UDP data packets to the receiving end according to the priority sorting based on the QoS control signal.

5. The UDP video transmission optimization system based on dynamic QoS according to claim 4, characterized in that: The dynamic QoS control module includes a network monitoring unit, a QoS decision unit and a control signal output unit connected in sequence; The network monitoring unit is used to monitor the network status between the priority queue management module and the receiving end, including normal status, high packet loss status and medium congestion status; The QoS decision unit is used to dynamically adjust the data transmission strategy according to the network status and generate a QoS control signal; The data transmission strategy includes: transmitting all UDP data packets in a normal state, transmitting only high-priority UDP data packets in a high packet loss state, and transmitting high-priority and medium-priority UDP data packets in a medium congestion state; The output end of the control signal output unit is connected to the control end of the dequeue arbitration logic unit, and is used to output the QoS control signal to the dequeue arbitration logic unit.

6. The UDP video transmission optimization system based on dynamic QoS according to claim 5, characterized in that: The receiving end includes a UDP packet parser, a packet reordering buffer, a frame status management and control logic unit, a packet loss compensation logic and a frame reassembler; The input end of the UDP packet parser is connected to the output end of the dequeue arbitration logic unit, and the output end is connected to the packet reordering buffer and the first input end of the frame status management and control logic unit respectively, and is used to obtain packet meta information according to the UDP data packet, and identify the priority of the video frame data according to the QoS field in the UDP data packet, obtain UDP payload data, and output it to the packet reordering buffer and the frame status management and control logic unit respectively; The second input end of the packet reordering buffer is connected to the first output end of the frame status management and control logic unit, and the output end is respectively connected to the second input end of the frame status management and control logic unit and the first input end of the frame reassembler, and is used to obtain storage status data and read data in sequence according to UDP payload data and the buffer read address control signal output by the frame status management and control logic unit, and output them to the frame status management and control logic unit and the frame reassembler respectively; The output end of the frame state management and control logic unit is connected to the input end of the packet loss compensation logic and the second input end of the frame reassembler respectively, and is used to obtain a buffer read address control signal, a compensation / discarding signal and a reassembly signal according to the packet metadata and the storage state data, and output them to the packet reordering buffer, the packet loss compensation logic and the frame reassembler respectively; The output end of the packet loss compensation logic is connected to the third input end of the frame reassembler, and is used to obtain a compensation data signal according to the compensation / discarding signal and output it to the frame reassembler; The output end of the frame reassembler is connected to an external display device or decoding device, and is used to generate a reassembled video frame according to the reassembled signal and sequentially read data, and output it to the external display device or decoding device; Alternatively, when the compensation data is received, the sequentially read data is compensated by the compensation data, and then reorganized to generate a reorganized and compensated video frame, which is output to an external display device or decoding device.

7. A UDP video transmission optimization method based on dynamic QoS, based on a UDP video transmission optimization system based on dynamic QoS according to any one of claims 1 to 6, characterized in that: The following steps are involved: Step 1: Receive the video data stream to be optimized for transmission through the video stream classification and marking module; Classify the video data stream into frame categories, including I-frames, P-frames and B-frames; Mark I frames as high priority, P frames as medium priority, and B frames as low priority; Output the video frame data with priority mark to the UDP data packet encapsulation module; Step 2: Add a custom QoS field to the video frame data with the priority mark through the UDP data packet encapsulation module, generate a complete UDP data packet based on the video frame data, and output it to the priority queue management module; Step 3: The monitoring end of the dynamic QoS control module monitors the network status between the priority queue management module and the receiving end, dynamically adjusts the data transmission strategy according to the network status, generates a QoS control signal, and outputs it to the priority queue management module; Step 4: The priority queue management module stores the UDP data packets according to the priority, and selectively outputs the UDP data packets to the receiving end according to the priority sorting according to the QoS control signal; Step 5. The receiving end identifies the priority of the video frame data according to the QoS field of the received UDP data packet, reassembles the video frame data according to the priority, compensates when the video frame data is lost, generates the reassembled and compensated video frame, and outputs it to an external display device or decoding device.

Images