A method and system for reliable transmission of real-time data

Abstract

The application belongs to the technical field of video transmission, and discloses a real-time data reliable transmission method and system, which comprises the following steps: a receiving end feeds back dynamic coding frame discarding based on network packet loss, adjusts a first-level or second-level coding mechanism according to a congestion level, discards part of P frames, re-encodes and sends after re-establishing the reference chain of the reserved frames, and when there is still packet loss after coding adjustment, splits the video code stream into multiple sub-streams, uses ECMP multi-path for load sharing, continuously monitors the packet loss rate of each sub-stream, dynamically migrates the sub-stream with the highest packet loss rate to a non-packet loss path, continues to try the next path if there is still packet loss after migration, and is fixed on the path with the lowest packet loss rate if there is still packet loss after trying all paths. The three-level linkage mechanism of the coding layer and the transmission layer effectively deals with network congestion without modifying the network infrastructure, and significantly improves the fluency and reliability of real-time video transmission.

Description

Technical Field

[0001] This invention belongs to the field of video transmission technology, and in particular relates to a method and system for reliable real-time data transmission. Background Technology

[0002] With the rapid development of the Internet and multimedia communication technologies, real-time video transmission has been widely used in scenarios such as video conferencing, online education, telemedicine, and security monitoring. However, existing network infrastructure struggles to guarantee the Quality of Service (QoS) of real-time video transmission, mainly due to the following issues:

[0003] First, network congestion leads to packet loss. When network traffic exceeds the link's capacity, routers actively drop data packets, resulting in incomplete video data and screen tearing or stuttering at the receiving end. Traditional solutions mainly rely on packet retransmission or forward error correction (FEC) mechanisms, but these methods increase additional transmission overhead, and retransmission itself can further exacerbate congestion when it is severe. Simple bitrate adaptation strategies (such as reducing resolution and frame rate) are often coarse-grained and have a slow response, making it difficult to meet the smoothness requirements of real-time video transmission.

[0004] Second, the ECMP single-path bottleneck. In Equal Cost Multipath (ECMP) scenarios, although the load-sharing mechanism based on 5-tuple hashes can ensure that packets of the same stream (with identical 5-tuples) travel along the same path, when a video stream uses only a single 5-tuple, the entire stream is fixed on a particular path, making it impossible to utilize the bandwidth of other idle links. Once this path becomes congested, the entire video stream is affected, while the bandwidth resources of other idle paths cannot be utilized.

[0005] Third, existing transmission strategies are rigid. Traditional congestion control strategies typically passively reduce the transmission rate after detecting packet loss, lacking proactive, multi-layered response mechanisms. For real-time video transmission, how to effectively cope with network congestion without relying on retransmissions, while making full use of multipath resources, is a pressing technical problem that needs to be solved.

[0006] To address the aforementioned issues, some studies have attempted to improve the reliability of video transmission through multipath transmission. For example, the video stream is split into multiple sub-streams and transmitted via different paths to distribute traffic and avoid single-point congestion. However, existing solutions still have shortcomings in sub-stream partitioning, path scheduling, and congestion handling, and lack a systematic linkage mechanism from the coding layer to the transport layer. Summary of the Invention

[0007] This invention provides a method and system for reliable real-time data transmission based on network congestion feedback and multi-path collaboration, aiming to solve technical problems such as packet loss, stuttering, and image quality degradation caused by network congestion in existing real-time video transmission. This invention achieves dynamic assurance of transmission quality through a three-level linkage mechanism, and is compatible with standard RTP / RTCP protocols and ECMP routers without modifying the network infrastructure, significantly improving the smoothness and reliability of real-time video transmission.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for reliable real-time data transmission includes the following steps:

[0010] Step 1: The receiving end feeds back dynamic encoded frame discarding based on network packet loss: The receiving end detects the packet loss rate of the video stream and determines the congestion level based on the packet loss rate, and feeds back a congestion notification to the sending end; The sending end selects to perform either the first-level encoding mechanism adjustment or the second-level encoding mechanism adjustment according to the congestion level, discards some P frames and reconstructs the reference chain of the retained frames before re-encoding and sending.

[0011] Step 2, the video stream is split into multiple sub-streams and load balancing is performed using ECMP multipath: if network packet loss still occurs after the encoding mechanism in Step 1 is adjusted, the receiving end sends a split request to the sending end; in response to the split request, the sending end divides the same video stream into multiple sub-streams and assigns a different 5-tuple to each sub-stream, and uses the ECMP hash mechanism to forward each sub-stream along different physical paths.

[0012] Step 3, Adaptive load sharing link dynamic adjustment: The receiving end continuously monitors the packet loss rate of each sub-stream and migrates the sub-stream with the highest packet loss rate to the path corresponding to the five-tuple used by the sub-stream with no packet loss. If packet loss still occurs after migration, the next path with no packet loss is tried. If packet loss still occurs after trying all paths with no packet loss, the path with the lowest packet loss rate is fixed.

[0013] Furthermore, in step 1:

[0014] The first-level coding mechanism is adjusted as follows: the P-frame sequence is grouped into groups of two from front to back. Each group discards the P-frame with the smaller number and retains the P-frame with the largest number as the reference point of the group. The retained frames are arranged in ascending order of their original numbers and the single-chain prediction structure is reconstructed. The first retained frame directly references the I-frame, and each subsequent retained frame references the previous retained frame.

[0015] The second-level coding mechanism is adjusted as follows: the P-frame sequence is grouped into groups of 3 from front to back. The first two P-frames of each group are discarded, and the P-frame with the largest number is retained as the reference point of the group. The retained frames are arranged in ascending order of their original numbers and the single-chain prediction structure is reconstructed. The first retained frame directly references the I-frame, and each subsequent retained frame references the previous retained frame.

[0016] Furthermore, step 1 also includes:

[0017] After adjusting the encoding mechanism, the receiving end continues to monitor packet loss: if the packet loss rate is still greater than 0, it skips the recovery and decoding of discarded frames and proceeds directly to step 2; if the packet loss rate is equal to 0, the receiving end performs the recovery and decoding of discarded frames.

[0018] The receiving end parses the encoded information carried in the strip header or SEI of the retained frame. The encoded information includes the original number of the current frame and the original number of the frame it references, thereby identifying the P-frame that was actively discarded and its original time position.

[0019] The receiving end decodes the received retained frames to obtain the pixel data of each retained frame;

[0020] For each discarded P-frame's original time position, temporal interpolation is performed using the two nearest decoded pixel frames before and after that time position to generate pixel data for that time position. When multiple P-frames are discarded consecutively, the two nearest decoded pixel frames before and after the discard interval are the nearest decoded pixel frame before the discard interval and the nearest decoded pixel frame after the discard interval. Interpolation is used to generate pixel data for all discarded time positions within that discard interval, and different interpolation weights are assigned to each time position.

[0021] The pixel data after frame decoding and the pixel data generated by interpolation of the discarded time and position are output together in the original display order.

[0022] Furthermore, in step 2:

[0023] The process of dividing the same video stream into multiple sub-streams includes: sequentially dividing the compressed bitstream of each original I-frame into m parts, and placing the j-th part into sub-stream S. j At the time position corresponding to the I-frame, each data belonging to the same I-frame in each sub-stream carries the same RTP timestamp; the same P-frame is completely allocated to the same sub-stream, the i-th P-frame is allocated to sub-stream S_(((i-1)modm)+1), the RTP timestamp of the same P-frame remains unchanged in different sub-streams; each sub-stream independently maintains the RTP sequence number space.

[0024] Furthermore, step 2 also includes quintuple negotiation and synchronization:

[0025] The sending and receiving ends use an in-band negotiation mechanism to allocate and synchronize the five-tuples. The sending end sends a five-tuple negotiation request message to the receiving end through the control channel of the original session. The message includes the number of sub-streams, the sub-stream ID, the five-tuples corresponding to each sub-stream, and the session identifier. After successful verification, the receiving end establishes a local mapping table to record the correspondence between each sub-stream ID and the five-tuples, and returns a negotiation confirmation message through the control channel. After receiving the confirmation message, the sending end begins to send the data of each sub-stream according to the negotiated five-tuples.

[0026] Furthermore, in step 3:

[0027] The receiver independently calculates the packet loss rate for each sub-stream, dividing the sub-streams into a set of no-packet-loss sub-streams C_good and a set of sub-streams with packet loss C_bad. It selects the sub-stream S_worst with the highest packet loss rate from C_bad, and selects the first no-packet-loss sub-stream S_target from C_good in ascending order of sub-stream ID. The receiver sends a 5-tuple adjustment request to the sender, modifying the 5-tuple of S_worst to match the 5-tuple of S_target, so that the packets carried by S_worst reuse the path where S_target is located.

[0028] Furthermore, step 3 also includes:

[0029] After the migration is completed, the receiving end continues to monitor the packet loss rate of each sub-stream and recalculates C_good and C_bad. If S_worst still experiences packet loss after migration, the next packet loss-free sub-stream from C_good is selected as the target for further migration attempts. If S_worst has already tried all packet loss-free sub-streams in C_good as target paths and packet loss still exists after migration, the path with the lowest packet loss rate after migration is selected from all the tried target paths, and S_worst is fixed to that path.

[0030] Furthermore, it also includes step 4, the recovery of the coding structure when it is stable in the long term:

[0031] When the receiver detects that the packet loss rate in the current statistical window is 0 and the packet loss rate in the previous statistical window is greater than 0, it starts a stabilization timer. If the stabilization timer continuously counts for a preset threshold L without detecting any new packet loss, the receiver sends a recovery notification to the transmitter. After receiving the recovery notification, the transmitter sends a complete I-frame carrying the recovery flag at the beginning of the next GOP period after the current GOP ends, and switches back to the original coding reference chain from this I-frame, sending all P-frames normally without executing the frame dropping strategy.

[0032] Furthermore, the multi-substream load balancing using ECMP multipath in step 2 and the adaptive load balancing link dynamic adjustment in step 3 remain effective after entry and do not exit due to the restoration of the encoding structure.

[0033] The present invention also discloses a system for reliable real-time data transmission, comprising:

[0034] The video transmitter supports dynamically adjusting the I / P frame reference relationship and supports dividing the same video stream into multiple RTP sub-streams and assigning an independent 5-tuple to each sub-stream for performing the transmitter operation in the method.

[0035] The video receiver has packet loss detection, multi-substream reassembly and decoding capabilities, and supports feeding back congestion status and five-tuple adjustment requests to the sender via RTCP extended messages, which are used to execute the receiver operations in the method described above.

[0036] The network system deploys routers that support Equal Cost Multipath (ECMP), which perform hash calculations based on the five-tuples of IP packets and select forwarding paths.

[0037] The sending end and the receiving end use an RTP / RTCP protocol stack for data transmission and control.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. By using a collaborative mechanism of dynamic frame dropping and receiver interpolation restoration, a two-level coding mechanism is triggered to adjust the strategy (one-chooses execution relationship) when congestion occurs. The sending end drops some P-frames and rebuilds the reference chain. The receiving end uses the pixel data of the previously decoded frames to restore the pixels at the time position of the dropped frames through temporal interpolation. This achieves the reduction of bit rate without relying on retransmission while ensuring the continuity of video images and avoiding retransmission from exacerbating congestion.

[0040] 2. By employing a multi-substream 5-tuple splitting mechanism, the same video stream is divided into multiple substreams and assigned different 5-tuples. Each substream independently maintains its RTP sequence number space, and the ECMP hash mechanism ensures that each substream is forwarded along different physical paths. This disperses video traffic, which was originally fixed to a single path, across multiple paths, effectively expanding the available bandwidth. The allocation and synchronization of 5-tuples between substreams are completed through in-band negotiation, eliminating the need for additional control connections.

[0041] 3. Through the adaptive quintuple dynamic migration mechanism, the receiving end continuously monitors the packet loss rate of each sub-stream and migrates the sub-stream with the highest packet loss rate to the path with no packet loss. If packet loss still occurs after migration, the next path is tried. If all paths fail, the path with the lowest packet loss rate is fixed, thus achieving continuous optimization of path load.

[0042] 4. Through a multi-level linkage mechanism (frame dropping → sub-stream splitting → path migration), the bit rate is preferentially reduced during mild congestion, and splitting and migration are initiated during severe congestion, thus maintaining image quality to the maximum extent while ensuring smooth transmission. Steps 2 (multi-sub-stream splitting) and 3 (path migration) remain active after being entered, without an active exit mechanism, to avoid re-triggering congestion due to bit rate concentration on a single link. Detailed Implementation

[0043] The specific embodiments of the present invention will be described in detail below. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention.

[0044] The video transmitter (such as a network camera or encoder) supports H.264 / H.265 encoding, has the ability to dynamically adjust the I / P frame reference relationship, and supports dividing the same video stream into multiple RTP sub-streams (according to a unified sub-stream division rule), allocating an independent UDP source port (independent 5-tuple) to each sub-stream. The video receiver (such as a decoder or player) has packet loss detection, multi-sub-stream reassembly (according to a unified sub-stream division rule), and decoding capabilities, and supports feeding back congestion status and 5-tuple adjustment requests to the transmitter via RTCP extended messages.

[0045] The network system deploys routers that support Equal Cost Multipath (ECMP). Routers select forwarding paths based on hash calculations of the five-tuples in IP packets. Data transmission and control between the sender and receiver are handled using the RTP / RTCP protocol stack.

[0046] The present invention provides a method for reliable real-time data transmission, comprising the following steps:

[0047] Step 1: The receiving end discards dynamically encoded frames based on network packet loss.

[0048] Step 1.1 Packet Loss Detection and Congestion Assessment

[0049] (1) The video sending end sends a video stream to the receiving end through the RTP protocol. The video stream adopts a GOP structure of I, P1, P2, P3, ..., Pn (GOP is an independent decoding basic unit in video coding that starts with an I frame and includes several subsequent predicted frames — P / B frames), where the I frame is a key frame, and Pi (i=1,2,...,n) is the i-th predicted frame. In the initial coding state (i.e., when no frame dropping adjustment is performed), the reference relationship of each frame is: P1 performs predictive coding with the I frame as the reference frame; for any i≥2, Pi performs predictive coding with its immediately preceding frame P{i-1} as the reference frame. The original reference chain is: I←P1←P2←P3←...←P{n-1}←Pn. The symbol "←" represents the "reference direction", for example, P1←P2 means that P2 refers to P1 for coding.

[0050] (2) The receiving end continuously monitors the received video data packets, and detects packet loss according to the Sequence Number in the RTP header: if the sequence numbers are not continuous, it is determined that packet loss has occurred. The receiving end counts the packet loss rate LossRate within a preset time window (F, configurable, such as 60s). The receiving end presets the packet loss rate threshold as T (configurable, such as 3%). When 0 < LossRate ≤ T, it is determined that mild network congestion occurs; when LossRate > T, it is determined that severe network congestion occurs. The receiving end feeds back a congestion notification to the sending end through an RTCP application-defined message or an extended report message, and the notification carries the current packet loss rate and the congestion level (mild / severe).

[0051] (3) When the receiving end detects that the packet loss rate LossRate in the current statistical window is 0 and the LossRate in the previous statistical window is greater than 0 (i.e., the state recovers from a packet loss state to a no-packet-loss state), the receiving end starts a stability timer. If no new packet loss is detected at all (that is, LossRate remains 0) during the period when the stability timer continuously counts up to a preset threshold L (L is configurable, with a recommended value of 30 to 60 minutes), the receiving end sends a "recovery notification" message to the sending end through the RTCP application-defined message or extended report message, and the message carries a notification type identifier (recovery notification tag set to 1) and the current packet loss rate (0). If new packet loss (LossRate > 0) is detected during the timing period of the stability timer, the timer is reset, and timing is restarted after the packet loss rate recovers to 0 again.

[0052] Step 1.2 First-level coding mechanism adjustment — interleaved single P frame dropping

[0053] (1) In response to a mild congestion notification (congestion level is mild), the sender initiates the first-level coding mechanism to adjust the strategy. For the P-frame sequence (P1, P2, ..., Pn) defined in step 1.1, the bit rate is reduced by step size d=2: the P-frames are divided into groups of d P-frames in the original order, i.e. (P1, P2), (P3, P4), (P5, P6) ..., and the bit rate is reduced by discarding the first P-frame and keeping the second P-frame in each group.

[0054] (2) After discarding the P-frames, the transmitting end re-encodes the remaining P-frames (i.e., P2, P4, P6...) and updates their reference relationships: the original P2 references P1, now it references the I-frame; the original P4 references P3, now it references P2; the original P6 references P5, now it references P4. And so on, forming a single-chain prediction structure of I←P2←P4←P6←... The re-encoded P-frames are encoded at the transmitting end and sent to the receiving end. It should be noted that the transmitting end not only needs to "discard" odd-numbered frames such as P1, P3, and P5, but more importantly, it must re-encode the remaining frames (even-numbered frames). If the transmitting end only discards odd-numbered frames without re-encoding even-numbered frames, the bitstream of P2 will still record the "reference P1" information, and the receiving end will fail to decode P2 in subsequent step 1.5 because it cannot find P1. Therefore, re-encoding the remaining frames is the core step to ensure that the receiving end can decode correctly.

[0055] (3) The specific implementation is as follows:

[0056] Group the n P frames P1, P2, ..., Pn into groups of two from front to back, i.e.:

[0057] Group 1: P1, P2

[0058] Group 2: P3, P4

[0059] Group 3: P5, P6

[0060] …

[0061] The last group: If n is even, the last group is P(n-1), Pn; if n is odd, the last group is only Pn (if the last group has less than 2 P frames, the entire group is retained and not discarded).

[0062] (4) The discarding rule is to discard the P-frame with the smaller number in each group and keep the P-frame with the largest number as the reference point of the group: For the complete group P(2t-1), P2t (t=1,2,...,n / 2), discard P(2t-1) and keep P2t; if the last group only has Pn (when n is odd), then keep Pn and do not discard it.

[0063] (5) The set of retained frames is: S1 = { P2t | t = 1, 2, ..., [n / 2]} ∪ { Pn | n is an odd number}. Reference frame reconstruction rule: Arrange the P frames in the retained frame set S1 in ascending order of their original numbers as P'1, P'2, ..., P'k, where P'1 is the first retained frame and P'k is the last retained frame. The new reference relationship is: Ref(P'1) = I frame; Ref(P't) = P'(t-1) (t ≥ 2). That is, the first retained frame directly references the I frame, and each subsequent retained frame references its previous retained frame, forming a single-chain prediction structure of I←P'1←P'2←...←P'k. The transmitting end re-performs inter-frame prediction coding on the retained frames according to the reconstructed reference relationship, generates a new compressed bitstream, and sends it to the receiving end.

[0064] (6) Example 1 (n=6, even number): Original frame structure I, P1, P2, P3, P4, P5, P6, original reference chain is I←P1←P2←P3←P4←P5←P6. First-level discarded groups are (P1,P2), (P3,P4), (P5,P6), discard P1, P3, P5, retain P2, P4, P6. The reconstructed reference chain is I←P2←P4←P6 (P2 references frame I, P4 references P2, P6 references P4).

[0065] (7) Example 2 (n=5, odd number): Original frame structure I, P1, P2, P3, P4, P5, original reference chain is I←P1←P2←P3←P4←P5. First-level discarded groups are (P1,P2), (P3,P4), (P5), discard P1 and P3, keep P2, P4, P5. The reconstructed reference chain is I←P2←P4←P5 (P2 references frame I, P4 references P2, P5 references P4).

[0066] Step 1.3 Second-level coding mechanism adjustment – ​​intermittent multi-P frame dropping

[0067] (1) In response to a severe congestion notification (congestion level is severe), the sender initiates the second-level coding mechanism adjustment strategy. (Note: Steps 1.2 and 1.3 are mutually exclusive. The sender directly selects the corresponding adjustment level based on the congestion level reported by the receiver, rather than executing step 1.2 first and then superimposing step 1.3.) For the P-frame sequence (P1, P2, ..., Pn) defined in step 1.1, the bit rate is reduced by a step size d=3 (i.e., every 3 P-frames are grouped together, the first two P-frames in each group are discarded, and the last P-frame is retained).

[0068] (2) After discarding the P-frames, the transmitting end re-encodes the remaining P-frames (i.e., P3, P6, P9...) and updates their reference relationships: the original P3 referenced P2, now it references the I-frame; the original P6 referenced P5, now it references P3; the original P9 referenced P8, now it references P6. And so on, forming a single-chain prediction structure of I←P3←P6←P9←... The re-encoded P-frames are encoded at the transmitting end and sent to the receiving end.

[0069] (3) The specific implementation is as follows:

[0070] Group the n P frames P1, P2, ..., Pn into groups of 3 from front to back, i.e.:

[0071] Group 1: P1, P2, P3

[0072] Group 2: P4, P5, P6

[0073] Group 3: P7, P8, P9

[0074] …

[0075] Last group: If there are fewer than 3 P-frames remaining, all P-frames in this group will be retained and not discarded.

[0076] (4) The discarding rule is to discard the first two P frames in each group and keep the P frame with the largest number as the reference point of the group: For the complete group P(3t-2), P(3t-1), P3t (t=1,2,...n / 3), discard P(3t-2) and P(3t-1) and keep P3t; if the last group has less than 3 P frames (i.e., 1 or 2 P frames remain), then all these P frames are kept and not discarded.

[0077] (5) The set of retained frames is: S2 = { P3t | t = 1, 2, ..., [n / 3]} ∪ { P([n / 3]·3+1), ..., Pn | if n mod 3 ≠ 0}. Reference frame reconstruction rule: Arrange the P frames in the set of retained frames S2 in ascending order of their original numbers as P”1, P”2, ..., P”k, where P”1 is the first retained frame and P”k is the last retained frame. The new reference relationship is: Ref(P”1) = I frame; Ref(P”t) = P”(t-1), (t ≥ 2). That is, the first retained frame directly references the I frame, and each subsequent retained frame references its previous retained frame, forming a single-chain prediction structure of I←P”1←P”2←...←P”k. The sending end re-performs inter-frame prediction coding on the retained frames according to the reconstructed reference relationship, generates a new compressed bitstream, and sends it to the receiving end.

[0078] (6) Example 1 (n=9, divisible by 3): The original frame structure is I, P1, P2, P3, P4, P5, P6, P7, P8, P9. The secondary discard groups are (P1,P2,P3), (P4,P5,P6), and (P7,P8,P9). P1, P2, P4, P5, P7, and P8 are discarded, while P3, P6, and P9 are retained. The reconstructed reference chain is I←P3←P6←P9 (P3 references frame I, P6 references P3, and P9 references P6).

[0079] (7) Example 2 (n=10, not divisible by 3): The original frame structure is I, P1, P2, P3, P4, P5, P6, P7, P8, P9, P10, grouped as (P1,P2,P3), (P4,P5,P6), (P7,P8,P9), (P10). P1, P2, P4, P5, P7, P8 are discarded, and P3, P6, P9, P10 are retained. The reconstructed reference chain is I←P3←P6←P9←P10 (P3 references frame I, P6 references P3, P9 references P6, and P10 references P9).

[0080] Step 1.4 Processing and routing at the receiving end

[0081] (1) After the sending end executes the frame dropping policy in step 1.2 or 1.3, the receiving end continues to monitor the packet loss situation:

[0082] (2) If LossRate > 0 (there is still network packet loss), it means that the current bit rate reduction measures are still insufficient to alleviate congestion. The receiver skips step 1.5 and directly enters the multi-substream five-element grouping mechanism in step 2.

[0083] (3) If LossRate = 0 (network packet loss has been eliminated), it means that the congestion has been relieved. The receiver performs step 1.5 to recover and decode the dropped frames and play the video normally.

[0084] Step 1.5 The receiving end restores the image data at the time of the discard and decodes the retained frame.

[0085] After receiving the bitstream adjusted by the encoding mechanism, the receiving end needs to restore the image data at the time positions that were discarded in order to ensure the integrity of the decoding reference chain and the continuity of video playback.

[0086] (1) Identification of discarded frames:

[0087] When encoding each reserved frame, the sending end carries the following information in the stripe header or SEI: ① the original number of the current frame; ② the original number of the frame referenced by the current frame.

[0088] By parsing the above information, the receiving end can determine which P frames were actively discarded and the reference relationships between the retained frames. For example, in the discard example in step 1.2 (n=6), the receiving end receives a bitstream containing only I, P2, P4, and P6, where P2 indicates a reference I frame, P4 indicates a reference P2 frame, and P6 indicates a reference P4 frame. Therefore, the receiving end knows that P1, P3, and P5 were discarded.

[0089] (2) Decoding of preserved frames:

[0090] It should be noted that the compressed bitstream at the discarded time position cannot be "reconstructed" from the bitstream of the retained frames. The fundamental reason is that the video compressed bitstream is a data stream generated by the encoder after performing DCT transformation, quantization, and entropy coding on the original image. There is no definite mathematical operation relationship between the bitstreams of different frames, and it is impossible to generate the bitstream at position P1 by performing any form of operation on the bitstreams of I-frame and P2 frame.

[0091] Therefore, pixel data restoration can be performed through temporal interpolation. The reference frames before and after the discarded time position must be decoded first to obtain their complete pixel data before they can be used as input for interpolation. Therefore, the receiving end first decodes the received retained frames (such as I, P2, P4, P6 in the scene adjusted by the first-level coding mechanism) to obtain the pixel data of each frame, and stores it in the reference frame buffer as the reference source for subsequent interpolation restoration.

[0092] (3) Interpolation restoration of discarded time and location data:

[0093] The receiver performs temporal interpolation (such as bidirectional motion-compensated interpolation) on the original position of each discarded P-frame using the two nearest decoded pixel frames before and after it to generate the pixel data of that frame, which is then stored in the reference frame buffer.

[0094] Level 1 coding mechanism adjustment scenario (d=2, 1 pixel dropped every 2 P frames): Taking n=6 as an example, the sender sends I, P2, P4, P6, and the data at positions P1, P3, and P5 in the original sequence is discarded. The generation method is as follows: the pixel data at the original time position P1 is generated based on the pixel interpolation of I frame and P2; the pixel data at the original time position P3 is generated based on the interpolation of P2 and P4; and the pixel data at the original time position P5 is generated based on the interpolation of P4 and P6.

[0095] Secondary encoding mechanism adjustment scenario (d=3, 2 frames dropped every 3 P frames): Taking n=9 as an example, the sender sends I, P3, P6, P9, and the data at positions P1, P2, P4, P5, P7, and P8 in the original sequence are discarded. The generation method is as follows: the pixel data at the original time positions P1 and P2 are generated based on interpolation of I and P3 frames, with different interpolation weights assigned according to their respective time positions; the pixel data at the original time positions P4 and P5 are generated based on interpolation of P3 and P6 frames; and the pixel data at the original time positions P7 and P8 are generated based on interpolation of P6 and P9 frames.

[0096] (4) Output:

[0097] The pixel data after retaining the frame decoding and the pixel data generated by interpolation of the discarded time position are output together in the original display order.

[0098] Step 2: Split the video stream into multiple sub-streams and use ECMP multipathing for load balancing.

[0099] When it is determined in step 1.4 that LossRate > 0 (i.e., network packet loss still exists after the adjustment of the first or second level encoding mechanism in step 1), the receiving end sends a "diversion request" message to the sending end.

[0100] Step 2.1 Sub-flow partitioning rules

[0101] (1) In response to the splitting request, the sending end divides the frame sequence of the same video stream into m sub-streams by staggering them in time (m is a configurable parameter, m≥2). The specific division method is as follows:

[0102] (2) I-frame splitting process: The compressed bitstream of each original I-frame is sequentially divided into m parts, where the j-th part (1≤j≤m) is placed in substream S. j The time position corresponding to the I-frame. Each data segment belonging to the same I-frame in each sub-stream carries the same RTP timestamp so that the receiver can identify and reassemble it.

[0103] (3) P-frame allocation: Each P-frame is completely allocated to the same substream without splitting it. Let the actual P-frame sequence sent after step 1 be P”1, P”2, ..., P”k (i.e., the P-frame sequence retained in step 1). The i-th P-frame P”i is allocated to substream S_(((i-1) mod m) + 1). Regardless of which substream the same P-frame is allocated to, its RTP timestamp remains unchanged, ensuring that the receiver can identify the frame's time position in the original video stream based on the timestamp. That is:

[0104] S1 carries: the first I-frame + P”1, P”m+1), P”2m+1), ...

[0105] S2 bearer: the second part of the I-frame + P”2, P”(m+2), P”(2m+2), ...

[0106] S3 bearer: the third copy of the I-frame + P”3, P”(m+3), P”(2m+3), ...

[0107] …

[0108] S m Bearer: the m-th I-frame + P”m, P”2m), P”3m), ...

[0109] The P-frames of each sub-stream are encoded and sent independently in their original order, and the same original P-frame will not be repeated in multiple sub-streams.

[0110] (4) Each sub-stream maintains its own sequence number space independently: Each sub-stream maintains its own RTP sequence number space independently when sending, and continuously increments from its own agreed starting value to ensure the sequence number is continuous within the same sub-stream. In this way, the packet loss detection method based on sequence number continuity defined in step 1.1 can be applied independently to each sub-stream.

[0111] Step 2.2 Five-tuple negotiation and synchronization

[0112] (1) The sender assigns a different 5-tuple to each sub-stream. Specifically, the source IP, destination IP, and protocol number remain unchanged, and different source port numbers are assigned to distinguish each sub-stream. To ensure that the receiver can correctly reassemble the sub-streams, the sender and receiver adopt an in-band negotiation mechanism: the sender sends a "5-tuple negotiation request" message through the control channel of the original session (such as RTSP / SIP signaling or RTCP). The message contains the number of sub-streams, sub-stream IDs (S1, S2, ...), the 5-tuples corresponding to each sub-stream, negotiation request tag (with tag set to 1), and carries the same session identifier (such as SSRC), so that the receiver can associate all sub-streams with the same original video session.

[0113] (2) After receiving the negotiation message, the receiving end verifies the validity of the message (such as matching session identifiers, the number of sub-streams, and the completeness of the 5-tuple information). If the verification is successful, a local mapping table is established to record the correspondence between each sub-stream ID and the 5-tuple, and a negotiation confirmation message is returned to the sending end through the control channel of the original session. The message contains: the negotiation result (success / failure), if successful, it carries the summary information of the established mapping table, if unsuccessful, it carries the error code, and the negotiation confirmation tag (and the tag is set to 1). After receiving the confirmation message, the sending end begins to send the data of each sub-stream according to the negotiated 5-tuple.

[0114] Step 2.3 Substream Transmission and ECMP Path Allocation

[0115] The sending end sends m sub-streams, each with its assigned 5-tuple, into the network. With ECMP enabled on the routers along the route to the receiving end, each router calculates a hash value for each IP packet based on its 5-tuple and takes the modulo of the number of equivalent paths N to select a forwarding path. Since each sub-stream has a different source port number, its 5-tuple hash value is different, and therefore they are assigned to different physical links. In this way, all packets from what was originally a single video stream are distributed across multiple paths, avoiding congestion on a single path and utilizing the total bandwidth of multiple links simultaneously.

[0116] Step 2.4 Substream Reassembly and Decoding Determination at the Receiver

[0117] The receiving end receives the RTP packets from each substream based on the negotiated five-tuple information. Reassembly is performed at two levels:

[0118] (1) RTP packet sorting and packet loss detection within the sub-stream

[0119] Each substream is processed independently: The receiver identifies each substream and receives its RTP packets according to the local mapping table established in step 2.2. All RTP packets received by the substream are then globally sorted by sequence number from smallest to largest. After sorting, the sequence numbers are checked for consecutive increments. If any sequence numbers are missing (e.g., 100, 101, 103 are received, but 102 is missing), the substream is considered to have packet loss. The number of missing sequence numbers is the packet loss count for the substream, used to calculate the substream packet loss rate LossRate_j within a preset time window (the time window is G, which can be set, e.g., 30 seconds). If all sequence numbers are consecutive (from the starting sequence number to the current maximum sequence number, each sequence number is received), the substream is considered to have no packet loss. If there is no packet loss in the substream, the receiver groups the sorted RTP packets by timestamp. Packets with the same timestamp belong to the same frame. These packets are then assembled into complete compressed bitstreams of each frame in sequence number order and stored in the substream buffer.

[0120] (2) Packet loss detection and traffic splitting

[0121] After each sub-stream completes packet loss detection, the receiving end checks the packet loss status of each sub-stream. If any sub-stream has packet loss (i.e., the packet loss rate of that sub-stream > 0), it indicates that multi-path splitting is still insufficient to alleviate congestion on the paths of each sub-stream. The receiving end abandons the complete video stream recovery and decoding, and directly proceeds to step 3 for adaptive load balancing and dynamic link adjustment. If all sub-streams have no packet loss, the next step is executed: complete video stream sequential recovery—recovering the complete video stream and decoding it for playback.

[0122] (3) Complete video stream sequence restoration:

[0123] When there is no packet loss in all substreams, the receiving end polls each substream in turn according to the number of substreams in the negotiation message and extracts P-frames from them based on the unified substream division rules:

[0124] Take P"1", P"m+1", P"2m+1", ... from S1

[0125] Take P”2, P”(m+2), P”(2m+2), ... from S2

[0126] …

[0127] From S m Let P”m, P”2m), P”3m), ...

[0128] For I-frames: Extract the data belonging to the same I-frame from each sub-stream, and assemble them in sequence (the first part, the second part, ..., the mth part) to restore the complete I-frame compressed bitstream.

[0129] The order of each P-frame in the original video stream is determined by the timestamps in the RTP header. Since the timestamp of the same frame remains unchanged regardless of which substream it is assigned to, the receiving end only needs to arrange the frames extracted from each substream in ascending order of timestamps to recover the original order of I,P”1,P”2,...,P”k.

[0130] After the complete video stream is recovered, the receiving end regenerates a set of continuously increasing RTP sequence numbers for the recovered frame sequence (starting from an agreed-upon initial value and increasing by 1 for each frame) so that the decoder can correctly identify the frame order when it is subsequently sent to the decoder. The original sequence numbers of each substream do not play a sorting role in cross-substream reconstruction and are no longer used after the recovery is complete.

[0131] (4) Decoding and playback

[0132] Finally, the recovered complete frame sequence is sent to the decoder for decoding.

[0133] Step 3: Adaptive load-sharing link dynamic adjustment – ​​optimizing sub-flow forwarding paths

[0134] When the receiver detects packet loss in any substream in step 2, proceed to step 3.

[0135] Step 3.1 Monitoring packet loss rate of each sub-stream

[0136] The receiving end independently calculates the packet loss rate LossRate_j for each sub-stream according to the packet loss detection method defined in step 1.1. That is, each sub-stream independently maintains a set of sequence number counts, and the receiving end determines that the sub-stream is losing packets by the discontinuity of the sub-stream sequence numbers, and calculates the packet loss rate of each sub-stream within a preset time window (G, configurable, such as 30s).

[0137] Step 3.2 Identification of streams with and without dropped buns

[0138] (1) The receiving end counts the packet loss rate of each sub-stream and divides the sub-stream into two groups: ① Sub-stream set with no packet loss C_good: the sub-stream set with LossRate_j = 0; ② Sub-stream set with packet loss C_bad: the sub-stream set with LossRate_j > 0.

[0139] (2) If C_bad is empty (no packet loss in any sub-stream), no adjustment is needed, and the current state is maintained; if C_good is empty (packet loss in all sub-streams), it means that all available paths are experiencing packet loss, and there are no lossless paths available for migration. At this time, the system maintains the current sub-stream division and bitrate state, and does not further degrade, waiting for the network to recover naturally or for an upper-layer alarm to be triggered.

[0140] Step 3.3 Dynamic migration of quintuples

[0141] (1) The receiver selects the sub-stream S_worst with the highest packet loss rate from C_bad, and selects the first sub-stream S_target without packet loss from C_good in ascending order of sub-stream ID (S1, S2, ..., Sm). The receiver sends a "five-tuple adjustment request" message to the sender through the control channel of the original session (such as RTSP / SIP signaling or RTCP). The message contains: the sub-stream ID of S_worst, the sub-stream ID of S_target, and the target five-tuple (i.e. the five-tuple of sub-stream S_target). The five-tuple adjustment request tag (with the tag set to 1) carries the same session identifier (SSRC), so that the sender can associate the request with the corresponding video session.

[0142] (2) After receiving the request, the sending end performs the change, modifying the quintuple of S_worst to the quintuple of S_target.

[0143] (3) After the change is completed, the sender sends a “five-tuple change notification” message to the receiver through the in-band negotiation channel. The message contains: the sub-stream ID of S_worst, the new five-tuple information of S_worst, the five-tuple change notification tag (with the tag set to 1), and carries the same session identifier. After receiving the notification, the receiver updates its local mapping table accordingly.

[0144] (4) After that, the P-frame packets carried by S_worst will be sent using the 5-tuple of S_target, and will be hashed to the same path as S_target on the ECMP router, thus reusing the lossless path.

[0145] Step 3.4 Iterative optimization of path migration

[0146] (1) After the migration is completed, the receiving end continues to monitor the packet loss rate of each sub-stream and re-counts C_good and C_bad (the packet loss status of each sub-stream may change after the migration. The sub-stream in the original C_good may become packet loss due to the new traffic, and the original S_worst may become packet loss-free. Therefore, it is necessary to re-count).

[0147] (2) If there is no more packet loss after S_worst migration (LossRate_worst = 0), then the migration is successful. The receiving end repeats steps 3.2 to 3.3 to continue processing the sub-stream with the highest packet loss rate in the current C_bad until there is no packet loss in all sub-streams, or C_good is empty.

[0148] (3) If packet loss still occurs after the migration of S_worst (LossRate_worst > 0), it indicates that the bandwidth of the target path is insufficient to carry the merged traffic. The receiving end selects the next packet-loss-free sub-stream from C_good in ascending order of sub-stream ID as the target, and sends a new adjustment request to the sending end according to the "Five-Tube Adjustment Request" message format defined in step 3.3, requesting that the five-tuple of S_worst be modified to the five-tuple of the target sub-stream; after receiving the request, the sending end executes the change and notifies the receiving end to update synchronously according to the "Five-Tube Change Notification" message format defined in step 3.3. The receiving end continues to monitor the packet loss situation after the migration.

[0149] (4) If S_worst has tried all the non-drop-loss sub-streams in C_good as target paths, and packet loss still exists after migration in all attempts, then the receiver selects the path with the lowest packet loss rate after migration from all the tried target paths, and sends a "five-tuple adjustment request" (final migration request) to the sender according to the five-tuple adjustment request message format defined in step 3.3, fixing S_worst on this path. At this time, the system maintains the current sub-stream division and bitrate state, waiting for the network to recover naturally or trigger an upper-layer alarm.

[0150] Step 3.5 Termination Judgment

[0151] After the path migration iteration optimization terminates, the following decision is made:

[0152] (1) If C_bad is empty (no packet loss in any sub-stream): The receiving end restores and decodes the playback of the complete video stream according to the complete video stream order defined in step 2.4, restores the frame data of each sub-stream into a complete video stream, and sends it to the decoder for decoding and playback. The process ends.

[0153] (2) If C_bad is not empty (there are still packet-loss sub-streams), and all packet-loss sub-streams have tried all packet-loss-free paths in C_good, but none have eliminated packet loss: this indicates that under the current network conditions, path migration alone cannot solve the packet loss problem. The system maintains the current sub-stream division and bitrate status, and will not make any further adjustments, waiting for the network to recover naturally or for an upper-layer alarm to be triggered. Video stream recovery is not performed in this state.

[0154] Step 4: Recovery of the coding structure under long-term stability

[0155] When the system is in the frame dropping state of step 1, and the sending end receives the "recovery notification" sent by the receiving end (triggered by step 1.1), the sending end restores the encoding mechanism of the complete video stream.

[0156] Step 4.1 Encoding Structure Recovery

[0157] (1) After receiving the “recovery notification”, the sending end sends a complete I-frame at the beginning of the next GOP period after the current GOP ends, and carries a recovery flag (recovery start tag set to 1) in the RTP packet extension header corresponding to the I-frame, indicating that the complete coding structure is recovered from the current GOP.

[0158] (2) At the start of the I-frame, the transmitting end switches back to the original coding reference chain and performs inter-frame predictive coding on all P-frames according to the complete reference relationship I←P1←P2←P3←...←Pn defined in step 1.1. All P-frames are transmitted normally, and the frame dropping strategy in step 1.2 or 1.3 is no longer executed.

[0159] (3) After receiving an I-frame carrying the recovery flag, the receiver actively disables the interpolation and restoration logic. Thereafter, the receiver processes the subsequent received complete frame sequences according to the standard decoding process and does not perform any interpolation or padding operations.

[0160] (4) If the video stream experiences packet loss again after the encoding mechanism is restored (i.e., the receiver detects LossRate > 0 again), the receiver immediately sends a congestion notification (mild / severe) to the sender. After receiving the notification, the sender will fall back to the frame discard state in step 1.2 or 1.3 in the next GOP period and re-encode the retained frames according to the corresponding frame discard strategy.

[0161] Step 4.2 Scope of Application

[0162] This step only restores the encoding structure (GOP integrity) of Step 1. Steps 2 (load sharing using ECMP multipath) and 3 (dynamic adjustment of load-sharing links) remain active after entry and no active exit mechanism is set for the following reasons: ① Substream splitting and path migration themselves do not degrade video encoding quality and do not affect user experience; ② Actively exiting Step 2 or Step 3 may re-trigger congestion due to bitrate concentration on a single link.

[0163] The present invention provides a system for reliable real-time data transmission, comprising:

[0164] The video transmitter supports dynamically adjusting the I / P frame reference relationship and supports dividing the same video stream into multiple RTP sub-streams and assigning an independent 5-tuple to each sub-stream for performing the transmitter operation in the method of this invention.

[0165] The video receiver has packet loss detection, multi-substream reassembly and decoding capabilities, and supports feeding back congestion status and five-tuple adjustment requests to the sender via RTCP extended messages, which are used to execute the receiver operations in the method of this invention.

[0166] The network system deploys routers that support Equal Cost Multipath (ECMP), which perform hash calculations based on the five-tuples of IP packets and select forwarding paths.

[0167] The sending end and the receiving end use an RTP / RTCP protocol stack for data transmission and control.

[0168] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for reliable real-time data transmission, characterized in that, Includes the following steps: Step 1: The receiving end feeds back dynamic encoded frame discarding based on network packet loss: The receiving end detects the packet loss rate of the video stream and determines the congestion level based on the packet loss rate, and feeds back a congestion notification to the sending end; The sending end selects to perform either the first-level encoding mechanism adjustment or the second-level encoding mechanism adjustment according to the congestion level, discards some P frames and reconstructs the reference chain of the retained frames before re-encoding and sending. The first-level coding mechanism is adjusted as follows: the P-frame sequence is grouped into groups of two from front to back. Each group discards the P-frame with the smaller number and retains the P-frame with the largest number as the reference point of the group. The retained frames are arranged in ascending order of their original numbers and the single-chain prediction structure is reconstructed. The first retained frame directly references the I-frame, and each subsequent retained frame references the previous retained frame. The second-level coding mechanism is adjusted as follows: the P-frame sequence is grouped into groups of 3 from front to back. The first two P-frames of each group are discarded, and the P-frame with the largest number is retained as the reference point of the group. The retained frames are arranged in ascending order of their original numbers and the single-chain prediction structure is reconstructed. The first retained frame directly references the I-frame, and each subsequent retained frame references the previous retained frame. Step 2, the video stream is split into multiple sub-streams and load balancing is performed using ECMP multipath: if network packet loss still occurs after the encoding mechanism in Step 1 is adjusted, the receiving end sends a split request to the sending end. In response to the splitting request, the sending end divides the same video stream into multiple sub-streams and assigns a different five-tuple to each sub-stream, and uses the ECMP hash mechanism to forward each sub-stream along different physical paths; Step 3, Adaptive load sharing link dynamic adjustment: The receiving end continuously monitors the packet loss rate of each sub-stream and migrates the sub-stream with the highest packet loss rate to the path corresponding to the five-tuple used by the sub-stream with no packet loss. If packet loss still occurs after migration, the next path with no packet loss is tried. If packet loss still occurs after trying all paths with no packet loss, the path with the lowest packet loss rate is fixed.

2. The method according to claim 1, characterized in that, Step 1 further includes: After adjusting the encoding mechanism, the receiving end continues to monitor packet loss: if the packet loss rate is still greater than 0, it skips the recovery and decoding of discarded frames and proceeds directly to step 2; if the packet loss rate is equal to 0, the receiving end performs the recovery and decoding of discarded frames. The receiving end parses the encoded information carried in the strip header or SEI of the retained frame. The encoded information includes the original number of the current frame and the original number of the frame it references, thereby identifying the P-frame that was actively discarded and its original time position. The receiving end decodes the received retained frames to obtain the pixel data of each retained frame; For each discarded P-frame's original time position, temporal interpolation is performed using the two nearest decoded pixel frames before and after that time position to generate pixel data for that time position. When multiple P-frames are discarded consecutively, the two nearest decoded pixel frames before and after the discard interval are the nearest decoded pixel frame before the discard interval and the nearest decoded pixel frame after the discard interval. Interpolation is used to generate pixel data for all discarded time positions within the discard interval, and different interpolation weights are assigned to each time position. The pixel data after frame decoding and the pixel data generated by interpolation of the discarded time and position are output together in the original display order.

3. The method according to claim 1, characterized in that, In step 2: The process of dividing the same video stream into multiple sub-streams includes: sequentially dividing the compressed bitstream of each original I-frame into m parts, and placing the j-th part into sub-stream S. j At the time position corresponding to the I-frame, each data belonging to the same I-frame in each sub-stream carries the same RTP timestamp; the same P-frame is completely allocated to the same sub-stream, the i-th P-frame is allocated to sub-stream S_(((i-1)modm)+1), the RTP timestamp of the same P-frame remains unchanged in different sub-streams; each sub-stream independently maintains the RTP sequence number space.

4. The method according to claim 1, characterized in that, Step 2 also includes quintuple negotiation and synchronization: The sending and receiving ends use an in-band negotiation mechanism to allocate and synchronize the five-tuples. The sending end sends a five-tuple negotiation request message to the receiving end through the control channel of the original session. The message includes the number of sub-streams, the sub-stream ID, the five-tuples corresponding to each sub-stream, and the session identifier. After successful verification, the receiving end establishes a local mapping table to record the correspondence between each sub-stream ID and the five-tuples, and returns a negotiation confirmation message through the control channel. After receiving the confirmation message, the sending end begins to send the data of each sub-stream according to the negotiated five-tuples.

5. The method according to claim 1, characterized in that, In step 3: The receiver independently calculates the packet loss rate for each sub-stream, dividing the sub-streams into a set of no-packet-loss sub-streams C_good and a set of sub-streams with packet loss C_bad. It selects the sub-stream S_worst with the highest packet loss rate from C_bad, and selects the first no-packet-loss sub-stream S_target from C_good in ascending order of sub-stream ID. The receiver sends a 5-tuple adjustment request to the sender, modifying the 5-tuple of S_worst to match the 5-tuple of S_target, so that the packets carried by S_worst reuse the path where S_target is located.

6. The method according to claim 5, characterized in that, Step 3 also includes: After the migration is completed, the receiving end continues to monitor the packet loss rate of each sub-stream and recalculates C_good and C_bad. If S_worst still experiences packet loss after migration, the next packet loss-free sub-stream from C_good is selected as the target for further migration attempts. If S_worst has already tried all packet loss-free sub-streams in C_good as target paths and packet loss still exists after migration, the path with the lowest packet loss rate after migration is selected from all the tried target paths, and S_worst is fixed to that path.

7. The method according to claim 1, characterized in that, It also includes step 4, recovery of the coding structure when it is stable in the long term: When the receiver detects that the packet loss rate in the current statistical window is 0 and the packet loss rate in the previous statistical window is greater than 0, the stabilization timer is started. If no new packet loss is detected during the period when the stabilization timer continuously counts up to the preset threshold L, the receiver sends a recovery notification to the sender. After receiving the recovery notification, the sender sends a complete I-frame carrying the recovery flag at the beginning of the next GOP period after the current GOP ends, and switches back to the original coding reference chain from the I-frame onwards, and sends all P-frames normally without executing the frame dropping strategy.

8. The method according to claim 7, characterized in that: The multi-substream in step 2 uses ECMP multipath for load sharing, and the adaptive load sharing link dynamic adjustment in step 3 remains effective after entering the process and does not exit due to the restoration of the encoding structure.

9. A system for reliable real-time data transmission, characterized in that, include: The video transmitter supports dynamically adjusting the I / P frame reference relationship and supports dividing the same video stream into multiple RTP sub-streams and assigning an independent 5-tuple to each sub-stream, for performing the transmitter operation in the method of any one of claims 1 to 8; The video receiver has packet loss detection, multi-substream reassembly and decoding capabilities, and supports feeding back congestion status and five-tuple adjustment requests to the sender via RTCP extended messages, and is used to perform the receiver operation in any one of claims 1 to 8. The network system deploys routers that support Equal Cost Multipath (ECMP), which perform hash calculations based on the five-tuples of IP packets and select forwarding paths. The sending end and the receiving end use an RTP / RTCP protocol stack for data transmission and control.