A multi-scenario adaptive packet loss and retransmission video transmission method and system

Abstract

The present invention discloses a multi-scenario adaptive packet loss and retransmission video transmission method and system. The method includes the following steps: each terminal is connected to a streaming media server; the first terminal sends an RTP packet containing a complete frame of data to a first processing endpoint; the first processing endpoint receives the RTP packet sent by the first terminal and calculates the packet loss rate; after the first terminal receives the retransmission request, it sends the RTP retransmission packet with the corresponding sequence number to the first processing endpoint; after the first processing endpoint receives the retransmission packet, it integrates and sorts all the RTP packets in the frame, and sends a complete frame of data to the second processing endpoint with a fixed delay; the second processing endpoint caches the complete frame of data sent by the first processing endpoint, and packages it into multiple RTP packets and sends them to the second terminal corresponding to the second processing endpoint. Utilizing the present invention, the video freeze phenomenon under network fluctuations is effectively improved.

Description

Technical Field

[0001] The present invention relates to a multi-scenario adaptive packet loss and retransmission video transmission method, and also relates to a corresponding adaptive packet loss and retransmission video transmission system, belonging to the technical field of multimedia communications. Background Art

[0002] In existing IP multimedia dispatch communication systems, video calls can experience video freezes under poor network conditions, severely impacting call quality. This freeze occurs because the receiving end sorts incoming video packets according to the RTP (Real-Time Transport Protocol) message sequence number. After receiving a complete frame, the receiving end sends the frame to the decoding layer for decoding and display. If network packet loss occurs, the incomplete video frame is buffered until a retransmission packet is received. Further processing is performed only after the frame is complete.

[0003] In poor network conditions, if packet loss occurs, the video packet sorting and caching process must wait for the lost packet to be transmitted locally. This wait time is set to a preset value. If this value is set too high, retransmissions may fail and the waiting time may continue, causing severe video decoding delays. If this value is set too low, the retransmitted packet may be discarded before it reaches the receiving end, resulting in severe video quality degradation. Furthermore, in video conferencing scenarios, packet loss between one terminal and the server can directly affect the reception and display of other terminals, causing video delays on other terminals as well.

[0004] To address the above issues, the applicant, in Chinese invention patent number ZL 201310037503.1, proposed a method for adaptive network transmission and automatic frame reduction control for streaming media distribution. This method utilizes a video streaming system with adaptive real-time data transmission control based on the TCP transmission protocol. This method creates a distribution service unit for each uniquely numbered video resource in the video streaming system and creates corresponding memory space to process the data frames of the video resource. This automatically determines the video data transmission status and assesses the client network level. Based on the specific assessment results, the method automatically implements QoS decision control for automatic frame reduction, thereby effectively ensuring the quality of real-time streaming media transmission, effectively avoiding abnormal phenomena such as video jitter, mosaics, screen distortion, and tailing, reducing the complexity of application-layer implementation logic, and improving the real-time and smoothness of the video experience. Summary of the Invention

[0005] The primary technical problem to be solved by the present invention is to provide a multi-scenario adaptive packet loss and retransmission video transmission method.

[0006] Another technical problem to be solved by the present invention is to provide a multi-scenario adaptive packet loss and retransmission video transmission system.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a multi-scenario adaptive packet loss and retransmission video transmission method is provided, comprising the following steps:

[0009] S1: Each terminal is connected to the streaming media server;

[0010] S2: The first terminal sends an RTP packet containing a complete frame of data to the first processing endpoint;

[0011] S3: The first processing endpoint receives the RTP packet sent by the first terminal and calculates the packet loss rate;

[0012] S4: After receiving the retransmission request, the first terminal sends the RTP retransmission packet with the corresponding sequence number to the first processing endpoint;

[0013] S5: After receiving the retransmission packet, the first processing endpoint integrates and sorts all RTP packets in the frame and sends a complete frame of data to the second processing endpoint with a fixed delay;

[0014] S6: The second processing endpoint buffers a complete frame of data sent by the first processing endpoint, and packages the data into multiple RTP packets and sends them to the second terminal corresponding to the second processing endpoint.

[0015] Preferably, step S3 further includes the following sub-steps:

[0016] If the packet loss rate remains equal to 0, the first processing endpoint buffers the RTP packet and sends a complete frame of data to the second processing endpoint;

[0017] If the packet loss rate continues to be greater than 0, the first processing endpoint determines the RTP sequence number of the lost packet, sends a retransmission request to terminal A, and waits for the retransmission packet;

[0018] If the packet loss rate of the previous cycle is greater than 0 and the packet loss rate of the next cycle is equal to 0, delay recovery is performed, and subsequent complete data frames are sent to the second processing endpoint at the delay recovery rate.

[0019] Preferably, the waiting time when the packet loss rate is between 0% and 10% and between 10% and 20% increases linearly.

[0020] Preferably, the waiting time is calculated as follows:

[0021] Waiting time = (X / 10+1)*RTT, where 0 < packet loss rate X% ≤ 20%, and RTT is the round-trip time of the data packet;

[0022] Waiting time = 3*RTT, where the packet loss rate X% is greater than 20%.

[0023] There is a one-to-one correspondence between the packet loss rate and the upper limit of the waiting time.

[0024] Preferably, the delay recovery includes the following steps: accelerating the transmission of data involving a total data volume of (X / 10+1) times the round-trip delay (RTT) at a fixed growth rate, and automatically switching to normal transmission speed when the total data volume is sent.

[0025] Preferably, the fixed speed increase ratio is in the range of 1.2 to 1.5 times the normal speed.

[0026] Preferably, the fixed growth rate ratio is RTT / 2.

[0027] Preferably, after step S6, the method further includes:

[0028] S7: The second terminal receives the RTP packet sent by the second processing endpoint and calculates the packet loss rate;

[0029] S8: The second terminal sorts and integrates the RTP packets, and decodes and displays a complete frame of data after the integration is completed.

[0030] According to a second aspect of an embodiment of the present invention, a multi-scenario adaptive packet loss and retransmission video transmission system is provided, comprising a streaming media server and a plurality of participating terminals, wherein the streaming media server comprises a plurality of processing terminals, and the processing terminals are connected to the participating terminals via a network to implement the above-mentioned video transmission method.

[0031] Compared with the prior art, the present invention has the following technical effects:

[0032] 1. Dynamically adjust the retransmission waiting time based on the network packet loss situation and the current RTT time. This is more flexible than a fixed time and avoids the abnormal situation of long waiting time when the network conditions are good and frame discarding when the network conditions are bad.

[0033] 2. By distinguishing between single call scenarios and video conferencing scenarios, different strategies are adopted for RTP packet sending and RTP retransmission application after packet loss detection.

[0034] (1) In a single call scenario, the streaming media server directly transmits signaling to reduce display delay;

[0035] (2) In a video conferencing scenario, different terminals are separated from the processing endpoints by a streaming media server. Network changes from one terminal to the server will not affect the communication effects from other terminals to the server.

[0036] Through the above strategies, the video freeze phenomenon under network fluctuations is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of an adaptive packet loss and retransmission video transmission system provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a flow chart of a video transmission method provided by an embodiment of the present invention;

[0039] Figure 3 This is a typical RTP packet structure diagram;

[0040] Figure 4 Schematic diagram of the relationship between packet loss rate and waiting time in an embodiment of the present invention;

[0041] Figure 5 This is a transmission principle diagram of a packet loss reselection scenario in an embodiment of the present invention;

[0042] Figure 6 This is a transmission principle diagram of a single call scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] The embodiment of the present invention first provides a multi-scenario adaptive packet loss and retransmission video transmission system. Figure 1 As shown, the adaptive packet loss retransmission video transmission system includes a streaming server and multiple (for example, 3, 6, 12, etc.) participating terminals, wherein the streaming server includes multiple processing terminals; wherein the processing terminals are connected to the participating terminals via a network to achieve Figure 2 The video transmission method shown.

[0045] like Figure 2 As shown, an embodiment of the present invention provides a multi-scenario adaptive packet loss and retransmission video transmission method, which is mainly applicable to one-to-one, one-to-many, or many-to-many video transmission scenarios, such as video conferencing, but can also be used in other application scenarios such as online teaching and scheduling. In one embodiment of the present invention, the multi-scenario adaptive packet loss and retransmission video transmission method includes at least the following steps:

[0046] S1: Each terminal is connected to the streaming media server.

[0047] The algorithm module has been embedded in the streaming media server. When users launch their own programs (including but not limited to mini-programs and apps), the algorithm module is loaded for their respective terminals.

[0048] The algorithm modules include: video stream caching mechanism, RTP packet sorting method, message (packet) statistics method, packet loss rate calculation method, etc.

[0049] exist Figure 2 In the illustrated embodiment, three terminals are used as an example for detailed description. Specifically, the processing endpoints in the streaming media server include a first processing endpoint (End 1), a second processing endpoint (End 2), and a third processing endpoint (End 3). The terminals that join the conference include a first terminal (Terminal A), a second terminal (Terminal B), and a third terminal (Terminal C).

[0050] S2: Terminal A sends an RTP packet (also called an RTP group) containing a complete frame of data to the first processing endpoint End1.

[0051] To determine whether a frame of data is complete, the following three conditions must be met simultaneously:

[0052] 1. The RTP packets are sorted by sequence number from the beginning of the frame to the end of the frame. If there is no missing sequence number in the middle, the frame data is continuous.

[0053] 2. The timestamp of a frame of data is the same;

[0054] 3. The frame end bit (Marker bit) of the last RTP packet is true.

[0055] If all three of the above aspects are met, the frame is judged to be complete. Figure 3 This is a typical RTP packet structure diagram.

[0056] S3: The first processing endpoint End 1 receives the RTP packet sent by terminal A and calculates the packet loss rate.

[0057] The calculation rule for the packet loss rate is:

[0058] Packet loss rate = number of lost packets / (RTP sequence number (maximum) - RTP sequence number (minimum) + 1) * 100%.

[0059] Here, there are three types of packet loss rates:

[0060] 1. The packet loss rate remains at 0.

[0061] 2. The packet loss rate is continuously greater than 0;

[0062] 3. The packet loss rate of the previous cycle is greater than 0 and the packet loss rate of the next cycle is equal to 0

[0063] The duration represented by "Continue" is a period of 5 seconds.

[0064] If the packet loss rate remains equal to 0, the first processing endpoint End 1 buffers the RTP packet and sends a complete frame of data to the second processing endpoint End 2, and then proceeds to step S6;

[0065] If the packet loss rate is continuously greater than 0, the first processing endpoint End 1 determines the sequence number of the discarded RTP packet through the algorithm module, sends a retransmission request (NACK) to terminal A, and waits for the retransmission packet, and enters step S4;

[0066] If the waiting time for the retransmission packet exceeds the waiting time, all RTP packets of the frame video data are discarded, the next frame data is received and verified, and then the process returns to step S3.

[0067] The time for waiting for packet retransmission starts when packet loss occurs.

[0068] Among them, the packet loss rate statistics will fluctuate due to network fluctuations. In order to ensure that the waiting time does not change in steps, such as Figure 4 As shown in the figure, the waiting time should increase linearly when the packet loss rate is between 0% and 10% and between 10% and 20%, and remain unchanged when the packet loss rate exceeds 20%. The specific method for calculating the waiting time is as follows:

[0069] Waiting time = (X / 10+1)*RTT, where 0 < packet loss rate X% ≤ 20%;

[0070] Waiting time = 3*RTT, where the packet loss rate X% is greater than 20%.

[0071] Where X% is the packet loss rate, the upper limit of (X / 10+1) is 3, and RTT is the round-trip time of the data packet.

[0072] In one embodiment of the present invention, the packet loss rate and the waiting time are in a one-to-one correspondence. For example, when the packet loss rate is 5%, the waiting time is 1.5*RTT; when the packet loss rate is 10%, the waiting time is 2*RTT, and so on.

[0073] If the packet loss rate of the previous cycle is greater than 0 and the packet loss rate of the next cycle is equal to 0, delay recovery is performed, and the complete data of subsequent frames are sent to the second processing endpoint End2 at the delay recovery rate, and step S6 is entered.

[0074] The specific steps of delay recovery are: the total amount of data involved is (X / 10+1) times the RTT, and the data is accelerated at a fixed speed ratio. When the total amount of data is sent, it automatically returns to the normal transmission speed.

[0075] As an example, when the packet loss rate is X%, (X / 10+1) times the RTT of data will be used as the total data volume for delay recovery. If the packet loss rate of the next frame drops to 0, the RTP packets cached within (X / 10+1) times the RTT will be accelerated at a fixed ratio (for example, a ratio of RTT / 2).

[0076] It should be noted that the inventors consider that the critical flicker frequency that the human eye can detect is generally around 45 Hz. The relationship between Hz and frame rate is equal, that is, 1 frame = 1 Hz. Therefore, under normal circumstances, the human eye is not easily aware of changes in playback speed below 45 frames per second. Based on these reasons, embodiments of the present invention recommend that the acceleration ratio of delay recovery be less than the RTT. For example, the acceleration ratio can be set to RTT / 2.

[0077] In one embodiment of the present invention, assuming the current RTT = 200ms and the packet loss rate is 10%, the send buffer time is (10 / 10+1)*RTT, that is, the data from 400ms ago. If the packet loss rate drops to 0 in the next second, the packet loss delay recovery state is entered. That is, in the next second, 400-RTT / 2 packets, that is, the packets received 300ms ago, must be sent. The delay is reduced by 100ms (RTT / 2) per second thereafter, i.e., 200ms, 100ms, 0ms. In other words, after 4 seconds, the data delay recovery is complete, and synchronous transmission is performed after recovery is complete.

[0078] By using the above method, the delay caused by the video freeze phenomenon can be restored at an accelerated rate that cannot be recognized by the naked eye, thereby greatly reducing the impact of the video freeze phenomenon.

[0079] S4: After receiving the retransmission request, terminal A sends the RTP retransmission packet with the corresponding sequence number to the first processing endpoint End1, and then goes to step S5.

[0080] like Figure 5 As shown, when packet loss occurs between terminal A and the first processing endpoint End 1, the algorithm module of the first processing endpoint End 1 counts the packet loss situation, sends a retransmission request to terminal A, and terminal A sends a retransmission RTP packet to the first processing endpoint End 1.

[0081] Because each processing terminal in the streaming media server and each participating terminal are loaded with algorithm modules, changes in the network conditions between one participating terminal and its corresponding processing endpoint will not affect the video stream transmission of other participating terminals.

[0082] The retransmission packet will be transmitted with a different RTP Payload Type to distinguish between the RTP packet and the retransmission packet.

[0083] S5: After receiving the retransmission packet, the first processing endpoint End 1 integrates and sorts all RTP packets in the frame, and sends a complete frame of data to the second processing endpoint End 2 with a fixed delay, and then enters step S6.

[0084] Due to network fluctuations, the packet loss rate may fluctuate. To ensure that the fixed delay does not experience step changes, the fixed delay should increase linearly when the packet loss rate is between 0% and 10% and between 10% and 20%. The calculation method is: Fixed delay = (Packet loss rate / 10 + 1) * RTT. The first processing endpoint, End 1, will not forward a video frame to the second processing endpoint, End 2, until it receives a complete frame of data. Since End 1 and End 2 are processing endpoints in the streaming media server, packet loss is not an issue during data transmission between them. Therefore, End 2 only caches complete frames of data.

[0085] In one embodiment of the present invention, when packet loss occurs, data will be sent with a fixed delay, so video freeze will only occur once when packet loss occurs, and video freeze will not occur again when packet loss continues, thereby greatly improving the video playback effect.

[0086] S6: The second processing endpoint End 2 buffers a complete frame of data sent by the first processing endpoint End 1, and packages the data into multiple RTP packets and sends them to the terminal B corresponding to the second processing endpoint End 2.

[0087] S7: Terminal B receives the RTP packet sent by the second processing endpoint End 2 and determines the packet loss rate.

[0088] The calculation method of the packet loss rate, the operations performed on different packet loss rates, and the specific operation methods are the same as those in steps S3 to S6 and will not be described in detail here.

[0089] S8: Terminal B sorts and integrates the RTP packets, and after integration is completed, decodes and displays a complete frame of data.

[0090] Terminal B's decoding is passive, meaning it decodes all data received simultaneously without counting the data. Therefore, when receiving data that has undergone delay recovery, Terminal B also decodes and displays it completely. However, the data has already been accelerated by the RTT / 2 ratio during the delay recovery process, increasing the amount of data required for decoding. Because a decoder can normally decode up to 120 frames per second, while a streaming server typically receives and sends 15 to 30 frames per second, the changes in data volume and decoding rate caused by delay recovery are negligible. Therefore, Terminal B's decoding and display process is unaffected by delay recovery.

[0091] Compared with the existing technology, the multi-scenario adaptive packet loss and retransmission video transmission method provided by the present invention dynamically adjusts the retransmission packet waiting time based on the network packet loss situation and the current RTT time. It is more flexible than a fixed time, avoids the abnormal situation of long waiting when the network conditions are good and frame discarding when the network conditions are bad, and effectively improves the video freeze phenomenon under network fluctuations.

[0092] It should be noted that, in the embodiments of the present invention, the data forwarding processes between the first processing endpoint End 1 and the third processing endpoint End 3, and between the third processing endpoint End 3 and terminal C, are consistent with the forwarding processes between the first processing endpoint End 1 and the second processing endpoint End 2, and between the second processing endpoint End 2 and terminal B. Specifically, the processes for forwarding RTP packets from terminal B to terminals A and C, and for forwarding RTP packets from terminal C to terminals A and B, are consistent with the aforementioned forwarding processes for terminal A and are not further described here.

[0093] Figure 6 The figure is a transmission principle diagram of a single call scenario in an embodiment of the present invention. A significant feature of the present invention is that different strategies are adopted for RTP packet sending and RTP retransmission application in the process of detecting packet loss by distinguishing between single call scenarios and video conferencing scenarios. That is,

[0094] (1) In a single call scenario, the streaming media server directly transmits signaling to reduce display delay;

[0095] (2) In a video conferencing scenario, different terminals are separated from the processing endpoints by a streaming media server. Network changes from one terminal to the server will not affect the communication effects from other terminals to the server.

[0096] Through the above strategies, the video freeze phenomenon under network fluctuations is further improved.

[0097] The above describes in detail the multi-scenario adaptive packet loss retransmission video transmission method and system provided by the present invention. For those skilled in the art, any obvious modifications made to the present invention without departing from its essence will constitute infringement of the present invention's patent rights and will result in corresponding legal liability.

Claims

1. A multi-scenario adaptive packet loss and retransmission video transmission method, characterized in that: include: S1: Each terminal is connected to the streaming media server; S2: The first terminal sends an RTP packet containing a complete frame of data to the first processing endpoint; S3: The first processing endpoint receives the RTP packet sent by the first terminal and calculates the packet loss rate; S4: After receiving the retransmission request NACK, the first terminal sends the RTP retransmission packet with the corresponding sequence number to the first processing endpoint; S5: After receiving the retransmitted packet, the first processing endpoint integrates and sorts all RTP packets in the frame and sends a complete frame of data to the second processing endpoint with a fixed delay; S6: The second processing endpoint buffers a complete frame of data sent by the first processing endpoint, and packages it into multiple RTP packets and sends them to the second terminal corresponding to the second processing endpoint; The step S3 comprises: If the packet loss rate remains equal to 0, the first processing endpoint buffers the RTP packet and sends a complete frame of data to the second processing endpoint; If the packet loss rate continues to be greater than 0, the first processing endpoint determines the RTP sequence number of the lost packet, sends a retransmission request NACK to the first terminal, and waits for the retransmission packet; If the packet loss rate of the previous cycle is greater than 0 and the packet loss rate of the next cycle is equal to 0, delay recovery is performed, and the complete data of subsequent frames are sent to the second processing endpoint at the rate of delay recovery; The waiting time between the packet loss rate of 0-10% and 10-20% increases linearly; The waiting time is calculated as follows: (0<packet loss rate X%≤20%); (Packet loss rate X%>20%); RTT is the round-trip time of a data packet.

2. The adaptive packet loss and retransmission video transmission method according to claim 1, wherein: The delay recovery method is to accelerate the transmission of data with a total data volume of (X / 10+1) times the RTT at a fixed growth rate. When the total data volume is sent, it will automatically return to the normal transmission speed.

3. The adaptive packet loss and retransmission video transmission method according to claim 2, wherein: The fixed growth rate ratio is RTT / 2.

4. The adaptive packet loss and retransmission video transmission method according to claim 1, wherein: After step S6, the method further includes: S7: The second terminal receives the RTP packet sent by the second processing endpoint and calculates the packet loss rate; S8: The second terminal sorts and integrates the RTP packets, and decodes and displays a complete frame of data after the integration is completed.

5. A multi-scenario adaptive packet loss and retransmission video transmission system, characterized in that: It includes a streaming media server and multiple participating terminals, the streaming media server includes multiple processing terminals, and is characterized in that the processing terminals and the participating terminals are connected through a network to implement the adaptive packet loss retransmission video transmission method described in any one of claims 1 to 4.

Images