A real-time multimedia streaming transmission method

Abstract

The present invention provides a real-time multimedia streaming transmission method in the field of multimedia streaming technology, including: step S10, the server calculates the playback expiration timestamp of the frame data of the first frame of the real-time multimedia after it arrives at the client; step S20, the server sends each frame data of the real-time multimedia to the client in sequence through QUIC datagrams, stores the sent QUIC datagrams in a retransmission buffer, updates the playback expiration timestamp and sequence number, and synchronizes the playback expiration timestamp to the client; step S30, the client feeds back a reception confirmation response to the server based on the received QUIC datagram, and the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response; step S40, the client performs validity verification on the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data. The advantages of the present invention are: greatly improving the reliability and real-time performance of real-time multimedia streaming transmission.

Description

Technical Field

[0001] The present invention relates to the field of multimedia streaming technology, and in particular to a real-time multimedia streaming transmission method. Background Art

[0002] Multimedia applications often use the RTP protocol to transmit real-time multimedia streams. RTP is an unreliable transport protocol that runs on top of the UDP protocol. UDP's retransmission-free nature makes it widely used in various low-latency applications, but RTP can cause noticeable lag due to packet loss. Using TCP to transmit real-time multimedia streams, while more reliable than RTP, also introduces high latency.

[0003] The QUIC protocol is a modern, reliable transport protocol that runs in user space. Based on the UDP protocol and running at the application layer, it is easier to modify than the kernel-level TCP protocol stack to adapt it to multimedia applications. As an alternative to the TCP protocol, it is being adopted in various environments. However, the reliability and on-demand delivery characteristics of the QUIC protocol may cause head-of-head blocking, which in turn leads to high latency in multimedia applications.

[0004] Therefore, how to provide a real-time multimedia streaming transmission method to improve the reliability and real-time performance of real-time multimedia streaming transmission has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a real-time multimedia stream transmission method to improve the reliability and real-time performance of real-time multimedia stream transmission.

[0006] The present invention is implemented as follows: a real-time multimedia stream transmission method comprises the following steps:

[0007] Step S10: The server calculates a playback expiration timestamp after the first frame of the real-time multimedia arrives at the client.

[0008] Step S20: The server sends each frame of the real-time multimedia data to the client in sequence via QUIC datagrams, stores the sent QUIC datagrams in a retransmission buffer, updates the playback expiration timestamp and sequence number, and synchronizes the playback expiration timestamp to the client;

[0009] Step S30: The client feeds back a reception confirmation response to the server based on the received QUIC datagram, and the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response;

[0010] Step S40: The client performs validity verification on the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data.

[0011] Furthermore, in step S10, the calculation formula of the playback end timestamp is:

[0012] RT=((path_latency / a)*b)+c;

[0013] Among them, RT represents the playback deadline timestamp; path_latency represents the delay of the network path; a represents the interval between the callback timers 60 times per second; b represents the time interval between each I frame; and c represents the playback buffer time of the buffered frame with four frames reserved.

[0014] Furthermore, in step S20, the frame type of the frame data is an I frame or a P frame; the I frame is an internal frame used to provide data for the entire image, the frame type of the frame data of the first frame is an I frame, and only one I frame is sent for every ten frames of frame data.

[0015] Furthermore, in step S20, after storing the sent QUIC datagram in the retransmission buffer, dependency information of the P frame carried in the QUIC datagram is also added; the dependency information is the I frame on which the P frame depends.

[0016] Furthermore, in step S20, the updating of the playback end timestamp and the sequence number is specifically as follows:

[0017] The server increases b when the playback end timestamp is called back, and accumulates the sequence number with a gradient of 1; the sequence number represents the total number of frame data that has been sent.

[0018] Furthermore, in step S20, before synchronizing the playback end timestamp to the client, the server adds four buffer frames to the playback end timestamp.

[0019] Furthermore, in step S30, the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response:

[0020] After the server receives the reception confirmation response, it determines whether the corresponding QUIC datagram is still useful based on the dependency information and the playback deadline timestamp. If so, it retains the corresponding QUIC datagram in the retransmission buffer; if not, it deletes the corresponding QUIC datagram in the retransmission buffer.

[0021] Furthermore, the step S40 is specifically as follows:

[0022] The client determines whether there is a missing data frame based on the frame number carried by the data frame. If not, the client plays the frame data carried by the QUIC datagram; if so, then:

[0023] Based on the expiration timestamp and adjacent data frames, the validity period of the missing data frame is checked. If it is expired, the missing data frame is skipped and the frame data of the next frame is played; if it is not expired, the data frame retransmitted by the buffer frame receiving server is played.

[0024] The advantages of the present invention are:

[0025] The server calculates the playback expiration timestamp of the frame data of the first frame of real-time multimedia after it arrives at the client, sends each frame data to the client in sequence through the QUIC datagram and caches it in the retransmission buffer, updates the playback expiration timestamp and sequence number, synchronizes the playback expiration timestamp to the client, and deletes the corresponding QUIC datagram in the retransmission buffer based on the reception confirmation response sent by the client; the client verifies the validity period of the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data; that is, the playback expiration timestamp is used to skip expired data frames to avoid the accumulated delay caused by the reliable transmission of the standard QUIC protocol, and by adding a four-frame buffer frame in the playback expiration timestamp to receive the data frames retransmitted by the server, that is, the lost data frames can be retransmitted and played before they expire, thereby improving the throughput of useful data and ensuring the playback quality of the real-time multimedia stream, ultimately greatly improving the reliability and real-time performance of real-time multimedia stream transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] Figure 1 The present invention is a flowchart of a real-time multimedia streaming transmission method. DETAILED DESCRIPTION

[0028] The technical solution in the embodiments of the present application has the following overall idea: skipping expired data frames by playing the deadline timestamp to avoid the accumulated delay caused by the reliable transmission of the standard QUIC protocol, and adding a four-frame buffer frame within the playing deadline timestamp to receive the data frames retransmitted by the server, that is, the lost data frames can be retransmitted and played before they expire, thereby improving the reliability and real-time performance of real-time multimedia streaming transmission.

[0029] Please refer to Figure 1 As shown, a preferred embodiment of a real-time multimedia stream transmission method of the present invention includes the following steps:

[0030] Step S10: The server calculates a playback expiration timestamp after the first frame of the real-time multimedia arrives at the client.

[0031] Step S20: The server sends each frame of the real-time multimedia data to the client in sequence via QUIC datagrams, stores the sent QUIC datagrams in a retransmission buffer, updates the play end timestamp and sequence number, and synchronizes the play end timestamp to the client. In a specific implementation, the play end timestamp can be inserted into the first four QUIC datagrams.

[0032] The length of the playback expiration timestamp and sequence number are both 32 bits, and are passed to the protocol stack through the send function call; regardless of the frame type, the sequence number will increase monotonically, and the playback expiration timestamp of the P frames associated with the same I frame will be the same;

[0033] Step S30: The client feeds back a reception confirmation response to the server based on the received QUIC datagram, and the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response;

[0034] Step S40: The client performs validity verification on the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data.

[0035] The present invention optimizes the transmission of real-time multimedia streams by modifying the standard QUIC protocol. The standard QUIC protocol will bring accumulated delays due to reliable transmission. The present invention allows the client to skip the lost frame data when the playback deadline timestamp is about to arrive, thereby solving the delay problem through partial reliable transmission.

[0036] In step S10, the calculation formula of the playback end timestamp is:

[0037] RT=((path_latency / a)*b)+c;

[0038] Where RT represents the playback end timestamp; path_latency represents the network path latency; a represents the interval between callback timers (60 times per second); b represents the interval between each I-frame; and c represents the playback buffer time for the four-frame buffer. The preferred values ​​for a, b, and c are 16.7 seconds, 3000 seconds, and 12000 seconds, respectively.

[0039] The playback deadline timestamp can be used to calculate whether a QUIC datagram contains a complete P frame or I frame fragment.

[0040] In step S20, the frame type of the frame data is an I frame or a P frame; the I frame is an internal frame used to provide data for the entire image, the frame type of the frame data of the first frame is an I frame, and only one I frame is sent for every ten frames of frame data.

[0041] Since the resolution of most images is relatively large, the amount of data they contain exceeds the payload size of the QUIC protocol (1232 bytes), so an I frame must be sent using multiple QUIC datagrams; the P frame only describes a small part of the changing data related to the I frame, which is small enough to be transmitted completely in one QUIC datagram; the frame type of frame data also includes B frames (bidirectional prediction frames), which are more compressed than P frames. Since the present invention focuses more on transmission delay rather than compression rate, the content of B frames is not included.

[0042] In step S20, after storing the sent QUIC datagram in the retransmission buffer, dependency information of the P frame carried in the QUIC datagram is also added; the dependency information is the I frame that the P frame depends on.

[0043] In step S20, the updating of the playback end timestamp and the sequence number is specifically as follows:

[0044] The server increases b when the playback end timestamp is called back, and accumulates the sequence number with a gradient of 1; the sequence number represents the total number of frame data that has been sent.

[0045] In step S20, before synchronizing the playback deadline timestamp to the client, the server adds a four-frame buffer frame to the playback deadline timestamp to allow the client to receive the retransmitted frame data: the offset will only increase when the playback deadline timestamp approaches, so the client waits for a maximum of 66ms to receive the lost frame data, and then increases the playback deadline timestamp and skips it. This waiting behavior is intended to improve the throughput of transmitting useful data without significantly affecting performance.

[0046] In step S30, the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response:

[0047] After receiving the reception confirmation, the server traverses the retransmission buffer based on the dependency information and the playback deadline timestamp to determine whether the corresponding QUIC datagram is still useful. If so, the corresponding QUIC datagram in the retransmission buffer is retained; if not, the corresponding QUIC datagram in the retransmission buffer is deleted. This process prevents old QUIC datagrams from lingering in the retransmission buffer, even if the client skips it without sending a reception confirmation, and reduces the amount of redundant data in the network by ensuring that only useful QUIC datagrams are retransmitted.

[0048] The process of the client receiving the QUIC datagram is specifically as follows: the client calls the read function at a frequency of 60 times per second, increases the playback deadline timestamp by b each time, and reads the QUIC datagram from the buffer of the protocol stack.

[0049] The step S40 is specifically as follows:

[0050] The client determines whether there is a missing data frame based on the frame number carried by the data frame. If not, the client plays the frame data carried by the QUIC datagram; if so, then:

[0051] Based on the expiration timestamp and adjacent data frames, the validity period of the missing data frame is checked. If it is expired, the missing data frame is skipped and the frame data of the next frame is played; if it is not expired, the data frame retransmitted by the buffer frame receiving server is played.

[0052] In summary, the advantages of the present invention are:

[0053] The server calculates the playback expiration timestamp of the frame data of the first frame of real-time multimedia after it arrives at the client, sends each frame data to the client in sequence through the QUIC datagram and caches it in the retransmission buffer, updates the playback expiration timestamp and sequence number, synchronizes the playback expiration timestamp to the client, and deletes the corresponding QUIC datagram in the retransmission buffer based on the reception confirmation response sent by the client; the client verifies the validity period of the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data; that is, the playback expiration timestamp is used to skip expired data frames to avoid the accumulated delay caused by the reliable transmission of the standard QUIC protocol, and by adding a four-frame buffer frame in the playback expiration timestamp to receive the data frames retransmitted by the server, that is, the lost data frames can be retransmitted and played before they expire, thereby improving the throughput of useful data and ensuring the playback quality of the real-time multimedia stream, ultimately greatly improving the reliability and real-time performance of real-time multimedia stream transmission.

[0054] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time multimedia streaming transmission method, characterized by: The steps include: Step S10: The server calculates a playback expiration timestamp after the first frame of the real-time multimedia arrives at the client. Step S20: The server sends each frame of the real-time multimedia data to the client in sequence via QUIC datagrams, stores the sent QUIC datagrams in a retransmission buffer, updates the playback expiration timestamp and sequence number, and synchronizes the playback expiration timestamp to the client; Step S30: The client feeds back a reception confirmation response to the server based on the received QUIC datagram, and the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response; Step S40: The client performs validity verification on the missing frame data in the received QUIC datagram based on the playback expiration timestamp, and then plays the corresponding frame data.

2. A real-time multimedia streaming transmission method according to claim 1, characterized in that: In step S10, the calculation formula of the playback end timestamp is: RT=((path_latency / a)*b)+c; Among them, RT represents the playback deadline timestamp; path_latency represents the delay of the network path; a represents the interval between the callback timers 60 times per second; b represents the time interval between each I frame; and c represents the playback buffer time of the buffered frame with four frames reserved.

3. The real-time multimedia streaming transmission method according to claim 1, wherein: In step S20, the frame type of the frame data is an I frame or a P frame; the I frame is an internal frame used to provide data for the entire image, the frame type of the frame data of the first frame is an I frame, and only one I frame is sent for every ten frames of frame data.

4. A real-time multimedia streaming transmission method according to claim 3, characterized in that: In step S20, after storing the sent QUIC datagram in the retransmission buffer, dependency information of the P frame carried in the QUIC datagram is also added; the dependency information is the I frame that the P frame depends on.

5. The real-time multimedia streaming transmission method according to claim 1, wherein: In step S20, the updating of the playback end timestamp and the sequence number is specifically as follows: The server increases b when the playback end timestamp is called back, and accumulates the sequence number with a gradient of 1; the sequence number represents the total number of frame data sent, and b represents the time interval between each I frame.

6. The real-time multimedia streaming transmission method according to claim 1, wherein: In the step S20, before synchronizing the playback deadline timestamp to the client, the server adds four buffer frames to the playback deadline timestamp.

7. A real-time multimedia streaming transmission method according to claim 4, characterized in that: In step S30, the server deletes the corresponding QUIC datagram in the retransmission buffer based on the received reception confirmation response: After the server receives the reception confirmation response, it determines whether the corresponding QUIC datagram is still useful based on the dependency information and the playback deadline timestamp. If so, it retains the corresponding QUIC datagram in the retransmission buffer; if not, it deletes the corresponding QUIC datagram in the retransmission buffer.

8. The real-time multimedia streaming transmission method according to claim 1, wherein: The step S40 is specifically as follows: The client determines whether there is a missing data frame based on the frame number carried by the data frame. If not, it plays the frame data carried by the QUIC datagram; if so, then: Based on the expiration timestamp and adjacent data frames, a validity period check is performed on the missing data frame; if the validity period is expired, the missing data frame is skipped and the frame data of the next frame is played; If it has not expired, the data frame retransmitted by the server is played through the buffer frame receiving server.

Images