A real-time video pull streaming method and system in a video distribution scenario

Abstract

A real-time video pull streaming method and system in a video distribution scene, wherein the real-time video pull streaming method comprises: 1, the server continuously maintains a buffer; 2, the server analyzes the I frame request of the entry, if the reference relationship of the cache sequence dependency chain is complete, enter S3, otherwise enter S5; 3, the current buffer size and the time difference value are calculated to construct the GOP sequence sending condition; if the condition is met, enter S4, otherwise enter S5; 4, the latest GOP sequence is sent to the new user, and the cache frame and the end cache frame are identified in the RTP packet extension header carrying the cache frame load; 5, the server forwards the I frame request of the new user to the sending end. The application can minimize the first frame delay of the new user, avoid unnecessary I frame refresh as much as possible, reduce the uplink bandwidth consumption of the sending end and the interference to other users, and optimize the server resource utilization.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a real-time video streaming method and system for video distribution scenarios. Background Technology

[0002] In multi-party real-time video scenarios, when a new user joins, the video stream needs to be quickly retrieved and decoded to synchronize the video feed. Traditional solutions typically have the following drawbacks:

[0003] (a) When a new user joins, if the first video frame they receive is a reference frame, they need to wait for the next I-frame, resulting in a long loading time for the first frame;

[0004] (b) To shorten the first frame loading time, one existing solution is for a new user or server to actively request the sender to immediately refresh an I-frame. However, frequently generating and sending additional I-frames significantly increases the sender's uplink bandwidth overhead. More importantly, this new I-frame will be sent to all existing clients, and the negative impacts such as bandwidth overhead will be introduced to other clients;

[0005] (c) Another approach is for the server to continuously cache complete GOP (Group of Pictures, which begins with an I-frame and contains several subsequent reference frames) sequences. When a new user joins, the cached GOP is sent directly. This will put pressure on the server's caching and on the client's instantaneous decoding. Summary of the Invention

[0006] This invention provides a real-time video streaming method and system for video distribution scenarios to solve the technical problems mentioned in the background.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0008] This invention provides a real-time video streaming method for video distribution scenarios, comprising the following steps:

[0009] S1. The server continuously maintains a buffer, overwriting the most recent GOP sequence. Starting with an I-frame, it caches a total of N frames of a finite length. During the process of caching the latest GOP sequence, it verifies the integrity of the inter-frame reference relationship in the newly added dependency chain.

[0010] S2. When a new user joins the membership, they send an I-frame request to the server. The server parses the new user's I-frame request. If the reference relationship between frames in the dependency chain is complete, it proceeds to S3; otherwise, it proceeds to S5.

[0011] S3. Calculate the current buffer size n and the time difference Δt between the current time and the arrival time of the last I-frame; construct the GOP sequence transmission conditions based on the current buffer size n and the time difference Δt; if the GOP sequence transmission conditions are met, proceed to S4, otherwise proceed to S5.

[0012] S4. Send the latest GOP sequence to the new user and identify the buffered frame and the end buffered frame in the RTP packet extension header carrying the buffered frame payload. The buffered frame identifier in the RTP extension header will be parsed by the receiving end to distinguish between the buffered frame, the end buffered frame and the server timely forwarding frame.

[0013] S5. The server forwards the new user's I-frame request to the sending end, and the sending end forces the generation of an I-frame.

[0014] Furthermore, the GOP sequence includes I-frames and subsequent reference frames.

[0015] Furthermore, step S1 specifically includes the following steps:

[0016] S11. The server receives a frame of data and determines whether the frame is an I-frame, where an I-frame is the first frame of each GOP sequence; if it is an I-frame, proceed to S12; otherwise, proceed to S13.

[0017] S12, release the buffer occupied by the original GOP sequence, and then proceed to S14;

[0018] S13. Determine if the length of the original GOP sequence is greater than N-1, where N represents the number of buffered frames of the GOP sequence; if so, release the buffer occupied by the original GOP sequence and proceed to S11; otherwise, proceed directly to S14.

[0019] S14. Store the data frame received in S11 into the latest GOP sequence;

[0020] S15, and during the caching of the latest GOP sequence, verify the integrity of the reference relationships between frames in the newly added dependency chain.

[0021] Furthermore, the specific formula for calculating N is as follows:

[0022] ;

[0023] Where 'a' represents the coefficient and 'GOP_len' represents the length of the GOP sequence.

[0024] Furthermore, the GOP sequence transmission condition in S3 is that the current buffer size n is less than or equal to the number of buffered frames N, and the time difference Δt is less than or equal to a preset time threshold T. The time threshold T is a limitation on the effective time of the buffer, such as one or several sending end GOP cycles.

[0025] In another aspect, the present invention provides a real-time streaming system for video distribution scenarios, including a device end, wherein the device end is configured or executes a real-time video streaming method.

[0026] The beneficial effects of this invention are:

[0027] This invention discloses a real-time video streaming method for video distribution scenarios, which can minimize the first frame delay for new users while avoiding unnecessary I-frame refreshes as much as possible, thereby reducing uplink bandwidth consumption at the sending end and interference to other users, and optimizing the utilization of server cache resources. Attached Figure Description

[0028] Figure 1 This is a flowchart of the real-time video streaming method in this invention. Detailed Implementation

[0029] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0030] Reference Figure 1 This application provides a real-time video streaming method for video distribution scenarios, including the following steps:

[0031] S1. The server continuously maintains a buffer, overwriting the most recent GOP sequence. Starting with an I-frame, it caches a finite length of N frames. During the caching of the latest GOP sequence, it verifies the integrity of the inter-frame reference relationships in the newly added dependency chain. The I-frame is a key frame, serving as a dependency for decoding subsequent reference frames.

[0032] S2. When a new user joins the membership, they send an I-frame request to the server. The server parses the new user's I-frame request. If the reference relationship between frames in the dependency chain is complete, it proceeds to S3; otherwise, it proceeds to S5.

[0033] S3. Calculate the current buffer size n and the time difference Δt between the current time and the arrival time of the last I-frame; construct the GOP sequence transmission conditions based on the current buffer size n and the time difference Δt; if the GOP sequence transmission conditions are met, proceed to S4, otherwise proceed to S5.

[0034] S4. Send the latest GOP sequence to new users and identify buffered frames and end-of-buffered frames in the RTP (Real-Time Transport Protocol) packet extension header carrying buffered frame payload. Buffered frames and server-forwarded frames may arrive at the receiving end out of order, requiring packet buffer rearrangement. The buffered frame identifier in the RTP extension header is parsed by the receiving end to distinguish between buffered frames, end-of-buffered frames, and server-forwarded frames. The receiving end's video decoding follows a complete decoding logic. After decoding, buffered frames are discarded, while end-of-buffered frames and timely forwarded frames normally enter the jitter buffer, which determines the frame rendering time.

[0035] S5. The server forwards the new user's I-frame request to the sending end, and the sending end forces the generation of an I-frame to achieve I-frame refresh.

[0036] In some embodiments, S1 specifically includes the following steps:

[0037] S11. The server receives a frame of data and determines whether the frame is an I-frame, where an I-frame is the first frame of each GOP sequence; if it is an I-frame, proceed to S12; otherwise, proceed to S13.

[0038] S12, release the buffer occupied by the original GOP sequence, and then proceed to S14;

[0039] S13. Determine if the length of the original GOP sequence is greater than N-1, where N represents the number of buffered frames of the GOP sequence; if so, release the buffer occupied by the original GOP sequence and proceed to S11; otherwise, proceed directly to S14.

[0040] S14. Store the data frame received in S11 into the latest GOP sequence;

[0041] S15, and during the caching of the latest GOP sequence, verify the integrity of the reference relationships between frames in the newly added dependency chain.

[0042] In some embodiments, the formula for calculating N is as follows:

[0043] ;

[0044] Where GOP_len represents the length of the GOP sequence, and a represents the coefficient, for example, a is 0.6. The lower limit of the value of a is to prevent frequent refresh of I-frames, and the upper limit is to avoid excessively long buffers, and a must be less than 1.

[0045] In some embodiments, the GOP sequence transmission condition in S3 is that the current buffer size n is less than or equal to the number of buffered frames N, and the time difference Δt is less than or equal to a preset time threshold T. The time threshold T is a limitation on the effective time of the buffer, such as one or several GOP sequence periods of the transmitting end.

[0046] This invention discloses a real-time video streaming method for video distribution scenarios, which can minimize the first frame delay for new users while avoiding unnecessary I-frame refreshes as much as possible, thereby reducing uplink bandwidth consumption at the sending end and interference to other users, and optimizing the utilization of server cache resources.

[0047] In another aspect, the present invention provides a real-time streaming system for video distribution scenarios, including a device end, wherein the device end is configured or executes a real-time video streaming method.

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time video pull streaming method in a video distribution scenario, characterized in that, The method comprises the following steps: S1, the server continuously maintains a buffer, covers and stores a latest GOP sequence, starts with an I frame, buffers a limited length N frames in total, and verifies the integrity of the inter-frame reference relationship in the newly added dependency chain in the process of buffering the latest GOP sequence; S2, a new user sends an I frame request to the server, the server analyzes the I frame request of the new user, if the inter-frame reference relationship in the dependency chain is complete, it enters S3, otherwise it enters S5; S3, calculate the current buffer size n and the time difference Δt between the current time and the time when the last I frame arrives; build a GOP sequence sending condition according to the current buffer size n and the time difference Δt; if the GOP sequence sending condition is met, that is, the current buffer size n is less than or equal to the buffer frame number N, and the time difference Δt is less than or equal to the pre-set time threshold T, then enter S4, otherwise enter S5; S4, send the latest GOP sequence to the new user, and mark the buffer frame and the last buffer frame in the RTP packet extension header carrying the buffer frame load; the buffer frame identifier in the RTP extension header will be parsed by the receiving end, to distinguish the buffer frame, the last buffer frame and the server forwarding frame; the receiving end video decoding follows the "decode as much as possible" logic, and the decoded buffer frame will be discarded, and the last buffer frame and the forwarding frame will enter the jitter buffer normally, and the frame rendering time is decided by the jitter buffer; S5, the server forwards the I frame request of the new user to the sending end, and the sending end forces to generate an I frame.

2. The real-time video pull streaming method in a video distribution scenario according to claim 1, characterized in that, The GOP sequence comprises an I frame and subsequent reference frames. 3.The real-time video pull streaming method in a video distribution scenario according to claim 1, characterized in that, The S1 specifically comprises the following steps: S11, the server receives a frame of data, and judges whether the frame of data is an I frame; if it is an I frame, it enters S12, otherwise it enters S13; S12, release the buffer occupied by the original GOP sequence, and then enter S14; S13, judge whether the length of the original GOP sequence is greater than N-1, wherein N represents the buffer frame number of the GOP sequence; if yes, release the buffer occupied by the original GOP sequence, and enter S11, otherwise directly enter S14; S14, store the frame of data received in S11 into the latest GOP sequence; S15, and verify the integrity of the inter-frame reference relationship in the newly added dependency chain in the process of buffering the latest GOP sequence.

4. The real-time video pull streaming method in a video distribution scenario according to claim 1, characterized in that, The calculation formula of N is specifically as follows: ； Wherein, a represents a coefficient; GOP_len represents the GOP sequence length.

5. A real-time pull streaming system in a video distribution scenario, characterized in that, The device end is configured or executes the real-time video pull streaming method according to any one of claims 1 to 4.

Images