A method and apparatus for determining a stall condition of a video transmission

Abstract

The application discloses a method for judging the pause state of video transmission, comprising the following steps: receiving a video frame, and putting the received video frame into a jitter buffer; obtaining the time index of the video frame to be processed, including a timestamp and a receiving time; defining a reference video frame, and obtaining the time index of the reference video frame; calculating the delay judgment index of the video frame to be processed; judging the receiving state of the video frame to be processed; and judging whether the video transmission is paused according to the timestamp, the receiving time and the delay judgment index of the video frame to be processed. According to the above technical scheme, the judgment accuracy of the pause state of the video transmission can be ensured under the condition that the timestamp interval of the video frame is not fixed.

Description

Technical Field

[0001] This invention relates to the field of streaming media technology, and more specifically, to a method and apparatus for determining the state of video transmission interruption. Background Technology

[0002] Streaming media technology is a technology that enables multimedia such as audio and video to be played in real time on the internet and intranet without downloading or waiting. It is indispensable for the implementation of current real-time communication. A video frame is the unit that makes up a video stream; video stream data is composed of individual video frames arranged sequentially.

[0003] During a video call, the video stream sent by the other party may be interrupted due to network fluctuations. In order to provide users with corresponding interface prompts based on the current interruption status of the video stream during the video call, so that users can understand the continuity of data stream transmission in a timely manner, it is necessary to "know" from the program's perspective whether the video transmission has been interrupted.

[0004] Existing methods for determining video stream pauses commonly rely on the frame playback interval. If a new frame doesn't play after a certain time following the previous one, the stream is considered paused. However, a video stream is actually composed of sequentially arranged video images, and the playback interval is determined by the timestamp interval between the images. The timestamps of two video frames are determined by their creation time, and the interval between any two frames can be relatively long or short. Therefore, judging whether a video stream is paused based on the frame playback interval is inaccurate. Summary of the Invention

[0005] To address the technical problem of inaccurate judgment of whether a video stream is paused, this invention provides a method and apparatus for judging the pause state of video transmission, ensuring accurate judgment of the pause state of video transmission even when the timestamp interval of video frames is not fixed.

[0006] Firstly, to achieve the above objective, this application provides a method for determining the state of video transmission pause, characterized by comprising the following steps:

[0007] Receive video frames and put the received video frames into the jitter buffer;

[0008] Obtain the time metrics of the video frames to be processed, including: timestamp and reception time;

[0009] The first decodeable and playable video frame (usually the first keyframe) is used as the reference video frame, and the timing index of the reference video frame is obtained.

[0010] Calculate the latency assessment index for the video frames to be processed;

[0011] Determine the reception status of the video frame to be processed;

[0012] The video transmission is determined to be interrupted based on the timestamp, reception time, and delay indicators of the video frame to be processed.

[0013] The latency assessment metric includes the estimated latest arrival time of the video frame to be processed relative to the reference frame, and its calculation method is as follows:

[0014] Estimated latest arrival time = Base frame arrival time + Timestamp interval + Maximum relative delay;

[0015] Relative delay = reception time interval - timestamp interval.

[0016] Furthermore, the receiving states of the video frames to be processed include: ready state and continuous state.

[0017] The readiness status determination refers to whether all RTP packets belonging to the video frame to be processed have been completely received;

[0018] The determination of continuity refers to whether the video frame to be processed is continuous with the latest played video frame.

[0019] If the video frame to be processed does not meet the ready or continuous state when it arrives at the estimated latest arrival time, the video transmission is determined to be in a paused state.

[0020] On the other hand, the present invention provides a device for determining the state of video transmission pause, comprising:

[0021] A receiving buffer device is used to receive video frames and put the received video frames into a jitter buffer.

[0022] An attribute acquisition device is used to acquire time indicators of the video frame to be processed, including: timestamp and reception time;

[0023] Logic control device: used to define a reference frame, calculate the delay judgment index of the video frame to be processed to determine the reception status of the video frame to be processed, and finally determine whether the video transmission is interrupted based on the timestamp, reception time and delay judgment index of the video frame to be processed.

[0024] The delay judgment index calculated by the logic control device includes the estimated latest arrival time of the video frame to be processed relative to the reference frame, and the calculation method is as follows:

[0025] Estimated latest arrival time = Base frame arrival time + Timestamp interval + Maximum relative delay;

[0026] Relative delay = reception time interval - timestamp interval.

[0027] The reception status of the video frame to be processed, as determined by the logic control device, includes: ready state and continuous state. The determination method for the ready state refers to whether all RTP packets belonging to the video frame to be processed have been completely received.

[0028] The method for determining continuity refers to whether the video frame to be processed is continuous with the latest played video frame.

[0029] If the video frame to be processed does not meet the ready state or continuous state when it arrives at the estimated latest arrival time, the logic control device can determine that the video transmission is in a paused state.

[0030] According to the present invention, it is not necessary to consider the pattern of timestamps generated by the sending end to generate video frames, nor is it necessary to adjust the criteria for judging delay for different processing intervals of the sending end, making the judgment of transmission delay more convenient and accurate. Attached Figure Description

[0031] Figure 1 This is a flowchart of a method for determining a video transmission pause state according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the timeline state 1 for video frame transmission and reception according to an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the timeline state 2 for video frame transmission and reception according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the timeline state 3 for video frame transmission and reception according to an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the timeline state 4 for video frame transmission and reception according to an embodiment of the present invention;

[0036] Figure 6 This is a structural diagram of a device for determining the video transmission pause state based on a video frame according to an embodiment of the present invention. Detailed Implementation

[0037] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0038] The method for determining the pause status of video frame transmission provided by this invention can use the relative characteristics of time attributes to determine whether video transmission is paused, without ignoring the time interval rules for video frame generation, and provide network status prompts to the user.

[0039] Figure 1This is a flowchart of a method for determining the pause state of video frame transmission according to an embodiment of the present invention. The method involved in the present invention is executed at the video receiving end, and as shown in the figure, includes the following steps:

[0040] Step S100: Receive video frames and put the received video frames into the jitter buffer.

[0041] The sending end transmits the captured video frames sequentially to the receiving end. Since video frames are generally large, the data for one frame is split into multiple network data packets before transmission. Upon receiving these data packets, the receiving end needs to identify which packets belong to a single video frame. Once all packets belonging to the same video frame have been received, the complete data of that video frame can be reconstructed, followed by decoding and playback. Before playback, the receiving end places the data packets into a jitter buffer. After playback is complete, the video frame is removed from the jitter buffer. The video frame processing described in this application—the process of recovering and playing back a video frame after receiving its data packet—includes packet assembly, decoding, and playback.

[0042] Step S111: Obtain the time metrics of the video frame to be processed from the jitter buffer, including: timestamp and reception time;

[0043] Before a video frame is sent from the sending end, its generation time is recorded, called a timestamp. At the receiving end, a time is also recorded when all data packets of a video frame are received, called the reception time. Due to network fluctuations, the interval between the timestamps of two video frames and the interval between their reception times are generally not the same; the reception time interval may be longer or shorter than the timestamp interval.

[0044] Figure 2 This explains the state 1 of the timeline for video frame transmission and reception: a2-a1 is greater than t2-t1, which means that video frame 2 arrived with a delay relative to video frame 1, indicating that the network latency has increased at this time;

[0045] exist Figure 3 This describes state 2 of the timeline for sending and receiving video frames. Since a2-a1 is less than t2-t1, it means that video frame 2 arrived "earlier" than video frame 1. This indicates that the network latency has decreased. Previously, the network latency was high, causing video frame 1 to arrive late.

[0046] Step S112: Define a reference video frame and obtain the timing metrics of the reference video frame;

[0047] To calculate the relative delay of each frame, a reference video frame needs to be determined, and the reference values ​​of timestamp and reception time need to be obtained. This reference frame is generally defined as the first decodeable and playable video frame received (usually the first keyframe).

[0048] Step S122: Calculate the latency judgment index of the video frame to be played;

[0049] Using the time index of the baseline frame as a standard, the relative latency is calculated. Then, the maximum relative latency of all frames within a recent period is calculated and used as an estimate of the upper limit of the relative latency to be reached by subsequent frames. The calculation method is as follows:

[0050] Relative delay = reception time interval - timestamp interval;

[0051] Estimated latest arrival time = Base frame arrival time + Timestamp interval + Maximum relative delay.

[0052] Figure 4 This illustrates state 3 of the timeline for video frame transmission and reception: the estimated latest arrival time and actual arrival time of video frames when the video stream is not interrupted, as shown in the figure.

[0053] a ’ n =a1+(t) n -t1)+Max{(a i -a1)-(t i -t1)}

[0054] Max{(a i -a1)-(t i `-t1)}` represents the maximum relative arrival delay of all frames received within a recent period relative to the first frame.

[0055] Playing video frames based on the estimated latest arrival time ensures that most frames have arrived by the time they are needed for playback.

[0056] Step S121: Determine the reception status of the video frame to be played;

[0057] After obtaining the estimated latest arrival time, the reception status of the video frame needs to be evaluated. The reception status includes "ready state" and "continuous state". The former determines whether the data packets of the video frame have been received completely, and the latter determines whether the sequence number of the video frame is continuous with the sequence number of the latest played video frame.

[0058] Step S130: Determine if video transmission is interrupted based on the timestamp, reception time, and delay indicators of the video frames to be processed. If a video frame arrives at the "estimated latest arrival time" before forming a complete frame (i.e., not in a continuous state), or if the oldest frame is complete but not continuous with the previous playback frame, or if there are several frames in between (i.e., not in a continuous state), then continuous video playback cannot be guaranteed, and an interruption will occur. Upon determining that an interruption has occurred, a notification mechanism can be activated to inform the user of the current network status.

[0059] Figure 5 The following is a timeline of video frame transmission and reception: State 4: When the video stream is interrupted, the estimated latest arrival time of the video frame is compared with the actual arrival time (it is also possible that the video frame will not arrive). In this case, the user needs to be notified that the network transmission has been interrupted.

[0060] The method for determining the pause state of video transmission provided by this invention uses a fixed time reference to determine the relative time attribute. With this method, there is no requirement for uniformity in the rules for the timestamps of the video frames generated by the sending end, nor is it necessary to adjust the judgment delay standard for different processing intervals of the sending end. Therefore, the method provided by this application has a wider range of applications and is more accurate in judgment.

[0061] Figure 6 The figure shows a structural diagram of a device for determining the video transmission pause state according to an embodiment of the present invention. As shown in the figure, the device of the present invention includes:

[0062] The P600 receiving buffer is used to receive video frames, put the received video frames into the P601 jitter buffer, and remove the video frames to the P602 removal buffer after playback.

[0063] The P610 attribute acquisition device is used to acquire time indicators of the video frame to be processed, including: timestamp and reception time.

[0064] P620 logic control device: used to define a reference frame, calculate the delay judgment index of the video frame to be processed to determine the reception status of the video frame to be processed, and finally determine whether the video transmission is interrupted based on the timestamp, reception time, and delay judgment index of the video frame to be processed.

[0065] The delay judgment index calculated by the P620 logic control device includes the estimated latest arrival time of the video frame to be processed relative to the reference frame, and the calculation method is as follows:

[0066] Estimated latest arrival time = Base frame arrival time + Timestamp interval + Maximum relative delay;

[0067] Wherein, relative delay = receiving time interval - timestamp interval.

[0068] The P620 logic control device determines the reception status of the video frame to be processed, including: ready state and continuous state.

[0069] The ready state is determined by whether all RTP packets belonging to the video frame to be processed have been completely received.

[0070] In the actual implementation, the frame header and frame tail are identified and distinguished:

[0071] Each RTP packet at the end of a frame has a flag in its RTP header. This flag can be used to identify the end of the frame. Different strategies are used to distinguish between keyframes and non-keyframes: For keyframes, the first packet always contains key information such as SPS or PPS. If a packet containing this key information is received, the frame header is considered received. For non-keyframes, the sequence number of the end-of-frame packets is used for deduction. Adding one to the sequence numbers of all received end-of-frame packets gives the sequence numbers of all frame header packets. If a packet's sequence number belongs to one of these header sequence numbers, then that packet is a frame header. This method can also be used to determine the continuity of video frames.

[0072] If all packets between the frame header and frame tail have been received, then all data packets of that frame have been received completely. If the sequence number of the frame header is consecutive to the sequence number of the frame tail of the processed frames, then the reception of that video frame is continuous.

[0073] If a video frame to be processed does not meet the ready state when the estimated latest arrival time arrives, the video transmission is determined to be in a paused state; similarly, if a video frame to be processed does not meet the continuous state when the estimated latest arrival time arrives, the video transmission is determined to be in a paused state.

[0074] The method and apparatus for determining the pause status of video frame transmission provided by this invention can determine whether video transmission is paused by using the relative characteristics of time attributes, without ignoring the time interval rules for the generation of video frames. Through the invention and apparatus of this application, there are no strict requirements for the time control of the sending end, so its application is more extensive and flexible, and the network status prompts to users are more accurate.

[0075] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for determining the pause state of video transmission, characterized in that, Includes the following steps: Receive video frames and put the received video frames into the jitter buffer; Obtain the time metrics of the video frames to be processed, including: timestamp and reception time; The first decodeable and playable video frame received is used as the reference video frame, and the timing index of the reference video frame is obtained. Calculate the latency judgment index of the video frame to be processed; wherein, the latency judgment index includes the estimated latest arrival time of the video frame to be processed relative to the reference video frame, and the estimated latest arrival time is calculated as follows: estimated latest arrival time = arrival time of reference video frame + timestamp interval + maximum relative delay; wherein, relative delay = reception time interval - timestamp interval; the maximum relative delay is the maximum relative delay of all video frames received in a recent period, which is used as an estimate of the upper limit of the relative delay of subsequent video frames. Determine the reception status of the video frame to be processed; the reception status of the video frame to be processed includes a ready state and a continuous state; the ready state means that all RTP packets belonging to the video frame to be processed have been completely received, and the continuous state means that the video frame to be processed is continuous with the latest played video frame; Determining whether video transmission is interrupted based on the timestamp, reception time, and delay judgment index of the video frame to be processed includes: if the video frame to be processed does not meet the ready state or the continuous state when the estimated latest arrival time arrives, then it is determined that the video transmission is in an interrupted state; after determining that an interruption has occurred, a prompting mechanism is activated to notify the user of the current network status.

2. A device for determining the pause state of video transmission, characterized in that, include: A receiving buffer device is used to receive video frames and put the received video frames into a jitter buffer. An attribute acquisition device is used to acquire time indicators of the video frame to be processed, including: timestamp and reception time; Logic control device: used to take the first decodeable and playable video frame received as the reference video frame, calculate the delay judgment index of the video frame to be processed, wherein the delay judgment index includes the estimated latest arrival time of the video frame to be processed relative to the reference video frame. The estimated latest arrival time is calculated as follows: estimated latest arrival time = arrival time of reference video frame + timestamp interval + maximum relative delay, relative delay = receiving time interval - timestamp interval, and the maximum relative delay is the maximum relative delay of all video frames received in a recent period, which is used as an estimate of the upper limit of the relative delay of subsequent video frames. Determine the reception status of the video frame to be processed. The reception status includes a ready state and a continuous state. The ready state means that all RTP packets belonging to the video frame to be processed have been completely received. The continuous state means that the video frame to be processed is continuous with the latest played video frame. Based on the timestamp, reception time, and delay judgment index of the video frame to be processed, it is determined whether the video transmission is interrupted. If the video frame to be processed does not meet the ready state or the continuous state when it arrives at the estimated latest arrival time, it is determined that the video transmission is in an interrupted state. After the interruption is determined, a prompting mechanism is activated to notify the user of the current network status.

Images