Packet loss processing method and device, equipment, storage medium and computer program product

Abstract

The invention provides a packet loss processing method and device, equipment, a storage medium and a computer program product, and relates to the technical field of wireless communication, and the method comprises the steps: receiving a video data stream; the video data stream comprises at least one I frame and a plurality of non-I frames, and each frame is associated with a timestamp; the I frame is a video frame which can be independently decoded, and the non-I frame is a video frame which needs to be decoded by referring to other frames; under the condition that discontinuous hopping of the timestamp is detected, it is judged that I-frame packet loss occurs; and decoding the continuous video frame data received before the discontinuous jump point of the timestamp, and discarding the video frame data received after the discontinuous jump point of the timestamp. According to the invention, when the I-frame packet is lost, the packet before the timestamp is changed is used as the end frame to be sent to the upper layer for decoding processing, and other subsequent packets are discarded, so that the fluency of the picture can be ensured, the blurred screen is reduced, and the delay time is shortened, thereby improving the service experience of video broadcasting.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a packet loss processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] 3GPP MBS (Multicast / Broadcast Service) broadcast services employ point-to-multipoint unidirectional downlink broadcast technology, which does not support L3 layer TCP (Transmission Control Protocol) reliable connection establishment, L2 layer retransmission, or mobility handover. Because the network side adopts a "throw it out and forget it, best-effort" propagation mechanism, in continuous coverage networks composed of small cells, when terminals are in weak coverage areas or cell boundaries, packet loss or missed reception of some PDCP (Packet Data Convergence Protocol) packets may occur due to high BLER (Block Error Rate). Therefore, how to implement packet loss handling has become an urgent problem to solve. Summary of the Invention

[0003] This invention provides a packet loss processing method, apparatus, device, storage medium, and computer program product to address the deficiencies in packet loss processing in the prior art, thereby improving the smoothness of the video, reducing screen tearing, and shortening the duration of lag, thus enhancing the service experience of video broadcasting.

[0004] This invention provides a packet loss handling method, comprising the following steps: Receive a video data stream; the video data stream contains at least one I-frame and multiple non-I-frames, each frame is associated with a timestamp; the I-frame is a video frame that can be decoded independently, and the non-I-frames are video frames that need to be decoded by referring to other frames. If a discontinuous jump in the timestamp is detected, it is determined that an I-frame packet loss has occurred; The continuous video frame data received before the discontinuous timestamp transition point is decoded, and the video frame data received after the discontinuous timestamp transition point is discarded.

[0005] According to a packet loss handling method provided by the present invention, determining that I-frame packet loss has occurred when a discontinuous change in the timestamp is detected includes: Cache the current I-frame in the video data stream, as well as the consecutive non-I-frames following the current I-frame, until the next I-frame is detected; If a discontinuous jump in the timestamp of a non-I-frame is detected between two I-frames, it is determined that an I-frame packet loss has occurred.

[0006] According to a packet loss processing method provided by the present invention, the step of decoding the continuous video frame data received before the timestamp discontinuity transition point and discarding the video frame data received after the timestamp discontinuity transition point includes: The current I-frame and the non-I-frames between the current I-frame and the discontinuous timestamp transition point are combined into an image group and sent to the decoder for decoding processing; Discard the non-I-frames between the discontinuous timestamp transition point and the next I-frame.

[0007] According to a packet loss handling method provided by the present invention, the method further includes: If the first Real-Time Transport Protocol (RTP) packet corresponding to the video data stream is a fragmented data packet, the timestamp of the first RTP packet will be used as the base timestamp. The timestamps of subsequently received RTP packets are compared with the reference timestamp. If the timestamps are inconsistent and no end-of-frame marker is detected, a packet loss event is determined to have occurred. If the packet loss event is an I-frame packet loss, then the RTP packets before the timestamp discontinuity transition point are decoded and the RTP packets after the timestamp discontinuity transition point are discarded until a complete I-frame fragment sequence is received again. If the packet loss event is a non-I-frame packet loss, then the entire non-I-frame fragment sequence is discarded.

[0008] According to a packet loss handling method provided by the present invention, the I-frame is identified based on the following method: Read the Service Data Unit (SDU) from the RTP packet corresponding to the video data stream; The binary code stream of the SDU is parsed to locate the starting position of the Network Abstraction Layer Unit (NALU) according to the preset start code; Determine the NALU frame header based on the starting position of the NALU; The I-frame is identified based on the type field value in the NALU frame header.

[0009] According to a packet loss handling method provided by the present invention, the method further includes: If a discontinuous jump in the timestamp is detected, it is determined that a non-I-frame packet loss has occurred; Discard non-I-frames following the discontinuous timestamp transition point.

[0010] The present invention also provides a packet loss processing device, comprising the following modules: A receiving module is used to receive a video data stream; the video data stream contains at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp; the I-frame is a video frame that can be decoded independently, and the non-I-frames are video frames that need to be decoded by referring to other frames. The packet loss determination module is used to determine that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected. The packet loss processing module is used to decode the continuous video frame data received before the discontinuous timestamp transition point and discard the video frame data received after the discontinuous timestamp transition point.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the packet loss processing methods described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the packet loss processing method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the packet loss processing methods described above.

[0014] The packet loss processing method, apparatus, device, storage medium, and computer program product provided by this invention receive a video data stream. The video data stream contains at least one I-frame and multiple non-I-frames, each frame associated with a timestamp. I-frames are video frames that can be decoded independently, while non-I-frames are video frames that require reference to other frames for decoding. When a discontinuous timestamp transition is detected, I-frame packet loss is determined. The continuous video frame data received before the discontinuous timestamp transition point is decoded, and video frame data received after the discontinuous timestamp transition point is discarded. This invention, when I-frame packet loss occurs, sends the packet before the timestamp change as the end frame to the upper layer for decoding and discards subsequent packets, ensuring smooth playback, reducing screen tearing, and decreasing stuttering time, thereby improving the service experience of video broadcasting. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1This is a flowchart illustrating the packet loss handling method provided by the present invention.

[0017] Figure 2 This is a flowchart illustrating the I-frame packet loss identification method provided by the present invention.

[0018] Figure 3 This is a flowchart illustrating the ultra-large frame end recognition method provided by the present invention.

[0019] Figure 4 This is a flowchart illustrating the I-frame recognition method provided by the present invention.

[0020] Figure 5 This is a schematic diagram of the packet loss processing device provided by the present invention.

[0021] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] In related technologies, broadcast service terminals are in an RRC (Radio Resource Control) idle state, without a dedicated bearer established with the network, and the network does not assign IP addresses to the terminals. There is no IP bearer between the terminals and the AF (Application Function). Broadcast services employ a layer-by-layer, copy-and-forward, best-effort unidirectional downlink "blind" broadcasting on the transmission network. That is, the H.264 video source is encapsulated in RTP (Real-time Transport Protocol) format, and an IP / UDP (User Datagram Protocol) link is established between the AF and MB-UPF (Multicast Broadcast-User Plane Function) to broadcast to multiple MB-UPFs via a common GTP-U (GPRS Tunneling Protocol-User Plane) tunnel (identified by Common TEID). The MB-UFP (Multicast Broadcast-User Forwarding Plane) further uses the same method to copy the RTP stream to the relevant base station. The base station then uses an L3-free scheme to statically and continuously transmit multicast control channels and multicast transmission channels to the specified time and frequency positions on the air interface. In idle state, the terminal selects a specific cell and obtains the MCCH (Multicast Control Channel) channel configuration and frequency selection area list by demodulating the cell's system broadcast messages SIB20 (System Information Block 20, containing multicast control channel configuration information) and SIB21 (System Information Block 21, containing a frequency point list of the frequency selection area). It then locks onto a specific frequency point, reads the MTCH (Multicast Traffic Channel) channel configuration from the MCCH channel, demodulates the MTCH channel to receive broadcast PDCP (Packet Data Convergence Protocol) packets, concatenates them into an RTP PDU (Real-time Transport Protocol Protocol Data Unit), decapsulates it to obtain an H.246 NALU (Network Abstraction Layer Unit), and sends it to the application layer player.From the perspective of network transmission characteristics of broadcast services, the network has no knowledge of the terminal's status and has not set up any mechanism to ensure reliable packet transmission. Whether a packet can be received and the handling scheme for received error packets depend entirely on the terminal.

[0024] Based on the above mechanism description, to improve the video playback quality of broadcast services, this invention provides a packet loss handling method. This method optimizes the process from the terminal side by: firstly, combining the H.264 encoding mechanism to optimize the RTP stream reception method, enabling parallel stream reception and decoding, and accelerating the push to the application layer; secondly, real-time detection of I-frames to reduce I-frame dropout and improve the decoding capability of the application layer. By optimizing the terminal's stream reception and decoding methods, real-time decoding is improved, and the I-frame drop rate is reduced, which is beneficial to improving the service experience of video broadcasting.

[0025] Related technologies rely on the premise of stable and reliable transmission at the transport layer. However, in broadcast services, unreliable transmission and short-term interruptions are common. This invention addresses this issue by optimizing decoding technology under adverse channel conditions. It achieves this by rapidly identifying and retaining decodeable packets while quickly discarding undecodeable ones, thereby preventing error accumulation, memory overflow, and high processor load.

[0026] The packet loss handling method is optimized mainly from the following two aspects: (1) Optimize video playback, quickly skip dropped and misaligned frames, and reduce error propagation and stuttering duration. It should be understood that in broadcast services, if the terminal loses an I-frame RTP packet due to high BLER, subsequent P-frames within the same GOP (Group of Pictures) will fail to decode due to the lack of a reference frame. Therefore, when decapsulating the RTP stream on the terminal side, for example, after detecting the loss of an I-frame, all P-frames in the current GOP are directly discarded (until the next I-frame arrives), replacing screen tearing with a brief pause (freeze) to improve the subjective experience.

[0027] (2) When the RTP stream header Markbit is set to 1 or the TimeStamp is changed, it is considered that an RTP stream has ended and is immediately sent to the player for decoding and playback to avoid the loss of the next group of I frames, which would cause stuttering and prolong the playback.

[0028] It should be understood that packet loss handling methods can be applied to both broadcast and multicast services.

[0029] The following is combined Figures 1-6 The present invention describes a packet loss handling method, apparatus, device, storage medium, and computer program product.

[0030] Figure 1 This is a flowchart illustrating the packet loss handling method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Receive video data stream.

[0031] A video data stream is a continuous data sequence consisting of a series of encoded video frames, including: At least one I-frame (Intra-coded Frame); a video frame that can be decoded independently, and whose decoding process does not depend on other video frames; Multiple non-I-frames, including forward predictive coding P-frames and / or bidirectional predictive coding B-frames: video frames that need to be decoded with reference to other frames; Each frame is associated with a timestamp: used to identify the display sequence of the frames to ensure that video frames are correctly processed in chronological order during decoding and playback.

[0032] It should be understood that an I-frame can be an IDR (Instantaneous Decoding Refresh) frame in the H.264 standard. An I-frame is a keyframe, a complete image frame in a video sequence. During video encoding, video data is compressed and encoded in a specific order and manner. An I-frame is an independent and complete frame within this encoded sequence, containing all the information of that frame's image. When a decoder receives I-frame data, it can reconstruct the image of that frame entirely based on the data carried by the I-frame itself, without relying on information from other frames. For example, during video playback, if there is a brief network interruption or data loss, and the next received frame is an I-frame, the player can reconstruct the image from this I-frame without being affected by previously lost data.

[0033] The base station encapsulates the video stream into RTP packets and transmits them via a 5G broadcast channel (such as 5G NR MBS) or an LTE eMBMS channel. The terminal receives the video data stream transmitted by the base station; for example, the terminal receives the RTP packets corresponding to the video data stream.

[0034] Step 102: If a discontinuous jump in the timestamp is detected, it is determined that an I-frame packet loss has occurred.

[0035] It should be understood that in video streaming, timestamps are numerical values ​​that mark the timing of frame display. Discontinuous timestamp jumps refer to: a sudden decrease in the timestamp size (e.g., jumping from 100ms to 20ms), or abnormal timestamp increments (e.g., a jump of 100ms when the interval between consecutive frames should be 33ms). This indicates frame loss or out-of-order delivery, especially in scenarios without retransmission, such as broadcast / multicast.

[0036] The timestamps of consecutive video frames should increase uniformly (e.g., +33ms per frame). If the timestamps suddenly regress or the increment is abnormal, it may be due to packet loss causing a break in decoding dependencies. For example, calculate the difference between the timestamps of the current packet and the previous packet: if the difference is approximately equal to the frame interval (e.g., 33ms@30fps), it is normal; if the difference is much greater than expected (e.g., 100ms), or the timestamps regress, it indicates packet loss.

[0037] Alternatively, packet loss can be determined using the RTP sequence number. It should be understood that each RTP packet has an incrementing sequence number (e.g., 1, 2, 3...). If a sequence number jumps (e.g., receiving 1 and 3, but missing 2), packet loss is determined. For example, record the sequence number of the previous packet (e.g., seq=1). When a new packet is received, check: if seq=2 (continuously incrementing), it's normal; if seq=3 (missing 2), packet loss occurs.

[0038] Optionally, if the sequence number is not consecutive and the timestamp changes, then packet loss is confirmed.

[0039] For example, the RTP packet corresponding to the video data stream is: RTP packet 1: [seq=1][timestamp=1000][NALU Type=5 (I-frame)]; RTP packet 2: [seq=2][timestamp=1033][NALU Type=1 (P-frame)]; RTP packet 3: [seq=3][timestamp=1066][NALU Type=1 (P-frame)]; RTP packet 4: [seq=4][timestamp=1100][NALU Type=5 (I-frame)]; Normal situation: Received seq=1→2→3→4, timestamp 1000→1033→1066→1100, no packet loss.

[0040] Packet loss scenarios: 1) Missing P-frame: Received seq=1→3 (missing 2), timestamp 1000→1066 (transition 33ms→66ms), determined: P-frame seq=2 is lost. 2) Missing I-frame: Received seq=2→3→4 (missing 1), timestamp 1033→1066→1100, determined: I-frame seq=1 is lost.

[0041] Step 103: Decode the continuous video frame data received before the discontinuous timestamp transition point, and discard the video frame data received after the discontinuous timestamp transition point.

[0042] It should be understood that if an I-frame is lost, subsequent P-frames / B-frames that depend on it cannot be decoded correctly (resulting in screen tearing / pixelation).

[0043] It should be understood that a timestamp discontinuity jump point refers to a point in the video stream where the timestamps suddenly change discontinuously. This jump indicates that there is data loss or error in the video stream, especially in the case of missing keyframes (I-frames).

[0044] When an I-frame is lost, the continuous video frame data received before the time stamp discontinuity transition point is sent to the upper layer as the end frame for decoding, and the video frame data received after the time stamp discontinuity transition point is discarded.

[0045] For example, suppose the received frame sequence and timestamps are as follows: Frame type: I, P, P, P, P, P, P, P, I...; Timestamps: 0, 10, 20, 30, 40, 10, 20, 30, 40...; Detection of timestamp discontinuity jump points: The timestamp of the 5th P-frame jumps from 40 back to 10, which is a discontinuity jump point. This jump indicates that the 2nd I-frame has been lost, because timestamp discontinuity is usually caused by the loss of I-frames.

[0046] Processing method: The continuous video frame data (I+P1~P4) received before the discontinuous timestamp transition point is sent to the upper layer for decoding, while the video frame data (P5~P8) received after the discontinuous timestamp transition point is discarded.

[0047] The packet loss handling method provided in this invention involves receiving a video data stream. The video data stream contains at least one I-frame and multiple non-I-frames, each frame associated with a timestamp. I-frames are video frames that can be decoded independently, while non-I-frames are video frames that require reference to other frames for decoding. When a discontinuous timestamp transition is detected, I-frame packet loss is determined. The continuous video frame data received before the discontinuous timestamp transition point is decoded, and video frame data received after the discontinuous timestamp transition point is discarded. This invention, when I-frame packet loss occurs, sends the packet before the timestamp change as the end frame to the upper layer for decoding and discards subsequent packets, ensuring smooth playback, reducing screen tearing, and decreasing stuttering time, thereby improving the service experience of video broadcasting.

[0048] Based on the above embodiments, determining that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected includes: Cache the current I-frame in the video data stream, as well as the consecutive non-I-frames following the current I-frame, until the next I-frame is detected; If a discontinuous jump in the timestamp of a non-I-frame is detected between two I-frames, it is determined that an I-frame packet loss has occurred.

[0049] Specifically, the current I-frame and subsequent consecutive non-I-frames are buffered until the next I-frame is detected; the continuity of timestamps of non-I-frames is detected; when a discontinuity in timestamps is detected, it is determined that I-frame packet loss has occurred.

[0050] For example, during the caching phase, the terminal caches the current I-frame in the video data stream (as the decoding reference frame, such as I0), as well as subsequent non-I-frames (P-frames / B-frames that depend on this I-frame, such as P1-P2-P3-P4), until it encounters the next I-frame (such as I5), at which point caching stops, forming a complete GOP (I0-P1-P2-P3-P4-I5). A continuity analysis is performed on the timestamps of non-I-frames between two I-frames. If the timestamps of non-I-frames increase uniformly (e.g., P1=33ms, P2=66ms, P3=99ms), it is normal. If the timestamps roll back or jump, it is determined that I-frame packet loss has occurred. For example, P2=66ms followed by P3=40ms (timestamp changes from 66 to 40); or P2=66ms followed by P3=200ms (the increment is much greater than expected).

[0051] This invention avoids screen distortion caused by decoding errors by using timestamp continuity detection and frame buffer management. It achieves accurate identification of I-frame loss, prevents error propagation, and improves the service experience of video broadcasting.

[0052] Based on the above embodiments, the step of decoding the continuous video frame data received before the timestamp discontinuity transition point and discarding the video frame data received after the timestamp discontinuity transition point includes: The current I-frame and the non-I-frames between the current I-frame and the discontinuous timestamp transition point are combined into an image group and sent to the decoder for decoding processing; Discard the non-I-frames between the discontinuous timestamp transition point and the next I-frame.

[0053] It should be understood that a Group of Pictures (GOP) refers to a group of consecutive frames at the coding level, which includes: one I-frame (keyframe) and multiple P / B frames (dependent frames), such as IBBBPBPBB.

[0054] The terminal continuously buffers all subsequent video frames (including P and B frames) starting from the most recently received complete I-frame. When a discontinuous jump in the timestamp of a video frame is detected (e.g., the timestamp increment is much larger than the normal frame interval), it is handled as follows: Complete frame processing: The sequence of complete frames with continuously increasing timestamps in the buffer (starting from the previous complete I-frame and ending at the last frame before the timestamp jump) is sent to the Video Coding Layer (VCL) for decoding. These frames constitute a complete set of images and can be decoded correctly.

[0055] Abnormal frame handling: Discard all P-frames and B-frames received after the timestamp jump point, as these frames cannot be guaranteed to be correctly decoded due to the lack of a reference frame (e.g., due to the loss of an I-frame).

[0056] Display control: The decoder maintains the output of the last fully decoded frame until a new complete I-frame is received, at which point normal video playback resumes. During this time, the image pauses, but no screen tearing occurs.

[0057] refer to Figure 2 The main steps for I-frame packet loss identification are as follows: Step 1: Buffer the first I-frame and subsequent P-frames and B-frames until the next I-frame; Step 2: Determine whether the timestamp of a non-I-frame between two I-frames has changed; Step 3: If the frame changes, it means that an I-frame packet loss has occurred. Treat the first I-frame and the non-I-frames before the timestamp change as a GOP and send them to the upper layer for decoding; at the same time, discard the other non-I-frames. Step 4: If nothing changes, it means that no I-frame packet loss has occurred. Treat the first I-frame and the non-I-frames before the second I-frame as a GOP and send them to the upper layer for decoding. Step 5: Continue receiving NALU frames until all video decoding is complete.

[0058] For example, first store the first I-frame (e.g., page 1), then continue storing the subsequent P / B frames (pages 2-9) until the next I-frame is encountered (page 10). Normally, the timestamps of P / B frames increase continuously (e.g., page 2 time = 10ms, page 3 = 20ms, etc.). If a sudden jump in timestamp is observed (e.g., page 5 time = 100ms, but page 6 = 10ms), it indicates that an I-frame was lost (e.g., an I-frame that should have appeared before page 6 was lost). The subsequent P / B frames (pages 6-9) actually belong to the next GOP, but were incorrectly linked to the previous GOP. In this case, treat the first I-frame plus the P / B frames before the abrupt change (pages 1-5) as a complete GOP, send it for decoding, and discard the P / B frames after the abrupt change (pages 6-9), because they are "unclaimed frames" (the reference I-frame was lost). If the timestamp is normal and there is no packet loss, treat the first I-frame plus all subsequent P / B frames (pages 1-9) as a GOP and decode it completely. Starting from the next I-frame (page 10), repeat the above process.

[0059] For example, a normal sequence: I0 (TS=0) → P1 (TS=33) → B2 (TS=66) → P3 (TS=99) → I4 (TS=132); Abnormal situation: Received I0(0) → P1(33) → [Packet loss] → P3(99) → I4(132); Processing method: When an abnormal jump of TS from 33 to 99 is detected (missing 66), the continuous sequence I0(0)+P1(33) is submitted for decoding, P3(99) is discarded, and the screen stays on the decoding result of P1 until I4(132) is received.

[0060] This invention only decodes frame sequences with complete reference relationships, avoiding decoding errors (screen tearing) caused by missing reference frames, and maintaining a smooth viewing experience by freezing the screen. Simultaneously, it can quickly and accurately identify I-frames, ensuring no I-frames are missed or not detected, thus guaranteeing no loss of key information and preventing interference with the decoding of non-I-frames within a GOP group, thereby reducing screen tearing time and providing a crucial foundation for subsequent rapid GOP group identification. On the other hand, it can quickly and accurately determine GOP groups, ensuring that accurately decodeable frames are not lost, resulting in longer smooth playback time, and also ensuring that undecodeable frames are accurately discarded, avoiding prolonged screen tearing. In particular, it may lead to excessive consumption of decoding computing power in the VCL layer, causing the next GOP group to fail to be decoded in time, and even more seriously, causing memory overflow, resulting in subsequent correct GOP groups being packet-dropped at the VCL layer.

[0061] Based on the above embodiments, the method further includes: If the first Real-Time Transport Protocol (RTP) packet corresponding to the video data stream is a fragmented data packet, the timestamp of the first RTP packet will be used as the base timestamp. The timestamps of subsequently received RTP packets are compared with the reference timestamp. If the timestamps are inconsistent and no end-of-frame marker is detected, a packet loss event is determined to have occurred. If the packet loss event is an I-frame packet loss, then the RTP packets before the timestamp discontinuity transition point are decoded and the RTP packets after the timestamp discontinuity transition point are discarded until a complete I-frame fragment sequence is received again. If the packet loss event is a non-I-frame packet loss, then the entire non-I-frame fragment sequence is discarded.

[0062] It should be understood that, according to RFC-3550, when the Markbit (7 bits) in the second byte of the RTP packet header is set to 1, it indicates that this large frame is the last packet in a fragmented encapsulation mode (such as FU-A mode). If the packet carrying the end of the frame is lost, this embodiment of the invention proposes a method to identify the end of the frame by combining changes in the RTP packet header timestamp, and consider that packet loss has occurred. Determining the end of the current frame by identifying a change in the timestamp or identifying a packet with the Markbit set to 1, and submitting it to the application layer for decoding, can achieve fast decoding and also avoid the incorrect handling of the start packet of the next frame being lost.

[0063] It should be understood that the Markbit (M flag) is the most significant bit (7th bit) of the 2nd byte of each RTP packet header. When M=1, it indicates that the current packet is the last fragment of the video frame. For example, an I-frame is sent as 3 RTP packets, marked as follows: Packet 1: M=0 (not the end); Packet 2: M=0 (not the end); Packet 3: M=1 ← This is the last slice.

[0064] When the first RTP packet in the video data stream is a fragmented data packet, the timestamp of that packet is used as the base timestamp. The timestamps of subsequent received RTP packets are compared with the base timestamp. If the timestamps are inconsistent and no end-of-frame marker is detected (E=1 in the FU-A header), it is determined to be a packet loss event. Differential processing is performed according to the type of lost frame: I-frame packet loss: valid fragments with consecutive decoding timestamps are discarded, and fragments after the transition point are discarded until a complete I-frame sequence is received; non-I-frame packet loss: the entire fragment sequence of the frame is directly discarded.

[0065] For example, for I-frame fragment packet loss: RTP1: [timestamp=100][seq=1][FU-A(S=1,Type=5)][I-frame fragment 1] → base timestamp=100; RTP2: [timestamp=100][seq=2][FU-A(Type=5)][I-frame fragment 2] → missing; RTP3: [timestamp=200][seq=3][FU-A(Type=1)][P-frame fragment 1] → timestamp jump and no E=1; Handling method: If an RTP3 timestamp jump (200≠100) is detected and there is no end marker, it is determined that the I-frame is lost; the RTP3 is discarded (because the I-frame is incomplete, the subsequent P-frames are invalid); wait to receive the complete I-frame sequence again.

[0066] For example, for P-frame fragment packet loss: RTP1: [timestamp=100][seq=1][FU-A(S=1,Type=1)][P-frame fragment 1] → base timestamp=100; RTP2: [timestamp=200][seq=2][FU-A(Type=5)][I-frame fragment 1] → timestamp jump; Processing method: If an RTP2 timestamp jump is detected (200≠100), it is determined that the P-frame packet is lost; RTP1 (incomplete P-frame) is discarded directly; I-frame sequence is processed starting from RTP2 as the new baseline.

[0067] refer to Figure 3 The main steps of ultra-large frame recognition are as follows: Step 1: When a very large frame exceeds the length of the MTU (Maximum Transmission Unit), multiple RTP packets are needed to encapsulate it. The most significant bit (mark bit) of the first byte of the RTP packet header, set to 1, indicates that this is the last packet. Step 2: Record the timestamp of the first RTP packet; Step 3: If the timestamp of the current packet is inconsistent with the timestamp of the record, it means that packet loss occurred before the packet end marker was received; Step 4: If the lost frame is not an I-frame, discard it; if the lost frame is an I-frame, send the packet before the timestamp changed as the end of the frame to the upper layer for decoding. Step 5: Discard any subsequent packets until an I-frame is found again.

[0068] The embodiments of the present invention can quickly and accurately determine the end of ultra-large frames, avoiding excessive cache usage and excessively long waiting time for the end of frames, thereby reducing the time required for frame recognition and GOP recognition and reducing the duration of stuttering.

[0069] Based on the above embodiments, the I-frame is identified in the following way: Read the Service Data Unit (SDU) from the RTP packet corresponding to the video data stream; The binary code stream of the SDU is parsed to locate the starting position of the Network Abstraction Layer Unit (NALU) according to the preset start code; Determine the NALU frame header based on the starting position of the NALU; The I-frame is identified based on the type field value in the NALU frame header.

[0070] It should be understood that due to the significant differences in H.264 frame sizes, RTP uses three methods to transmit H.264 frames: a single NALU mode is used when the packet size is smaller than the MTU; a combined frame encapsulation mode is used for multiple smaller frames; and a fragmented encapsulation mode is used for large frames exceeding the MTU. However, the frame start bit is 0x00000001, and the following byte is the NALU frame header. The NALU frame header is defined by RFC-3984, and it identifies whether the current frame is an I-frame or a combined frame containing I-frames.

[0071] It should be understood that an H.264 video stream consists of multiple NALUs, each with a start code of 0x00000001 (or 0x000001), followed by a 1-byte (8 bits) NALU header, the structure of which is as follows: | F (1 bit) | NRI (2 bits) | Type (5 bits) |; F (Forbidden Bit): Error flag, usually 0, 1 indicates illegal data; NRI (NAL Ref IDC): Priority. The higher the value, the more important the NALU (e.g., the NRI of an I-frame is usually 3). Type (NAL Unit Type): A key field used to determine the frame type (such as I-frame, P-frame, etc.).

[0072] RFC 3984 defines the NALU Type value range (0~31), among which the types directly related to I-frames include: Type=1: Slice of non-IDR frames, slices of ordinary P frames or B frames (non-key frames). Type=5: Slice of IDR frame, I frame (key frame), IDR means that the decoder can decode independently from this frame; Type=7: SPS (Sequence Parameter Set), which contains global encoding parameters (such as resolution and frame rate) and is required before decoding; Type=8: PPS (Image Parameter Set), which depends on the decoding parameters of SPS; Type=28: FU-A Fragmentation Unit, NALU segment in fragmentation mode, which needs to be determined by the fragmentation header to determine whether it belongs to an I-frame.

[0073] If Type=5, the current NALU is a complete I-frame (IDR frame); if Type=7 or 8, although it is not an I-frame, it is the necessary parameter set (SPS / PPS) for decoding an I-frame.

[0074] I-frame identification in Fragmented Mode (FU-A): When the NALU length exceeds the MTU, H.264 will fragment it for transmission (Type=28, FU-A mode). The fragment header (FU Header) structure is as follows: | S (1 bit) | E (1 bit) | R (1 bit) | Type (5 bits) |; S (Start Bit): 1 indicates the first packet in the fragment; E (End Bit): 1 indicates the last packet in the fragment; R (Reserved): Reserved bit; Type: The original NALU's Type (e.g., Type=5 of an I-frame will be retained in this field).

[0075] If the fragment header has Type=5 and S=1, then the fragment belongs to the beginning part of the I-frame. The receiver needs to reassemble all fragments to restore the complete I-frame NALU.

[0076] I-frame identification in STAP-A mode: Multiple small NALUs can be combined into a packet (Type=24, STAP-A mode), with each sub-NALU preceded by a 2-byte length field. The Type of each sub-NALU in the combined packet is parsed; if any sub-NALU has Type=5, then it contains an I-frame.

[0077] refer to Figure 4 The main steps of I-frame recognition are as follows: Step 1: The terminal reads the SDU (Service Data Unit) from the RTP packet corresponding to the video data stream and parses the H.264 encoded NALU frames from the SDU. The H.264 bitstream uses three modes (single NALU encapsulation mode, combined frame encapsulation mode, and fragmented encapsulation mode) to encapsulate the NALU frames within the SDU. Step 2: Parse the binary code stream of the SDU and determine the start bit of the NALU frame based on 0x00000001; Step 3: The next byte is the NALU frame header, where F (7 bits) is fixed at 0, NRI (5-6 bits), and type (0-4 bits). For example, scan the binary code stream byte by byte until a continuous 0x00000001 bit sequence is matched, and take the next byte after the matching position as the starting parsing position of the NALU frame header. Step 4: The F bit of the H.264 encoding is fixed at 0. If F is 1, it is considered an incorrect frame and is discarded. Step 5: When the Type field is 5, 7, or 8, determine that the current NALU frame is an I-frame or a combined frame containing I-frames; specifically, when Type=5, it is determined to be an IDR frame; when Type=7, it is determined to be a Sequence Parameter Set (SPS); and when Type=8, it is determined to be a Picture Parameter Set (PPS). When NRI is not 0 and Type is 28 or 29, determine that the current NALU frame is a non-single NALU encapsulated frame.

[0078] For example, (1) for a single NALU mode (I-frame): Hexadecimal stream: 00 00 00 01 05 28 AB CD...; Start code: 0x00000001; NALU header: 0x05 → Type=5 (IDR frame, i.e. I frame).

[0079] (2) For the sharding mode (FU-A): Fragment 1: 00 00 00 01 1C 85... (FU-A, S=1, Type=5); Fragment Packet 2: 00 00 00 01 1C 05... (FU-A, E=1, Type=5); In this case, Type=5 in the slice header indicates that the original NALU is an I-frame.

[0080] (3) For the combination pattern (STAP-A): Hexadecimal stream: 00 00 00 01 18 00 0A 05 28 AB...; Among them, 0x18 → Type=24 (STAP-A), 0x000A → First sub-NALU length=10 bytes, 0x05 → Sub-NALU Type=5 (I-frame).

[0081] This invention identifies I-frames through the Type field of the NALU frame header, ensuring that the decoder can obtain an independent and decorable starting point, avoiding the failure of the entire GOP decoding due to the loss of reference frames, and providing a basic guarantee for smooth video playback.

[0082] Based on the above embodiments, the method further includes: If a discontinuous jump in the timestamp is detected, it is determined that a non-I-frame packet loss has occurred; Discard non-I-frames following the discontinuous timestamp transition point.

[0083] It should be understood that when the terminal detects a discontinuous change in timestamps and confirms that the lost packets are not I-frames (P-frames or B-frames), the processing logic differs from that for I-frame packet loss. The terminal's specific response strategy can be: 1) If the timestamp jumps discontinuously (e.g., suddenly jumps from 100ms to 150ms), but the current NALU's Type=1 (P frame / B frame), then non-I frame packet loss occurs, and the current frame may be incomplete or missing.

[0084] 2) Since non-I-frames (P / B frames) rely on the decoding of preceding frames but do not carry complete image information themselves, the terminal can directly discard incomplete non-I-frames. This is because when a P / B frame loses part of its data, forced decoding will cause error propagation (such as mosaic or screen tearing), and the error will spread to all subsequent dependent frames. For example, discard non-I-frames after a discontinuous timestamp transition point and clear the corresponding decoding buffer for that frame.

[0085] Optionally, the terminal can fill the gaps in the screen caused by packet loss: 1) fill the current frame with the corresponding position of the previous frame, such as if a P-frame is lost, the previous frame is displayed directly; 2) predict the content of the lost frame based on the motion vectors of the previous and next frames; 3) perform smoothing filtering on the lost area to reduce visual abruptness.

[0086] For example, assuming the video sequence is: I0-P1-P2-P3-P4-I5, the RTP packet of P2 is lost, and the terminal receives I0-P1-[lost]-P3-P4-I5. The terminal detects that the timestamp of P3 is not continuous with P1 (e.g., P1=33ms→P3=100ms), confirming that the NALU header Type=1 (P-frame) of P3. It determines that P2 is lost, and subsequent P3 and P4, which depend on the decoding result of P2, are all discarded. Simultaneously, the missing P2-P4 frame is filled with a copy of the P1 frame (the frame is briefly frozen). It waits for the next complete I-frame (I5) to re-establish the decoding baseline.

[0087] In this embodiment of the invention, when incomplete non-I-frame data is detected, the decoding of that frame is discarded, which can save the decoding power of invalid data. At the same time, it avoids the propagation of errors caused by forced decoding and improves the service experience of video broadcasting.

[0088] The packet loss processing device provided by the present invention is described below. The packet loss processing device described below can be referred to in correspondence with the packet loss processing method described above.

[0089] refer to Figure 5 The packet loss processing device provided by the present invention includes: a receiving module 501, a packet loss determination module 502, and a packet loss processing module 503.

[0090] The receiving module 501 is used to receive a video data stream; the video data stream includes at least one I-frame and multiple non-I-frames, each frame is associated with a timestamp; the I-frame is a video frame that can be decoded independently, and the non-I-frames are video frames that need to be decoded by referring to other frames. The packet loss determination module 502 is used to determine that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected. The packet loss processing module 503 is used to decode the continuous video frame data received before the discontinuous timestamp transition point and discard the video frame data received after the discontinuous timestamp transition point.

[0091] The packet loss processing device provided in this embodiment of the invention receives a video data stream. The video data stream contains at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp. I-frames are video frames that can be decoded independently, while non-I-frames are video frames that require reference to other frames for decoding. When a discontinuous timestamp transition is detected, I-frame packet loss is determined. The device decodes the continuous video frame data received before the timestamp transition point and discards the video frame data received after the timestamp transition point. This invention, when I-frame packet loss occurs, sends the packet before the timestamp change as the end frame to the upper layer for decoding and discards subsequent packets, ensuring smooth playback, reducing screen tearing, and decreasing stuttering time, thereby improving the service experience of video broadcasting.

[0092] In one embodiment, the packet loss determination module 502 is specifically used for: Cache the current I-frame in the video data stream, as well as the consecutive non-I-frames following the current I-frame, until the next I-frame is detected; If a discontinuous jump in the timestamp of a non-I-frame is detected between two I-frames, it is determined that an I-frame packet loss has occurred.

[0093] In one embodiment, the packet loss processing module 503 is specifically used for: The current I-frame and the non-I-frames between the current I-frame and the discontinuous timestamp transition point are combined into an image group and sent to the decoder for decoding processing; Discard the non-I-frames between the discontinuous timestamp transition point and the next I-frame.

[0094] In one embodiment, the packet loss processing module 503 is further configured to: If the first Real-Time Transport Protocol (RTP) packet corresponding to the video data stream is a fragmented data packet, the timestamp of the first RTP packet will be used as the base timestamp. The timestamps of subsequently received RTP packets are compared with the reference timestamp. If the timestamps are inconsistent and no end-of-frame marker is detected, a packet loss event is determined to have occurred. If the packet loss event is an I-frame packet loss, then the RTP packets before the timestamp discontinuity transition point are decoded and the RTP packets after the timestamp discontinuity transition point are discarded until a complete I-frame fragment sequence is received again. If the packet loss event is a non-I-frame packet loss, then the entire non-I-frame fragment sequence is discarded.

[0095] In one embodiment, the packet loss processing module 503 is further configured to: Read the Service Data Unit (SDU) from the RTP packet corresponding to the video data stream; The binary code stream of the SDU is parsed to locate the starting position of the Network Abstraction Layer Unit (NALU) according to the preset start code; Determine the NALU frame header based on the starting position of the NALU; The I-frame is identified based on the type field value in the NALU frame header.

[0096] In one embodiment, the packet loss processing module 503 is further configured to: If a discontinuous jump in the timestamp is detected, it is determined that a non-I-frame packet loss has occurred; Discard non-I-frames following the discontinuous timestamp transition point.

[0097] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a packet loss processing method, which includes: receiving a video data stream; the video data stream contains at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp; the I-frame is a video frame that can be decoded independently, and the non-I-frames are video frames that need to be decoded with reference to other frames; determining that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected; decoding the continuous video frame data received before the discontinuous jump point in the timestamp, and discarding the video frame data received after the discontinuous jump point in the timestamp.

[0098] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the packet loss processing method provided by the above methods. The method includes: receiving a video data stream; the video data stream includes at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp; the I-frame is a video frame that can be decoded independently, and the non-I-frames are video frames that need to be decoded with reference to other frames; determining that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected; decoding the continuous video frame data received before the discontinuous jump point in the timestamp, and discarding the video frame data received after the discontinuous jump point in the timestamp.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the packet loss processing method provided by the above methods. The method includes: receiving a video data stream; the video data stream comprising at least one I-frame and a plurality of non-I-frames, each frame being associated with a timestamp; the I-frame being a video frame that can be decoded independently, and the non-I-frames being video frames that need to be decoded with reference to other frames; determining that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected; decoding the continuous video frame data received before the discontinuous jump point in the timestamp, and discarding the video frame data received after the discontinuous jump point in the timestamp.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A packet loss handling method, characterized in that, include: Receive video data stream; The video data stream contains at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp. The I-frame is a video frame that can be decoded independently, and the non-I-frame is a video frame that needs to be decoded by referring to other frames. If a discontinuous jump in the timestamp is detected, it is determined that an I-frame packet loss has occurred; The continuous video frame data received before the discontinuous timestamp transition point is decoded, and the video frame data received after the discontinuous timestamp transition point is discarded.

2. The packet loss handling method according to claim 1, characterized in that, The step of determining that I-frame packet loss has occurred when a discontinuous change in the timestamp is detected includes: Cache the current I-frame in the video data stream, as well as the consecutive non-I-frames following the current I-frame, until the next I-frame is detected; If a discontinuous jump in the timestamp of a non-I-frame is detected between two I-frames, it is determined that an I-frame packet loss has occurred.

3. The packet loss handling method according to claim 2, characterized in that, The process of decoding the continuous video frame data received before the timestamp discontinuity transition point and discarding the video frame data received after the timestamp discontinuity transition point includes: The current I-frame and the non-I-frames between the current I-frame and the discontinuous timestamp transition point are combined into an image group and sent to the decoder for decoding processing; Discard the non-I-frames between the discontinuous timestamp transition point and the next I-frame.

4. The packet loss handling method according to claim 1, characterized in that, The method further includes: If the first Real-Time Transport Protocol (RTP) packet corresponding to the video data stream is a fragmented data packet, the timestamp of the first RTP packet will be used as the base timestamp. The timestamps of subsequently received RTP packets are compared with the reference timestamp. If the timestamps are inconsistent and no end-of-frame marker is detected, a packet loss event is determined to have occurred. If the packet loss event is an I-frame packet loss, then the RTP packets before the timestamp discontinuity transition point are decoded and the RTP packets after the timestamp discontinuity transition point are discarded until a complete I-frame fragment sequence is received again. If the packet loss event is a non-I-frame packet loss, then the entire non-I-frame fragment sequence is discarded.

5. The packet loss handling method according to claim 1, characterized in that, The I-frame was identified based on the following method: Read the Service Data Unit (SDU) from the RTP packet corresponding to the video data stream; The binary code stream of the SDU is parsed to locate the starting position of the Network Abstraction Layer Unit (NALU) according to the preset start code; Determine the NALU frame header based on the starting position of the NALU; The I-frame is identified based on the type field value in the NALU frame header.

6. The packet loss handling method according to any one of claims 1 to 5, characterized in that, The method further includes: If a discontinuous jump in the timestamp is detected, it is determined that a non-I-frame packet loss has occurred; Discard non-I-frames following the discontinuous timestamp transition point.

7. A packet loss processing device, characterized in that, include: The receiving module is used to receive video data streams; The video data stream contains at least one I-frame and multiple non-I-frames, each frame being associated with a timestamp. The I-frame is a video frame that can be decoded independently, and the non-I-frame is a video frame that needs to be decoded by referring to other frames. The packet loss determination module is used to determine that I-frame packet loss has occurred when a discontinuous jump in the timestamp is detected. The packet loss processing module is used to decode the continuous video frame data received before the discontinuous timestamp transition point and discard the video frame data received after the discontinuous timestamp transition point.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the packet loss handling method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the packet loss handling method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the packet loss handling method as described in any one of claims 1 to 6.

Images