Video decoding adaptation for jitter

Abstract

Embodiments presented herein relate to a method (100) of decoding pictures from media packets P in a video stream, the method (100) comprising determining (110) if a first media packet P1, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D and, if so, skipping (120) decoding the first media packet P1, and decoding (130) at least one of the following received media packets Pn by the at least one decoder D Then, when receiving (140) the first media packet P1, determining (150) if all media packets P until a current picture PINOW have been received by the at least one decoder D, and, if so, resuming (160) error-free decoding with the received first media packet P1 at the current picture PINOW.

Description

[0001] VIDEO DECODING ADAPTATION FOR JITTER

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, device, computer program, computer program product and an apparatus for decoding pictures from media packets in a video stream.

[0004] BACKGROUND

[0005] In recent years, video streaming has become increasingly popular around the world and, as a consequence, more focus and research has been put into improving video quality in video streaming applications, for example by reducing the total latency of the video transcoding in live transmissions (see Jaseviciute et al., 2014).

[0006] SUMMARY

[0007] Current video decoding methods, which aim at decompressing digital video, typically suffer from lossy compression, which may result in an impaired video quality compared to the original, with compression artifacts such as blockiness or loss of fidelity. Other types of impairments from compression and the video transport of the compressed video include increase in latency (also called end-to-end delay), jitter (i.e., difference in media packet arrival time) or data errors, e.g, due to data loss in the transport. Consequently, there is a need for improved and reliable devices and methods for decoding pictures from media packets in a video stream.

[0008] Embodiments presented herein relate to a method, device, computer program, computer program product and an apparatus for decoding pictures from media packets in a video stream. It should be appreciated that these embodiments can be implemented in numerous ways. Several of these embodiments are described below.

[0009] According to a first aspect there is presented a method of decoding pictures from media packets in a video stream. Furthermore, the method comprises determining if a first media packet, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder. If the first media packet has not arrived in time to the at least one decoder, the method further comprises skipping decoding the first media packet and decoding at least one of the following received media packets by the at least one decoder. Moreover, if and when receiving the first media packet, the method comprises determining if all media packets until a current picture have been received by the at least one decoder and, if all media packets until the current picture have been received by the at least one decoder, the method further comprises resuming error-free decoding with the received first media packet at the current picture.

[0010] According to a second aspect there is presented a device for decoding pictures from media packets in a video stream. Furthermore, the device is adapted to determine if a first media packet, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder. If the first media packet has not arrived in time to the at least one decoder, the device is further adapted to skip decoding the first media packet and decode at least one of the following received media packets by the at least one decoder. Moreover, if and when the device receives the first media packet, the device is further adapted to determine if all media packets until a current picture have been received by the at least one decoder and, if all media packets until the current picture have been received by the at least one decoder, the device is further adapted to resume error-free decoding with the received first media packet at the current picture.

[0011] According to a third aspect there is presented an apparatus configured to perform the method according to the first aspect.

[0012] According to a fourth aspect there is presented a computer program comprising instructions, which when executed by processing circuitry, carries out the method according to the first aspect.

[0013] According to a fifth aspect there is presented a computer program product comprising a non-transitory storage medium including program code to be executed by a processing circuitry of a device or a cloud operatively connected to the device, whereby execution of the program code causes the device to perform operations comprising determining if a first media packet, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder and, if the first media packet has not arrived in time to the at least one decoder, the operations further comprising skipping decoding the first media packet and decoding at least one of the following received media packets by the at least one decoder. Moreover, if and when the first media packet is received, the operations further comprising determining if all media packets until a current picture have been received by the at least one decoder and, if all media packets until the current picture have been received by the at least one decoder, the operations further comprising resuming error-free decoding with the received first media packet at the current picture.

[0014] These aspects provide embodiments for decoding pictures from media packets in a video stream. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed description, from the attached dependent claims as well as from the drawings.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Fig. 1 is showing an example of functional units for a method of video decoding.

[0017] Fig. 2 is showing an example of functional units for video decoding comprising a decoder D for decoding pictures from media packets in a video bitstream.

[0018] Fig. 3 is showing an example of functional units for video decoding comprising a first decoder DA and a second decoder DB for decoding pictures from media packets in a video bitstream.

[0019] Fig. 4 is showing an example of functional units for a device adapted to video decoding.

[0020] Fig. 5 is showing a computer program product and a computer program, according to an embodiment of the disclosure.

[0021] DETAILED DESCRIPTION

[0022] At present, the disadvantages with current technology for video decoding are, for example, an increase in latency when packets arrive too late or - when continuing decoding with losses - that the recovery from losses is not carried out as efficiently as possible. For instance, when relying on requesting a new intra-coded picture refreshing the video, e.g., by sending, in an RTP session, an RTCP feedback message containing a NACK-PLI (not acknowledged, Picture-Loss-Indication), an intra-coded picture is typically quite large in terms of bits compared to an inter-coded picture and may cause a spike in the bitrate. It will also take some time from the requesting of the intra-coded picture until it is received, and error-free decoding can start.

[0023] Moreover, requesting intra-coded pictures could also be a vicious loop spiraling out of context. Since the requested intra-coded picture will typically create a spike in the bitrate, there is a risk of more packet loss from the intra-coded picture, leading to another intra-coded picture being requested, which could lead to a spiral of more and more intra-coded pictures being requested.

[0024] For "self-repairing video", such as using periodic intra-coded pictures and gradual decoding refresh, the overall bitrate is increased compared to only compressing the video with normal inter-coded pictures. It may also take some time for the video to be recovered depending on the chosen periodicity. Moreover, periodic intra-coded pictures are typically not suitable for low latency video as it typically causes the bitrate to spike periodically.

[0025] The aim of embodiments presented herein is to improve the decoding of pictures from media packets in a video stream.

[0026] Video coding

[0027] Before a video stream is transmitted over a network, it is compressed into a coded video bitstream using a video codec. The bitstream is then decompressed on the receiving side before further processing and / or display. Examples of widely deployed video codecs include H.264 / AVC, H.265 / HEVC and H.266 / VVC, all developed and standardized jointly by MPEG and ITU-T. A video codec typically supports both intra- coded (l-pictures or l-frames) and inter-coded pictures. An intra-coded picture may only predict from samples of the same picture, whereas inter-coded pictures may also predict from previously decoded pictures, referred to as reference pictures, thus enabling higher compression ratios than intra-coded pictures. Inter-coded pictures may be divided into P-pictures, which may only predict from one reference picture at a time for each coding block and bidirectional B-pictures, which may predict from up to two reference pictures simultaneously for each coding block.

[0028] Pictures in AVC, HEVC and VVC are identified by their picture order count (POC) values. POC determines the output order of decoded pictures. A picture in AVC, HEVC and VVC is divided into one or more slice network abstraction layer (NAL) units for easier packetization into network packets, where a slice may be a full picture or a part of a picture.

[0029] Low latency video coding

[0030] Low latency video is a key requirement for many applications of video, including conversational services, cloud gaming and services for extended reality (XR) comprising real-time virtual reality (VR), augmented reality (AR) and mixed reality (MR). Some of the factors that affect the latency of a coded video bitstream are as follows:

[0031] Reference pictures. Video for broadcast TV and video on demand (VOD) streaming is typically encoded with a group of pictures (GOP) structure where pictures are referencing both past and future pictures to achieve a good prediction with high- level of compression, often using bidirectional predicted (B) pictures that predicts from two reference pictures at a time. To achieve low latency, it is not recommended to reference future pictures as this builds up latency. Furthermore, in professional encoders for broadcasting and VOD it is common with a look-ahead of the coming pictures. This is also not recommended, since it increases latency significantly.

[0032] Rate control and buffer management. The number of bits needed to compress video pictures highly depends on the content of the video. Low motion scenes with little detail are relatively easy to compress, while scenes with lots of motion and detailed structures are more complex and require much more bits. To keep a constant quality of the video variable bitrate (VBR) is typically used in broadcasting and VOD services. To handle variable bitrate the encoding and decoding buffers must be sufficiently large which adds to the latency. To achieve lower latency with smaller buffer sizes the video can be encoded with constant bitrate (CBR). CBR has a lower compression efficiency than VBR but is recommended for real-time video services. Random access point pictures, a.k.a. key pictures, may be inserted into a video bitstream for three reasons: 1) as a start of the bitstream, 2) to refresh the video at certain intervals and 3) to let a decoder tune into the bitstream, e.g., when switching channel in broadcasted TV or moving to a different part of the video in VOD. Random access point pictures are typically intra-coded, which means that the picture does not predict from other pictures than itself, resulting in a much lower compression efficiency compared to if inter-picture prediction is allowed. Intra random access point (IRAP) pictures therefore tend to produce bitrate spikes in the bitstream, resulting in increased jitter and overall latency. An alternative to using IRAP pictures, recommended for low-latency video, is to use gradual decoding refresh (G DR) pictures, that refreshes a part of the video at a time over a range of pictures, allowing for a much smoother bitrate at the cost of slightly decreased compression efficiency and longer tune-in time. GDR is mandatory in H.266 / VVC and optional in H.264 / AVC and H.265 / HEVC. The drawback of using GDR for real-time streaming is that recovery from packet loss can be delayed since the refresh is spread across multiple frames. In addition, the overall compression efficiency is reduced when using GDR, since periodic intra-coded blocks are inserted into the bitstream to allow for the video to be refreshed.

[0033] Low latency video streaming

[0034] When streaming video with low latency, it is important to use a transport protocol that meets the latency requirements. A popular transport protocol for achieving latencies below 500 ms is the Real-time Transport Protocol (RTP), RFC 3550, which is typically run over the User Datagram Protocol (UDP). The Transmission Control Protocol (TCP) is normally not recommended as it requires retransmission of lost packets, which rapidly increases the latency. AVC, HEVC, VVC all have RTP payload formats to properly pack the elementary bitstream compressed using the codec in RTP.

[0035] To achieve low latency even when network conditions vary significantly, a congestion control algorithm is essential. Congestion occurs when the transmitted bitrate is higher than the available capacity over a given transmission path. Applications for video streaming may employ congestion control to achieve robust performance and to avoid congestion collapse.

[0036] Packet loss and packet loss concealment

[0037] In networks of today, it is common for nodes to have large buffers to avoid having to drop media or other data packets when there is congestion. The latter is often referred to as "buffer-bloat". A jitter buffer is a temporary storage buffer used to capture incoming media packets or other data packets. For real-time streaming applications, this means that the most common form of media packet loss is late- loss, i.e., media packets arriving too late to a receiver, and being dropped by the jitter buffer.

[0038] Different real-time video applications have different strategies for dealing with media packet loss. In video conference applications, it is generally agreed that dropping all frames until an error-free frame can be decoded will provide the best user experience, since there is usually not much motion in the video. In addition, it is generally preferable to have the video freeze rather than getting decoding artifacts that distort, e.g., somebody's face. In a cloud gaming application, on the other hand, there is usually a lot of motion, and the game play could be negatively affected if the game freezes. In such instances, the most common strategy is to allow for decoding with errors, and letting the decoder conceal the errors as much as possible.

[0039] The packet loss concealment methods differ between decoder implementations, with some coping better than others. Hardware decoding is most often preferred in battery constrained devices, which means that the application will have to use whatever concealment that the hardware decoder can provide for that specific hardware platform.

[0040] Jitter

[0041] Media packets or data packets over a network do not always arrive to the receiver at the same rate. The difference in media packet arrival time is referred to as jitter. The varying media packet arrival time could depend on many different things, including different encoding times, network congestion, network rerouting, etc. Media packets may also arrive out of order, and reordering algorithms are therefore typically applied so that media packets can be fed in the right order to the depacketizer and video decoder.

[0042] Playing out video with jitter may give rise to jerkiness artifacts. To avoid jitter, a playout buffer may be used as a jitter buffer. A jitter buffer may also be used in the receiving of media packets prior to the decoding of the video. As described in Morton and Claise, 2009, a jitter buffer is recommended for real time applications including video rendering. Typically, it is preferred to only delay the decoding when media packets have not yet arrived and immediately forward media packets to the decoder as they are received, and then add the jitter buffer delay to the playout of the video frames, to be prepared for media packets being held up by the jitter buffer in case some media packets are delayed.

[0043] The media packet arrival time in a network may be modelled with a Poisson process or a Gaussian probability distribution. Furthermore, there are many algorithms to dynamically adapt a jitter buffer depending on the traffic pattern. However, given the random nature of jitter, setting the jitter buffer large enough so that all media packets arrive in time to the jitter buffer will potentially increase the latency of the output significantly.

[0044] Requesting a new intra-coded picture

[0045] In the event of media packet loss, the decoder may not be able to decode an error- free video and, as a consequence, may need to recover from the media packet loss. In real-time video streaming applications, it is common to have the decoder recover from packet loss by having the receiver request the encoder on the sender side to produce an intra-coded picture. In applications using RTP, this is achieved by the receiver sending an RTCP feedback message containing a NACK-PLI (not acknowledged, Picture-Loss-Indication). Once the decoder has received the intra- coded picture, it will be able to resume decoding without errors.

[0046] Since the requested intra-coded picture will typically create a spike in the bitrate, there is a risk of more packet loss from the intra-coded picture, leading to another intra-coded picture being requested, which could lead to a spiral of more and more intra-coded pictures being requested.

[0047] Periodic intra-coded pictures and gradual decoding refresh

[0048] In order to tune into a bitstream or recover from lost or corrupted frames, a video bitstream may include periodic intra-coded pictures, often referred to as a key frame in the video coding community. In AVC, HEVC and VVC, such pictures are called intra random access point (IRAP) pictures. AVC, HEVC and VVC also support tuning into the bitstream at an inter-coded picture using gradual decoding refresh (GDR), where each picture from the tune-in point typically refreshes a new area of the picture, by coding that area with intra-coded blocks, until the whole picture has been refreshed a few pictures later. One main advantage with GDR is that a more even bitrate can be achieved compared to using periodic intra-coded pictures, as intra-coded blocks typically require significantly more bits than inter-coded blocks for a given quality. GDR is supported normatively in VVC using the GDR picture type, and optionally supported in AVC and HEVC using the recovery point supplemental enhancement information (SEI) message. The pictures in the video bitstream from an IRAP / GDR picture to the next IRAP / GDR picture in VVC is referred to as a sequence.

[0049] Periodic intra-coded pictures are not normally used in real-time video streaming applications, since the intra-coded pictures typically require more data to be sent unnecessarily even when there is no media packet loss, and recovery from media packet loss is typically delayed compared to requesting a new intra-coded picture, since the decoder will have to wait for the next periodic intra-coded picture.

[0050] No waiting for late or lost media packets

[0051] The idea of the proposed solution is to decode pictures without waiting for late or lost media packets. Thus, the latency can be kept below a certain threshold, where the certain threshold may be set depending on the service requirements and the network conditions, e.g., a low latency service may require a latency below 100 ms. As in the current art, the decoder creates concealment data or some other logic to gracefully handle the missing media packets. In the current art, media packets arriving too late are typically considered to be lost media packets, and techniques such as requesting a new intra-coded picture, using periodic intra-coded pictures, or using gradual decoding refresh are needed to recover from the losses. In the proposed solution, instead of relying on techniques from the current art to recover from losses, the media packets that arrived too late and were skipped for the decoding are used to recover to error-free decoding (or at least improve the video quality) at a later point in time.

[0052] In one embodiment, when a media packet at an instance t (where t may mean a certain time t or a at a certain picture t, e.g., t could be an identifier of the picture of the media packet such as a picture order count (POC) value) has not arrived in time to the decoder D, all media packets following said media packet are stored, together with the decoder state Dxof the decoder D. The decoding may continue with erroneous decoding of the bitstream, where at least the said media packet not arriving in time is skipped for the decoding. When said media packet and all media packets following said media packet up to the current time have been received, the decoder D restores to the decoder state Dxprior to said media packet, whereupon error-free decoding can be resumed.

[0053] In this disclosure, the decoder state Dxrefers to the state of a decoder D at an instance x, where x could mean at a certain time x or at a certain picture x. In one version, decoder state Dxis the state after time (or picture) t-1, i.e. the state of the picture prior to the picture with losses or late packets in time (or picture) t. In another version, the stored decoder state Dxis the decoder state directly prior to the lost or late packet, which could mean that the decoding, after the decoder D is restored to the decoder state Dx, would resume in the middle of a picture if the packet lost is not the first packet in the picture. Storing the decoder state Dxcomprises storing all data necessary such that decoding of the bitstream at the picture following the current picture x could be resumed after restoring that stored data to an initialized (i.e., reset) decoder, in a way that the decoder would output pictures identical to as if the decoding process had continued without initializing the decoder. Data necessary to store for the decoder state Dxmay include, but are not limited to, decoded reference pictures, data associated with the reference pictures, such as motion vectors, picture types, CABAC states, etc., parameter sets (or parameters from parameter sets) such as video parameter sets (VPS), sequence parameter sets (SPS), picture parameter sets (PPS), adaptation parameter sets (APS), supplemental enhancement information (SEI) messages or parameters from SEI messages. In a preferred version, the decoder state Dxis only stored if the decoding up to decoder state Dxhas been error-free.

[0054] In the disclosure, error-free decoding is defined as decoding the bitstream without packet loss. Packet loss may render in visual artifacts in the current picture which may propagate to following pictures if the current picture is used as a reference picture for following pictures. It is noted that error-free decoding can still have lossy compression.

[0055] In another embodiment, a second decoder DB is decoding all packets in parallel with a first decoder DA. However, instead of continuing decoding in spite of media packet loss, the second decoder DB always waits for media packets to arrive before decoding, meaning that the latency will be too high when a packet is late. In the embodiment, the second decoder DB decodes faster than real-time and will catch up with the latency requirements followed by the first decoder DA after some time after a late media packet has been encountered. How long it takes will depend on when all media packets have arrived and how fast the decoder is. When it has caught up with the first decoder DA, the state of the second decoder DB is copied to the state of the first decoder DA, whereupon error-free decoding with low latency can resume. In some embodiments, the second decoder DB is only run when there is delay and media packets are skipped. In another embodiment, instead of copying the state of the second decoder DB, the decoding device, apparatus, computer program, computer program product or system simply uses the second decoder DB as the first decoder DA and continues to output pictures from that and continues using the first decoder DA as the second decoder DB after the video has been refreshed, e.g., through gradual decoding refresh or by requesting a new intra-coded picture.

[0056] In another embodiment, the first decoder DA and the second decoder DB are instances of the same decoder. Late and lost media packets may be requested to be retransmitted from the encoder in any of the before-mentioned embodiments, which means that packets lost in the network can be recovered.

[0057] Embodiment with a single decoder

[0058] Fig. 1 comprises an example of functional units for a method 100 of decoding pictures from media packets P in a video stream in accordance with some embodiments in the disclosure. The method 100 comprises determining 110 if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D. If the first media packet Pi has not arrived in time to the at least one decoder D, the method 100 further comprises skipping 120 decoding the first media packet Pi, and decoding 130 at least one of the following received media packetsPn by the at least one decoder D. Then, when receiving 140 the first media packet Pi, the method 100 further comprises determining 150 if all media packets P until a current picture PI NOW have been received by the at least one decoder D. If all media packets P until the current picture PI NOW have been received by the at least one decoder D, the method 100 further comprises resuming 160 error-free decoding with the received first media packet Pi at the current picture PI NOW.

[0059] A media packet P (also called a packet) is defined as a video packet of the coded video bitstream, the media packet of which comprises a coded picture or coded pictures, part of a coded picture such as a slice, or additional data used to decode the picture, e.g., parameter sets.

[0060] The method 100 uses a decoder D for decoding a video bitstream. According to some embodiments of the disclosure, if a media packet Pi is lost or arrives after a threshold time Ti (e.g., 1.5 ms), the decoder D does not wait for the media packet Pi, or the following one or more media packets Pn, to arrive in time but continues decoding the bitstream with best-effort decoding, i.e., the decoder makes the best out of the situation where packet(s) has been lost, e.g., by applying packet loss concealment to mitigate artifacts from doing the decoding without the late or lost media packets. Decoding with packet loss may render in spatial coding artifacts for a current picture, which may propagate over time if the current picture is used as a reference picture in inter-coding for following pictures. The method 100 may further comprise determining 110 that the first media packet Pi has not arrived in time by determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti.

[0061] The method 100 may further comprise storing an error-free state Dxof the decoder D before the skipped media packet Pi, while the best-effort decoding is continued. Then, all media packets arriving after the lost or late media packet Pi are stored.

[0062] The method 100 may further comprise, after determining 110 that the first media packet Pi has not arrived in time, storing 110a a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dx, and storing 150a all media packets Pnreceived by the at least one decoder D following the present decoder state Di. In addition, if all media packets Pnuntil the current picture PI NOW have been received by the at least one decoder D, the method 100 may further comprise restoring 150b a state of the at least one decoder D to the latest error-free decoder state Dx.

[0063] Storing the error-free state Dxof the decoder D means storing the data of the decoder D needed to resume the decoding of the video at a specific point (associated with Dx) in the bitstream. The stored data may include parameter sets and reference pictures and data associated with the reference pictures, such as motion vectors, picture types, CABAC states, etc.

[0064] The method 100 may further comprise, if all media packets Pnuntil the current picture PI NOW have been received by the at least one decoder D, decoding 170 the received 140 first media packed Pi and all stored 150a media packets Pnfollowing the present decoder state Di up until the current picture PI NOW. Furthermore, the method 100 may comprise decoding and outputting 180 the current picture PI NOW.

[0065] Media packets P that have not arrived are requested to be retransmitted. Once all media packets P up to the current time have been received, the state of the decoder is set to the latest stored error-free state Dx, whereupon the decoding of the received media packets Pn(following the first media packet Pi) are resumed. Pictures are started to being output after the latest output picture of the best-effort decoding, e.g., if the latest picture of the best-effort decoding was done at time t, pictures from the error- free decoding are started to being output from time t+1. An example of the process for decoding and outputting a picture at an output time t for some embodiments is illustrated with functional units in Fig. 2. In the example, the process starts with an encoded picture ePt, which is at least partly part of a media packet with the corresponding state Di. Then, a check is made whether all media packets after the state Dxhave been received in time. If so, and the previous decoded output is error-free, the current encoded picture ePt, corresponding to output time t, is decoded into decoded picture dPt. If all media packets after the last error-free state, state Dx, have not been received in time, another check is made whether all media packets for the current encoded picture, ePt, have arrived before threshold time Ti. If so, the current encoded picture, ePt, is decoded to decoded picture dPt(but note that the state is not error-free, since there are still media packets from earlier pictures missing). If not all media packets for the current encoded picture, ePt, have arrived before threshold time Ti, the late media packets (which may be one or more, or even all media packets) are skipped before decoding the current encoded picture, ePt, to the decoded picture dPt. A request is made to the encoder to retransmit late or lost media packets. The state of the decoding is set to a non-error- free state. If all media packets after the latest error-free state, state Dx, have been received in time, but the decoding has not been error-free, the present decoder state Di is set to the latest error-free state, state Dx, the decoding state is set to error-free, and the encoded pictures from the last error-free state, state Dx, [ePx, ePx+i, ..., ePt], are decoded to decoded pictures [dPx, dPx+i, ..., dPt].

[0066] After the current encoded picture, ePt, has been decoded to dPt, it is output from the decoding device, apparatus, computer program, computer program product or system. If the decoding state is error-free, the last error-free decoder state, state Dx, is updated to the present decoder state, state Di.

[0067] Note that outputting a picture from the decoder D does not necessarily mean outputting the picture directly for display but could mean outputting the picture for further processing or to an output buffer.

[0068] In one embodiment, one or more media packets may not arrive before a threshold time TLATE (e.g., at a delay of 500 ms). These media packets may have been lost or may be arriving later. If the threshold time TLATE has been exceeded without the corresponding media packet being received, the decoding device, apparatus, computer program, computer program product or system would assume that the media packet was lost and could request for the packet to be retransmitted. Alternatively, other fallback strategies for recovering from lost media packets could be applied, including requesting a new intra-coded picture from the encoder to refresh the video stream for the decoder D and present decoder state Di.

[0069] In another embodiment, different values of threshold times Ti, TPI and / or TLATE could be used depending on the number of media packets being lost, e.g., it is a great difference if just a few media packets are lost or if most of the media packets are lost. For instance, the threshold times Ti, TPI and / or TLATE could be adapted to the statistics of the jitter and packet loss rate, e.g., such that if there are a massive loss of packet losses the threshold time TLATE could be lowered to trigger requesting a new intra-picture earlier. For instance, in a handover, if no media packets are received, the decoder D can be adapted to directly request a new intra-coded picture.

[0070] In another embodiment, the method 100 further comprises determining, from an indicator value, if the decoding in the at least one decoder D is error-free. For instance, the decoder D may store a value in a variable indicating whether the current state of the decoding is error-free or not. The indicator value may be a flag.

[0071] In another embodiment, a jitter buffer is used for the received media packets Pn. In yet another embodiment, a jitter buffer is used for the outputted pictures.

[0072] Embodiment with two decoders

[0073] Fig. 3 is showing an example of functional units for video decoding comprising a first decoder DA and a second decoder DB for decoding pictures from media packets in a video bitstream. In these embodiments of the disclosure, the decoding device, apparatus, computer program, computer program product or system comprises two decoder instances a first decoder DA and a second decoder DB, which are used to decode a video bitstream in parallel. The first decoder DA decodes all media packets arriving before a first threshold time Ti (e.g., a delay of 1.5 ms) and discards media packets arriving after the threshold time Ti in order to maintain a low latency decoding process such that pictures can be ready for output before a threshold time Tz. In other words, Ti can be defined as the maximum allowed delay for receiving media packets before decoding, and T2 can be defined as the maximum allowed delay for output. Thus, if a packet is lost or arrives too late, the first decoder DA does not wait for the packet but continues decoding the bitstream with best-effort decoding, e.g., by applying packet loss concealment. This may render in spatial coding artifacts that may propagate over time.

[0074] The second decoder DB decodes all media packets regardless of if they are late or not, including the media packets discarded by the first decoder DA, to form an error- free decoded stream of pictures, but with potentially too high latency to be ready for output at the threshold time T2. The second decoder DB is decoding faster than realtime, thus catching up over time to be ready for output at the threshold time T2 for a future picture if the coming media packets arrive in time.

[0075] The decoding device, apparatus, computer program, computer program product or system of the embodiment may follow the below rules for outputting a picture for a certain output time t (e.g., corresponding to a specific POC or timestamp):

[0076] 1. If the decoded picture diPtof the first decoder DA for time t is error-free, then output diPtfrom the decoding device, apparatus, computer program, computer program product or system.

[0077] 2. Else, if the decoded picture d2Pt of the second decoder DB for the certain time t is ready for output before threshold time T2, then output d2Pt from the decoding device, apparatus, computer program, computer program product or system. Since the second decoder DB has caught up with threshold time Ti, the first decoder DA is reset to error-free decoding. This may be done by copying the decoder state of the second decoder DB to the decoder state of the first decoder DA.

[0078] 3. Else, output diPtfrom the decoding device, apparatus, computer program, computer program product or system (which is not error-free, but could use some kind of error concealment)

[0079] Error concealment may for instance comprise skipping pictures or parts of pictures, copying pixels from a previously decoded picture, motion compensate pixels from a previously decoded picture, interpolating or extrapolating values from pixels in decoded parts within a picture, or interpolating or extrapolating pixel values between pictures.

[0080] In the example illustrated in Fig. 3, a check is first made whether all packets have been received before threshold time Ti for the encoded picture ePtcorresponding to output time t. If all pictures have been received in time, both decoders DA and DB decode the encoded picture ePtinto decoded pictures diPtand d?Pt. Otherwise, if all media packets have not been received before threshold time Ti, decoder DA skips the late or lost media packets and resumes the decoding of diPt, which is now set to not error-free, with best-effort decoding, e.g., by applying error concealment, and decoder DB waits for the late packets before decoding d?Pt without errors, however resulting in increased latency. Note that all packets may be late, in which case the decoding for decoder DA may be skipped or may only comprise applying error concealment for time the output picture at time t. A request to the sender (comprising the encoder) to retransmit the late or lost media packets can also be made.

[0081] Then, it is determined, e.g., by tracking with an indicator, if the decoding in the first decoder DA is error-free. If the decoding in the first decoder DA is error-free, diPtis output from the decoding device, apparatus, computer program, computer program product or system. Otherwise, if the decoding in DA is not error-free, a check is made whether d?Pt is ready for output before threshold time T2. If not, diPtis output from the decoding device, apparatus, computer program, computer program product or system. Otherwise, if dzPt is ready for output before threshold time T2, d2Pt is output from the decoding device, apparatus, computer program, computer program product or system. Since the decoding in the second decoder DB has now caught up to have pictures ready for output before threshold time T2, the first decoder DA may be reset to an error-free state by copying the decoder state of DB to the decoder state of DA.

[0082] Note that outputting a picture from the decoder D does not necessarily mean outputting the picture directly for display but could mean outputting the picture for further processing or to an output buffer. In another embodiment, the second decoder DB is only run if the first decoder DA is skipping a delayed (or lost) media packet of a reference picture and thus not outputting error-free decoded pictures. Then, the second decoder DB is initiated by copying the decoder state of the first decoder DA to the decoder state of the second decoder DB at the error-free state where the packet was skipped, and then continue decoding from there, but with an increased latency due to the one or more delayed media packets and the time it takes to decode these media packets. Once the second decoder DB has caught up to have pictures ready for output before threshold time T2, the first decoder DA may be reset to error-free decoding (e.g., by copying the state of DB to DA) and the second decoder DB may be stopped.

[0083] In another embodiment, the states of the decoders DA and DB may not be possible to reset or copy. Instead of copying the state of the second decoder DB to the first decoder DA, the decoding device, apparatus, computer program, computer program product or system simply swaps the first decoder DA and the second decoder DB, using the second decoder DB as the first decoder DA and continues to output pictures from that. The former first decoder DA is then continued to be used as the second decoder DB, but the video needs to be refreshed in order to provide error-free decoding. This may be done e.g., through requesting or using periodic gradual decoding refresh or by requesting a new intra-coded picture.

[0084] In another embodiment, one or more media packets may not arrive before a threshold time TLATE (e.g., at a delay of 500 ms). These media packets may have been lost or may be arriving later. If the threshold time TLATE has been exceeded without the corresponding media packet being received, the decoding device, apparatus, computer program, computer program product or system would assume that the media packet was lost and could request for the media packet to be retransmitted. Alternatively, other fallback strategies for recovering from lost media packets could be applied, including requesting a new intra-coded picture from the encoder to refresh the video stream for the first decoder DA (and, if still in operation, the second decoder DB).

[0085] In another embodiment, different values of threshold times Ti, T2 and TLATE could be used depending on the number of media packets being lost, e.g., it is a great difference if just a few media packets are lost or if most of the media packets are lost. For instance, in a handover, if no media packets are received, the decoder could go directly to request a new intra-coded picture.

[0086] In another embodiment, a jitter buffer is used for the received media packets Pn. In yet another embodiment, a jitter buffer is used for the outputted pictures.

[0087] Device for video decoding

[0088] Fig. 4 is showing an example of functional units for a device 200 with steps performed by the device 200 adapted to video decoding. The device 200 for decoding pictures from media packets P in a video stream is adapted to determine 210 if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D. If the first media packet Pi has not arrived in time to the at least one decoder D, the device 200 is further adapted to skip 220 decoding the first media packet Pi and decode 230 at least one of the following received media packets Pnby the at least one decoder D. Then, if and / or when receiving 240 the first media packet Pi, the device 200 is further adapted to determine 250 if all media packets P until a current picture PI NOW have been received by the at least one decoder D. If all media packets P until the current picture PI NOW have been received by the at least one decoder D, the device 200 is further adapted to resume 260 error-free decoding with the received first media packet Pi at the current picture PI NOW.

[0089] The device 200 comprises a decoder D for decoding a video bitstream. According to some embodiments of the disclosure, if a media packet Pi is lost or arrives after a threshold time Ti (e.g., 1.5 ms), the decoder D does not wait for the media packet Pi, or the following one or more media packets Pn, to arrive in time but continues decoding the bitstream with best-effort decoding, i.e., the decoder makes the best out of the situation where packet(s) has been lost, e.g., by applying packet loss concealment to mitigate artifacts from doing the decoding without the late or lost media packets. Decoding with packet loss may render in spatial coding artifacts for a current picture, which may propagate over time if the current picture is used as a reference picture in inter-coding for following pictures. The device 200 may further comprise determining 210 that the first media packet Pi has not arrived in time by determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti.

[0090] The device 200 may further comprise storing an error-free state Dxof the decoder D before the skipped media packet Pi, while the best-effort decoding is continued. Then, all media packets arriving after the lost or late media packet Pi are stored.

[0091] The device 200 may further comprise, after determining 210 that the first media packet Pi has not arrived in time, storing 210a a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dx, and storing 250a all media packets Pnreceived by the at least one decoder D following the present decoder state Di. In addition, if all media packets Pnuntil the current picture PI NOW have been received by the at least one decoder D, the device 200 may further comprise restoring 250b a state of the at least one decoder D to the latest error-free decoder state Dx.

[0092] Storing the error-free state Dxof the decoder D means storing the data of the decoder D needed to resume the decoding of the video at a specific point (associated with Dx) in the bitstream. The stored data may include parameter sets and reference pictures and data associated with the reference pictures, such as motion vectors, picture types, CABAC states, etc.

[0093] The device 200 may further comprise, if all media packets Pnuntil the current picture PI NOW have been received by the at least one decoder D, decoding 270 the received 240 first media packed Pi and all stored 250a media packets Pnfollowing the present decoder state Di up until the current picture PI NOW. Furthermore, the device 200 may comprise decoding and outputting 280 the current picture PI NOW.

[0094] In one embodiment, one or more media packets may not arrive before a threshold time TLATE (e.g., at a delay of 500 ms). These media packets may have been lost or may be arriving later. If the threshold time TLATE has been exceeded without the corresponding media packet being received, the decoding device, apparatus, computer program, computer program product or system would assume that the media packet was lost and could request for the packet to be retransmitted.

[0095] Alternatively, other fallback strategies for recovering from lost media packets could be applied, including requesting a new intra-coded picture from the encoder to refresh the video stream for the decoder D and present decoder state Di.

[0096] In another embodiment, different values of threshold times Ti, TPI and / or TLATE could be used depending on the number of media packets being lost, e.g., it is a great difference if just a few media packets are lost or if you lose most of the media packets. For instance, the threshold times Ti, TPI and / or TLATE could be adapted to the statistics of the jitter and packet loss rate, e.g., such that if there are a massive loss of packet losses the threshold time TLATE could be lowered to trigger requesting a new intra-picture earlier. For instance, in a handover, if no media packets are received, the decoder D can be adapted to directly request a new intra-coded picture.

[0097] In another embodiment, the device 200 further comprises determining, from an indicator value, if the decoding in the at least one decoder D is error-free.

[0098] In another embodiment, a jitter buffer is used for the received media packets Pn. In yet another embodiment, a jitter buffer is used for the outputted pictures.

[0099] Corresponding method and computer program product

[0100] Fig. 5 is showing a computer program product and a computer program, according to an embodiment of the disclosure. A computer program product comprising a non- transitory storage medium including program code to be executed by a processing circuitry of a device 200 or a cloud operatively connected to the device 200, whereby execution of the program code causes the device 200 to perform operations comprising determining if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D. If the first media packet Pi has not arrived in time to the at least one decoder D, the operations further comprising skipping decoding the first media packet Pi and decoding at least one of the following received media packets Pnby the at least one decoder D. Then, if or when receiving the first media packet Pi, the operations further comprising determining if all media packets P until a current picture PI NOW have been received by the at least one decoder D. If all media packets P until the current picture PI NOW have been received by the at least one decoder D, the operations further comprising resuming error-free decoding with the received first media packet Pi at the current picture PI NOW. The operations may further comprise decoding the at least one of the following received media packets Pnby the at least one decoder D is carried out with best-effort decoding. Moreover, the operations may further comprise determining that the first media packet Pi has not arrived in time by determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti. In addition, the operations may further comprise, after determining that the first media packet Pi has not arrived in time, storing a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dxand storing all media packets Pnreceived by the at least one decoder D following the present decoder state Di. If all media packets P until the current picture PI NOW have been received by the at least one decoder D, the operations may further comprise restoring a state of the at least one decoder D to the latest error-free decoder state Dx. Furthermore, the operations may comprise decoding and outputting the current picture PI NOW.

[0101] The following advantages of the embodiments described in this disclosure may be, for example:

[0102] • A lower latency can be achieved when there are media packets arriving late compared to a solution that waits for each media packet to arrive before decoding.

[0103] • Bandwidth and / or time may be saved in an ideal case where there are just a few late packets, as compared with solutions relying on requesting a new intra-coded picture. An intra-coded picture is typically quite large in terms of bits compared to inter-coded pictures and may cause a spike in the bitrate. It will also take some time from requesting the intra-coded picture until it is received and error-free decoding can start.

[0104] • Overall bandwidth does not need to be increased as compared to solutions using periodic intra-coded pictures or gradual decoding refresh, which would be the case when sending periodic intra-coded pictures or partial intra-coded pictures, which have lower compression efficiency than normal inter-coded pictures. Moreover, periodic intra-coded pictures are typically not suitable for low latency video as it typically causes the bitrate to spike periodically. REFERENCES

[0105] V. Jaseviciute, D. Plonis and A. Serackis, "Dynamic adaptation of the jitter buffer for video streaming applications," 2014 IEEE 2nd Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE), Vilnius, Lithuania, 2014, pp. 1-4, doi: 10.1109 / AIEEE.2014.7020318.

[0106] A. Morton and B. Claise, "RFC5481 Packet delay variation applicability statement," 2009, page 8. https: / / www.rfc-editor.orQ / rfc / rfc5481. html#section-3.2 [visited 2024-04-30],

Claims

CLAIMS1. A method (100) of decoding pictures from media packets P in a video stream, the method (100) comprising: determining (110) if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D; and, if the first media packet Pi has not arrived in time to the at least one decoder D: skipping (120) decoding the first media packet Pi; decoding (130) at least one of the following received media packets Pnby the at least one decoder D; receiving (140) the first media packet Pi; determining (150) if all media packets Pnuntil a current picture PINOW have been received by the at least one decoder D; and, if all media packets Pnuntil the current picture PINOW have been received by the at least one decoder D: resuming (160) error-free decoding with the received first media packet Pi at the current picture PINOW.

2. The method (100) according to claim 1, wherein decoding (130) the at least one of the following received media packets Pnby the at least one decoder D is carried out with best-effort decoding.

3. The method according to claim 1 or 2, wherein determining (110) that the first media packet Pi has not arrived in time comprises determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti.

4. The method according to claim 1, 2 or 3, wherein the determining (110) that the first media packet Pi has not arrived in time further comprises: storing (110a) a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dx; storing (150a) all media packets Pnreceived by the at least one decoder D following the present decoder state Di; and, if all media packets P until the current picture PINOW have been received by the at least one decoder D:restoring (150b) a state of the at least one decoder D to the latest error-free decoder state Dx.

5. The method (100) according to claim 4, wherein if all media packets P until the current picture PI NOW have been received by the at least one decoder D, the method further comprises: decoding (170) the received (140) first media packed Pi and all stored (150a) media packets Pnfollowing the present decoder state Di up until the current picture PI NOW.

6. The method (100) according to any one of claims 1 to 5, wherein the method (100) further comprises: decoding and outputting (180) the current picture PI NOW.

7. The method (100) according to any one of claims 1 to 6, wherein the at least one decoder D comprises a first decoder DA and a second decoder DB.

8. The method (100) according to claim 7, wherein the first decoder DA is configured to decode each picture from all media packets P in time and the second decoder DB is configured to do error-free decoding.

9. The method (100) according to claim 7 or 8, wherein the method (100) comprises determining if the decoding in the first decoder DA is not error-free and if a current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T2.

10. The method (100) according to claim 9, wherein the method (100) comprises, if the first decoder DA is not error-free and the current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T2: copying the decoder state DDB of the second decoder DB to the decoder state DDA of the first decoder DA.

11. The method (100) according to claim 9, wherein the method (100) comprises, if the first decoder DA is not error-free and the current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T?: swapping the decoder state DDB of the second decoder DB and the decoder state DDA of the first decoder DA.

12. The method (100) according to any one of claims 7 to 11, wherein the second decoder DB is only used when an arrival time TP of any of the media packets P is past a corresponding threshold arrival time Ti .

13. The method (100) according to any one of claims 1 to 12, wherein the method (100) further comprises: if at least one of the media packets P has not arrived before a threshold time TLATE: requesting retransmission of the at least one media packets P that has not arrived before a threshold time TLATE.

14. The method (100) according to any one of claims 1 to 13, wherein the method (100) further comprises: determining from an indicator value if the decoding in the at least one decoder D is error-free.

15. A device (200) for decoding pictures from media packets P in a video stream, the device (200) being adapted to: determine (210) if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D; and, if the first media packet Pi has not arrived in time to the at least one decoder D: skip (220) decoding the first media packet Pi; decode (230) at least one of the following received media packets Pnby the at least one decoder D; receive (240) the first media packet Pi;determine (250) if all media packets P until a current picture PINOW have been received by the at least one decoder D; and, if all media packets P until the current picture PINOW have been received by the at least one decoder D: resume (260) error-free decoding with the received first media packet Pi at the current picture PINOW.

16. The device (200) according to claim 15, wherein the device (200) is further being adapted to: decode (230), with best-effort decoding, the at least one of the following received media packets Pnby the at least one decoder D.

17. The device (200) according to claim 15 or 16, wherein the device (200) is further being adapted to: determine (210) that the first media packet Pi has not arrived in time comprising determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti.

18. The device (200) according to claim 15, 16 or 17, wherein the device (200) is further being adapted to: store (210a) a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dx; store (250a) all media packets Pnreceived by the at least one decoder D following the present decoder state Di; and, if all media packets Pnuntil the current picture PINOW have been received by the at least one decoder D: restore (250b) a state of the at least one decoder D to the latest error-free decoder state Dx.

19. The device (200) according to any one of claims 15 to 18, wherein if all media packets Pnuntil a current picture PINOW have been received by the at least one decoder D, the device (200) is further being adapted to: decode (270) the received (240) first media packed Pi and all stored (250a) media packets Pnfollowing the present decoder state DI up until the current picture PINOW.

20. The device (200) according to any one of claims 15 to 19, wherein the device (200) is further being adapted to: decode and output (280) the current picture PI NOW.

21. The device (200) according to any one of claims 15 to 20, wherein the at least one decoder D comprises a first decoder DA and a second decoder DB.

22. The device (200) according to claim 21, wherein the first decoder DA is configured to decode each picture from all media packets Pnin time and the second decoder DB is configured to do error-free decoding.

23. The device (200) according to claim 21 or 22, wherein the device (200) is further being adapted to: determine if the decoding in the first decoder DA is not error-free and if a current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T2.

24. The device (200) according to claim 23, wherein the device (200) is further being adapted to, if the first decoder DA is not error-free and the current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T2: copy the decoder state DDB of the second decoder DB to the decoder state DDA of the first decoder DA.

25. The device (200) according to claim 23, wherein the device (200) is further being adapted to, if the first decoder DA is not error-free and the current picture PI DB decoded by the second decoder DB is ready for output before a threshold time T2: swap the decoder state DDB of the second decoder DB and the decoder state DDA of the first decoder DA.

26. The device (200) according to any one of claims 21 to 25, wherein the second decoder DB is only used when an arrival time TP of any of the media packets P is past a corresponding threshold arrival time Ti.T1. The device (200) according to any one of claims 15 to 26, wherein the device (200) is further being adapted to, if at least one of the media packets P has not arrived before a threshold time TLATE: request retransmission of the at least one media packets P that has not arrived before a threshold time TLATE.

28. The device (200) according to any one of claims 15 to T1 , wherein the device (200) is further being adapted to: determine from an indicator value if the decoding in the at least one decoder D is error-free.

29. An apparatus configured to perform the method (100) according to at least one of claims 1 to 14.

30. A computer program comprising instructions, which when executed by processing circuitry or a cloud, carries out the method (100) according to any one of claims 1 to 14.

31. A computer program product comprising a non-transitory storage medium including program code to be executed by a processing circuitry of a device (200) or a cloud operatively connected to the device (200), whereby execution of the program code causes the device (200) to perform operations comprising: determining if a first media packet Pi, comprising at least one picture or a part thereof, has not arrived in time to an at least one decoder D; and, if the first media packet Pi has not arrived in time to the at least one decoder D: skipping decoding the first media packet Pi; decoding at least one of the following received media packets Pnby the at least one decoder D; receiving the first media packet Pi;determining if all media packets P until a current picture PI NOW have been received by the at least one decoder D; and, if all media packets P until the current picture PI NOW have been received by the at least one decoder D: resuming error-free decoding with the received first media packet Pi at the current picture PI NOW.

32. The computer program product according to claim 31, wherein decoding the at least one of the following received media packets Pnby the at least one decoder D is carried out with best-effort decoding.

33. The computer program product according to claim 31 or 32, wherein determining that the first media packet Pi has not arrived in time comprises determining that an estimated arrival time TPI for the first media packet Pi is past a threshold arrival time Ti.

34. The computer program product according to any one of claims 31 to 33, wherein the determining that the first media packet Pi has not arrived in time further comprises: storing a present decoder state Di of the at least one decoder D as a latest error-free decoder state Dx; storing all media packets Pnreceived by the at least one decoder D following the present decoder state Di; and, if all media packets P until the current picture PI NOW have been received by the at least one decoder D: restoring a state of the at least one decoder D to the latest error-free decoder state Dx.

35. The computer program product according to any one of claims 31 to 34, wherein the computer program product is further adapted to: decoding and outputting the current picture PI NOW.

Images