Intra-coded picture refresh with reference picture resampling (RPR)

Abstract

An encoder (206) encodes pictures into a coded video sequence (CVS) of a bitstream (130). More particularly, the encoder encodes refresh pictures at a resolution that is lower than the resolution used to encode one or more inter-coded pictures that follow the refresh picture in the CVS. The encoded pictures are then sent to a decoder (306). Upon receipt, the decoder decodes the pictures from the CVS in which the refresh picture was encoded at a resolution that is lower than the resolution used to encode one or more inter-coded pictures that follow the refresh picture in the CVS.

Description

[0001] INTRA-CODED PICTURE REFRESH WITH REFERENCE PICTURE RESAMPLING (RPR)

[0002] TECHNICAL FIELD

[0003] This application relates generally to video encoding and decoding techniques, and more particularly to refresh techniques for refreshing a video in a bitstream.

[0004] BACKGROUND

[0005] Using a video codec, an encoder compresses a video stream into a coded video bitstream prior to transmitting the bitstream to a receiving device over a network. This so-called compression process is generally referred to as “encoding.” Upon receipt, the receiving device uses the video codec to decompress the coded video bitstream before forwarding the video for further processing and / or display to a user. The process of decompressing the encoded video bitstream is generally referred to as “decoding.” Generally, video streams can be encoded and decoded using any video codec needed or desired. However, some examples of widely deployed video codecs include those that are configured to operate according to H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), and / or H.266 / Versatile Video Coding (WC), each of which was developed and standardized jointly by the Moving Picture Experts Group (MPEG) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T).

[0006] Regardless of the particular standard, however, video codecs typically support both intra-coded and inter-coded pictures. An intra-coded picture may only predict from samples of the same picture, whereas inter-coded pictures may also predict from previously decoded pictures referred to as “reference pictures.” Inter-coded pictures may further be divided into “P- pictures,” which may only predict from one reference picture at a time for each coding block, and bi-directional “B-pictures,” which may predict from up to two reference pictures simultaneously for each coding block.

[0007] During the encoding process, conventional video codecs determine a difference between the pixel data of a first image (e.g., a reference image) and the pixel data of a second, subsequent image that used the first image as a reference image. This difference is typically referred to as the “residual,” and is transformed into the frequency domain by the encoder. The transformed residual is then quantized and entropy-coded before being transmitted to a decoder together with one or more necessary entropy-coded prediction parameters such as prediction mode and motion vectors. The level of quantization is controlled with a quantization parameter (QP) value. The decoder performs the reverse process. That is, the decoder applies inverse quantization and inverse transformation to a received bitstream to obtain the residual. So obtained, the decoder adds the residual to an intra-prediction or inter-prediction to reconstruct the picture. SUMMARY

[0008] The present disclosure provides methods and corresponding devices for encoding and decoding a CVS of a bitstream. More particularly, the present embodiments encode / decode refresh pictures at a resolution that is lower than the resolution(s) used to encode one or more inter-coded pictures that follow the refresh picture. Reducing the resolution of the refresh picture in this manner avoids or at least greatly reduces the bitrate spikes associated with large intracoded pictures and / or with inter-coded pictures having a large proportion of intra-codec blocks.

[0009] Accordingly, in a first aspect, the present disclosure provides a method, implemented by an encoder, for encoding pictures to a coded video sequence (CVS) in a bitstream. The method comprises generating the CVS in the bitstream, which in this aspect, comprises encoding a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded, encoding a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture, and sending the CVS of the bitstream to a receiving device.

[0010] In a second aspect, the present disclosure provides a method, implemented by a decoder, for decoding pictures from a coded video sequence (CVS) in a bitstream. In this embodiment, the method comprises decoding a first picture from the CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution, decoding a second picture from the CVS in the bitstream, wherein the second picture follows the first picture in decoding order, has a second resolution that is different from the first resolution, and is decoded as an inter-coded picture predicted from the first picture. So decoded, the method further comprises outputting the decoded second picture.

[0011] In a third aspect, the present disclosure provides an encoder for encoding pictures to a coded video sequence (CVS) in a bitstream. In this embodiment, the encoder is configured to generate the CVS in the bitstream. To generate the CVS, the encoder is configured to encode a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded, and encode a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture. So encoded, the encoder sends the CVS of the bitstream to a receiving device.

[0012] In a fourth aspect, the present disclosure provides an encoder encoding pictures to a coded video sequence (CVS) in a bitstream. In this aspect, the encoder comprises communications circuitry configured to communicate with a receiving device and processing circuitry operatively connected to the communications circuitry. In this aspect, the processing circuitry is configured to generate the CVS in the bitstream by encoding a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded, and encoding a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture. Once encoded, the encoder of this aspect sends the CVS of the bitstream to a receiving device.

[0013] In a fifth aspect, the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry in an encoder, causes the encoder to implement the functions of the first aspect.

[0014] In a sixth aspect, the present disclosure provides a decoder for decoding pictures from a coded video sequence (CVS) in a bitstream. In this aspect, the decoder is configured to decode a first picture from the CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution, decode a second picture from the CVS in the bitstream, wherein the second picture follows the first picture in decoding order, has a second resolution that is different from the first resolution, and is decoded as an inter-coded picture predicted from the first picture. So decoded, the decoder outputs the decoded second picture.

[0015] In a seventh aspect, the present disclosure provides a decoder for decoding pictures from a coded video sequence (CVS) in a bitstream. In this aspect, the decoder comprises communications circuitry configured to communicate with a sending device and processing circuitry operatively connected to the communications circuitry. The processing circuitry is configured to decode a first picture from the CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution, decode a second picture from the CVS in the bitstream, wherein the second picture follows the first picture in decoding order, has a second resolution that is different from the first resolution, and is decoded as an inter-coded picture predicted from the first picture. So decoded, the processing circuitry outputs the decoded second picture.

[0016] In an eighth aspect, the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry in a decoder, causes the decoder to implement the functions of the second aspect.

[0017] BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a functional block diagram illustrating a communications network configured according to embodiments of the present disclosure.

[0019] Figure 2 illustrates periodic Intra-Random Access Point (IRAP) pictures in a bitstream according to embodiments of the present disclosure. Figure 3 is a signaling diagram illustrating the signaling for refreshing a picture of a bitstream according to embodiments of the present disclosure.

[0020] Figure 4A is a graph illustrating the results of not using reference picture resampling (RPR) for intra-coded pictures according to embodiments of the present disclosure.

[0021] Figure 4B is a graph illustrating the results of using RPR for intra-coded pictures according to embodiments of the present disclosure.

[0022] Figure 4C is a graph illustrating the results of using a 2-step RPR approach for intra- coded pictures according to embodiments of the present disclosure.

[0023] Figures 5A-5D are graphs illustrating different methods for handling the size of an intra frame according to embodiments of the present disclosure.

[0024] Figure 6 illustrates periodic IRAP pictures in a bitstream in which the IRAP pictures are encoded at a lower resolution than the inter-coded pictures between them according to embodiments of the present disclosure.

[0025] Figure 7 illustrates a stepwise approach to encoding a CVS where periodic IRAP pictures are encoded to CVSs of a bitstream at a lower resolution than the inter-coded pictures disposed between them, and where an inter-coded picture appearing immediately after an IRAP picture is encoded to the CVS comprising the IRAP picture at an intermediate resolution between those of the IRAP picture and at least one of the other inter-coded pictures in the CVS according to embodiments of the present disclosure.

[0026] Figure 8 is a flow diagram illustrating a method for encoding pictures into a CVS according to embodiments of the present disclosure.

[0027] Figure 9 is a flow diagram illustrating a method for decoding pictures from a CVS according to embodiments of the present disclosure.

[0028] Figure 10 is a functional block diagram illustrating some exemplary components of a sending device (e.g., an encoder) configured according to embodiments of the present disclosure.

[0029] Figure 1 1 is a functional block diagram illustrating some exemplary components of a receiving device (e.g., a decoder) configured according to embodiments of the present disclosure.

[0030] DETAILED DESCRIPTION

[0031] This application claims the benefit of U.S. Provisional Application No. 63 / 632614 filed 1 1 April 2024, the entire disclosure of which is incorporated herein by reference.

[0032] The present disclosure provides a technique and corresponding apparatus for encoding refresh pictures at a resolution that is lower than the resolution(s) used to encode one or more inter-coded pictures that follow the refresh picture. Reducing the resolution of the refresh picture in this manner avoids or at least greatly reduces the bitrate spikes associated with large intra- coded pictures and / or with inter-coded pictures having a large proportion of intra-codec blocks. In more detail, the present disclosure provides a technique and corresponding apparatus for encoding pictures to, and decoding pictures from, a coded video sequence (CVS) in a bitstream. More particularly, when refreshing a video of the bitstream, a sending device configured according to the present embodiments (e.g., an encoder) encodes a refresh picture at a resolution that is lower than those used to encode one or more inter-coded pictures following the refresh picture. The sending device may use reference picture resampling (RPR) to predict from the refresh picture when encoding the inter-coded pictures. The receiving device (e.g., a decoder) decodes the encoded pictures from the CVS in the bitstream. The receiving device is particularly configured to request a refresh picture from the sending device, and upon receipt of the bitstream, decode the pictures from the CVS using the refresh picture as a reference picture.

[0033] Turning now to the drawings, Figure 1 is a functional block diagram illustrating a communications network 10 configured according to one embodiment of the present disclosure. As seen in Figure 1 , network 10 comprises an access network 12 communicatively connecting a client device 300 (e.g., an HMD, a mobile phone or a computer) with a network node 200 (e.g., a server node) disposed in a cloud network 14. In some embodiments, a computing device 20 may be disposed between client device 300 and server node 200 and is configured to perform at least some of the processing functions of client device 300.

[0034] The access network 12 may be any type of communications network (e.g., Wireless Fidelity (WiFi), ETHERNET, Wireless Local Area Network (WLAN), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), etc.), and functions to connect subscriber devices, such as client device 300, to one or more service provider nodes, such as network node 200. The cloud network 14 provides such subscriber devices with “on-demand” availability of computer resources (e.g., memory, data storage, processing power, etc.) without requiring the user to directly, actively manage those resources. According to embodiments of the present disclosure, such resources include, but are not limited to, one or more conversational services, cloud gaming applications, or XR applications being executed on network node 200. The XR applications, which are described in more detail below, may comprise, for example, those used in connection with gaming applications and / or simulation applications.

[0035] Video Coding

[0036] As previously stated, video codecs are used to compress (i.e., encode) pictures to a coded video sequence (CVS) in a bitstream. In the context of Figure 1 , for example, a video codec executing on network node 200 may encode images for a video (e.g., pictures generated by a game engine) to send to client device 300 for decoding and display to a user. Some of the more widely known and used video codecs were developed and standardized by the MPEG and ITU-T, and include, but are not limited to, H.264 / AVC, H.265 / HEVC, and H.266 / WC. Pictures in AVC, HEVC, and WC are identified by their picture order count (POC) values. The POC determines the output order of decoded pictures. Additionally, according to the H.264 / AVC, H.265 / HEVC video coding standards, a picture is typically divided into one or more slice Network Abstraction Layer (NAL) units. A slice may be a full picture or a part of a picture. Further, each NAL unit is essentially a data packet, thereby facilitating the packetization of the NAL units into network packets. In this disclosure, the terms “picture” and “frame” are used interchangeably.

[0037] Random Access Point (RAP) Pictures

[0038] In AVC, HEVC, and WC, as well as in most other modern video codecs, random access point pictures, which are commonly referred to by those of ordinary skill in the art as “key pictures,” are typically used in a video bitstream for three reasons.

[0039] 1 . as a start of a bitstream;

[0040] 2. to refresh a video; or

[0041] 3. to let a decoder tune into the bitstream, such as when switching channels in broadcast TV scenario or when moving to a different part of the video in a video- on-demand (VOD) scenario.

[0042] Random Access Point (RAP) pictures are typically intra-coded, which means that a RAP picture predicts only from itself and not from other pictures. Such prediction, however, results in a much lower compression efficiency compared to scenarios where inter-picture prediction is allowed. Therefore, intra-random access point (IRAP) pictures tend to produce undesirable bitrate spikes in the bitstream, which can result in increased jitter and overall latency.

[0043] WC in particular supports two types of IRAP pictures. These are:

[0044] 1. instantaneous decoding refresh (IDR) pictures; and

[0045] 2. clean random access (CRA) pictures.

[0046] IRAP pictures may have “trailing pictures”, that follow the associated IRAP picture in both decoding and output order, and “leading pictures” that follow the associated IRAP picture in decoding order but precede the IRAP picture in output order. There are two types of leading pictures in WC - i.e., random access decodable leading (RADL) pictures that are correctly decoded when starting the decoding at the associated IRAP picture, and random access skipped leading (RASL) pictures that may not be correctly decoded when starting the decoding at the associated IRAP picture. IDR pictures may have associated RADL pictures but not RASL pictures and it is thus ensured that all pictures in the bitstream following the IDR picture in decoding order can be decoded correctly. However, CRA pictures may also have RASL pictures. In some cases, e.g., when decoding begins at the CRA picture, such leading pictures may need to be dropped.

[0047] Gradual Decoding Refresh

[0048] In some cases, gradual decoding refresh (GDR) pictures may be used as an alternative to using IRAP pictures. GDR pictures, which are recommended for low-latency video scenarios, refresh only a portion of a video at a time over a range of pictures. This allows for a much smother bitrate, but at the cost of a slightly decreased compression efficiency and longer tune-in time. Another drawback of using GDR for real-time streaming is that recovery from packet loss can be delayed. This is because the refresh is spread across multiple frames (i.e., pictures).

[0049] Additionally, the overall compression efficiency is reduced when using GDR since periodic intracoded blocks are inserted in the bitstream to allow for the video to be refreshed. Despite these drawbacks, however, GDR is optional for supporting in H.264 / AVC and H.265 / HEVC, but it is mandatory for decoders to support H.266 / WC.

[0050] Coded video sequence

[0051] In WC and HEVC, a coded video sequence (CVS) is a sequence of access units (AUs) followed by zero or more AUs up to, but not including, the AU starting the next CVS in decoding order. In WC, the AU starting a CVS may include an IRAP picture, such as an IDR or a CRA picture, for example, or it may comprise a GDR picture. In HEVC, however, the AU starting the CVS is an IDR picture. Regardless of the particular standard, however, a CVS in the context of this disclosure is broadly defined as a sequence of coded video frames (i.e., pictures) in a bitstream.

[0052] Reference picture resampling (RPR)

[0053] A new feature in WC is reference picture resampling (RPR). RPR allows the spatial resolution (i.e., the width and height) of the pictures being encoded to a CVS to change in the middle of the bitstream without having to encode an IRAP picture into the CVS. In WC, the spatial resolution of a picture may be modified on a picture level and is therefore signaled in the picture parameter set (PPS). In comparison, both AVC and HEVC signal the spatial resolution of a picture on the sequence level in the sequence parameter set (SPS). Notably, neither AVC nor HEVC currently support RPR functionality.

[0054] When the current picture and a reference picture have different spatial resolutions, RPR enables the encoder to use the reference picture to predict the current picture by scaling the reference picture to the same spatial resolution as the current picture before prediction occurs. This scaling is done on the block level.

[0055] As defined in the context of this disclosure, RPR functionality is not limited to the RPR functions of WC. Rather, RPR is defined more generically as reference picture resampling according to codecs, such as WC and AV1 , as well as reference picture resampling functionality that may be provided by future video codecs.

[0056] Low latency video coding

[0057] Low latency video coding is a key requirement for many applications of video, including conversational services, cloud gaming, and services for extended reality (XR). As defined in this disclosure, XR comprises real-time virtual reality (VR), augmented reality (AR), and mixed reality (MR). There are various factors that can affect the latency of a coded video bitstream. Among these factors are those related to reference pictures and those related to rate control and buffer management.

[0058] Reference Pictures

[0059] Video for broadcast TV and video on demand (VOD) streaming is typically encoded with a group of pictures (GOP) structure. With such structures, pictures reference both past and future pictures to achieve a sufficient level of prediction with high-level of compression. To accomplish this, encoders often use bidirectional predicted pictures (i.e. , B pictures) that predict from two reference pictures at a time. However, while such functionality may help with prediction and compression, using future pictures for prediction also increases latency. Therefore, it is not recommended to reference future pictures for low latency video coding. Similarly, it is common for professional encoders used in broadcasting and VOD scenarios to provide a look-ahead feature to look ahead to the pictures that are coming. However, this feature also significantly increases latency, and thus, is also not recommended for low latency video coding.

[0060] Rate Control and Buffer Management

[0061] The number of bits needed to compress video pictures is highly dependent on the content of the video. Low motion scenes with little detail are relatively easy to compress, while scenes with lots of motion and detailed structures are more complex and require many more bits. Therefore, broadcasting and VOD services typically use a variable bitrate (VBR) to maintain a constant quality of the video. To handle variable bitrate, the encoding and decoding buffers must be sufficiently large. However, this adds latency. To achieve lower latency with smaller buffer sizes, the video can be encoded with constant bitrate (CBR). CBR has a lower compression efficiency than VBR but is not recommended for use with broadcasting and VOD services. Rather, CBR is recommended for use with real-time video services.

[0062] Rate control algorithms typically modify the QP value per picture, per slice, or per block to achieve desired bitrates. Alternatively, or additionally, the lambda value associated with the rate control algorithm can be changed.

[0063] Video Streaming

[0064] MPEG-DASH

[0065] MPEG-Dynamic Adaptive Streaming over HTTP (MPEG-DASH), specified in ISO / IEC 23009-1 dated August 2022 and which is incorporated herein by reference in its entirety, is an adaptive bitrate (ABR) streaming technology. With MPEG-DASH, a multimedia file is partitioned into one or more segments and delivered to a client using Hypertext Transfer Protocol (HTTP), typically over Transmission Control Protocol (TCP). An MPEG-DASH session is set-up using a media presentation description (MPD) that describes segment information including timing, Uniform Resource Locator (URL), and media characteristics such as video resolution and bit rates. MPDs, which are Extensible Markup Language (XML)-based, can be static (e.g. for movies) or dynamic (e.g., for live content). Commonly, VOD services using ABR modify both the frame rate and the resolution of a video in chunks or segments, such as in MPEG-DASH, to achieve appropriate bitrates depending on the available network bandwidth. Particularly, each segment starts with an intracoded picture to facilitate the switch between different resolutions. For each segment, a plurality of different bitrates is typically pre-encoded where each bitrate corresponds to a certain quality in a bitrate ladder. The following is an example of a bitrate ladder.

[0066] Table 1

[0067] Segments are typically in the order of 6 to 20 seconds long. Depending on the available network bandwidth at a given segment time, a quality for the segment is requested / chosen with a bitrate that matches the available bandwidth. Additionally, the different qualities Q1-Q5 may be encoded using different codecs.

[0068] Low Latency Video Streaming

[0069] When streaming video with low latency, it is important to use a transport protocol that meets the latency requirements. A popular transport protocol for achieving latencies below 500ms is the Real-time Transport Protocol (RTP), which is typically run over the User Datagram Protocol (UDP). The standard for RTP, i.e., RFC 3550 entitled “RTP: A Transport Protocol for Real-Time Applications” and dated July 2003, is expressly incorporated herein by reference. TCP is normally not recommended as it requires the retransmission of lost packets, which rapidly increases the latency. AVC, HEVC, and WC all have RTP payload formats to properly pack the elementary bitstream that was compressed using the codec in RTP.

[0070] A congestion control algorithm is essential to achieve low latency even when the conditions of the network vary. Congestion occurs when the transmitted bitrate is higher than the available bitrate capacity over a given transmission path. Applications for video streaming may employ congestion control to achieve robust performance and avoid congestion collapse (i.e., a state where the network is stable but there is low throughput).

[0071] Packet loss and packet loss concealment

[0072] Nodes in currently existing networks often have large “jitter” buffers. This helps the nodes avoid having to drop packets when faced with congestion. However, this is often referred to as “buffer-bloat.” For real-time streaming applications, this means that the most common form of packet loss is “late-loss.” Such late-loss occurs when packets successfully arrive at the receiver but took longer than an acceptable delay period to arrive at the receiver. In these cases, the “late-loss” packets are simply dropped by the jitter buffer.

[0073] Different real-time video applications have different strategies for dealing with packet loss. For example, consider video conference applications. In such applications, it is generally agreed that dropping all frames until an error-free frame can be decoded will provide the best user experience. This is because the video may not exhibit much motion. Therefore, having a video freeze during a video conference is generally preferrable over obtaining decoding artifacts that, when displayed, can distort an object in the video, such as a person’s face. On the other hand, a cloud gaming application usually has a lot of motion. In such cases, game “freezes” can negatively affect the game play. Therefore, the most common strategy for dealing with such freezes is to allow for decoding with errors, and then letting the decoder conceal the errors as much as possible.

[0074] However, packet loss concealment methods differ between decoder implementations with some packet loss concealment methods being better able to cope with errors than others. For example, hardware-based decoding is most often preferred in battery constrained devices. While this prolongs battery life, it also constrains the application into using whatever concealment methodologies the hardware decoder can provide for that specific hardware platform.

[0075] Recovering from Loss

[0076] Full Intra Request

[0077] Decoders are unable to decode an error-free video in cases where packets are lost, and thus, need time to recover. Therefore, decoders in real-time video streaming applications commonly recover from packet loss by requesting an encoder to produce an intra-picture. In applications using RTP, this is achieved by the receiving device sending a Real-Time Transport Control Protocol (RTCP) feedback message comprising a Negative Acknowledged, Picture- Loss-Indication (NACK-PLI) or a NACK-Slice-Loss-lndication (NACK-SLI). Once the decoder has received the intra-picture, it will be able to resume decoding without errors.

[0078] However, the requested intra-picture will typically create a spike in the number of transmitted packets. As such, there is a risk of additional packet loss from the intra-picture. The additional packet loss can lead the decoder to issue a second intra-picture request to the encoder, which in turn, could cause additional packet loss, thereby leading the decoder to send even more requests for intra-pictures from the encoder. This can create a negative spiral of requesting intra-pictures to correct for packet loss, which only leads to additional packet loss.

[0079] L4S (Low Latency, Low Loss, Scalable)

[0080] L4S is specified in RFC 3168 “The Addition of Explicit Congestion Notification (ECN) to IP” September 2001 , which is expressly incorporated herein by reference in its entirety. As described in RFC 3168, L4S is a network feature for early network congestion indication. It is a technology used to reduce queue delay to provide low latency to IP flows and provide higher performance throughput. L4S uses Explicit Congestion Notification (ECN) marking in order to indicate congestion in the network to hosts and nodes.

[0081] Periodic Intra-Pictures and Gradual Decoding Refresh

[0082] A video bitstream, such as bitstream 30 illustrated in Figure 2, for example, may include periodic intra-coded pictures in order to tune into a bitstream or recover from lost or corrupted frames. Those of ordinary skill in the art typically refer to such periodic intra-coded pictures as “key frames.” In AVC, HEVC, and WC, such pictures are called intra-random access point (IRAP) pictures 32. As seen in Figure 2, a bitstream 30 typically includes a sequence of intercoded pictures 34 between periodic IRAP pictures 32.

[0083] AVC, HEVC, and WC also support tuning into the bitstream at an inter-coded picture using gradual decoding refresh (GDR). With GDR, each picture from the tune-in point typically refreshes a new area of the picture by coding that area with intra-coded blocks until the entire picture has been refreshed a few pictures later. One main advantage using GDR is that it allows a more even bitrate to be achieved compared to using periodic intra-pictures, such as intra- coded blocks, which typically require significantly more bits than inter-coded blocks for a given quality. GDR is supported normatively in WC using the GDR picture type, and optionally supported in AVC and HEVC using the recovery point supplemental enhancement information (SEI) message. The pictures in the video bitstream from an IRAP / GDR picture to the next IRAP / GDR picture in WC is referred to as a sequence.

[0084] Typically, periodic intra-pictures are not used in real-time video streaming applications. This is because intra-pictures generally require more data to be sent unnecessarily, even in cases where there is no packet loss. Moreover, recovery from packet loss is delayed compared to using forced intra-pictures since the decoder will have to wait for the next periodic intrapicture.

[0085] Resolution change

[0086] Typical VOD services use ABR streaming to control quality delivered to an end-user while simultaneously mitigating the effects of a varying bandwidth in the transmission channel. In ABR streaming, multiple renditions of the same content are produced at the server (e.g., network nodes, content delivery networks (CDNs), etc.). However, the renditions differ in terms of codec resolution and bitrates. In some cases, a decision on which particular resolution is fetched for playback depends on the client device. In other cases, such as in broadcast scenarios where the content is encoded at a chosen resolution, the encoder needs to select whichever resolution that is optimal for coding performance. The same is for low latency applications that use a 1 -to-1 distribution model (i.e., where content served only to a user is specific to that user and is not redistributed to other users). An example of this is cloud gaming content where each gaming session will differ depending on user interactions (e.g., input). In all these video distribution cases, where video resolution changes based on available bandwidth (either by client device or by encoding device), video can be seamlessly decoded on the client device, but only in situations where the resolution changes in the resulting bitstream do not impact the decoding process. In most cases, this requires the encoder to insert an intrapicture to signal the decoder that subsequent pictures are coded at a new resolution. However, intra-pictures are not always needed. By way of example only, a WC RPR feature allows for resolution changes without inserting an intra-picture.

[0087] While useful, the existing state-of-the-art techniques for changing resolution in the middle of a bitstream are problematic. For example, with ABR services, each segment is encoded with an intra-picture to cause the decoder to switch to a new resolution for a new segment. However, it is more inefficient to encode intra-coded pictures than it is to encode intercoded pictures. This inefficiency only increases the overall bitrate, and thus, may cause undesirable bitrate spikes in the bitstream.

[0088] Current solutions require encoders to use either full intra-pictures, which generates large bitrate spikes, or GDR, which as described previously, splits an intra-picture over several pictures. The latter provides ultra-low latency capability at the expense of longer tune-in time. For several applications, however, GDR may be too excessive, and a softer control of intra- picture size could be more beneficial.

[0089] RPR was developed for WC in order to facilitate a change in resolution of a picture in a bitstream without having to insert a new intra-coded frame. However, there is no apparent guidance in existing literature that describes how to determine RPR usage, especially in the case of low latency video, its impact on management of rate control buffers under low latency constraints, and its impact on video coding performance and quality. For cases dealing specifically with video coding performance and quality, there is no apparent discussion or solution on how to mitigate the effects of a delayed increase in bits when intra-pictures are encoded at a lower resolution. Nor does there appear to be any discussion on the impact that this has on the coding performance of the pictures that reference such an intra-picture. Additionally, existing state-of-the-art solutions do not appear to sufficiently handle refreshing a video after a packet loss or video corruption has occurred.

[0090] In more detail, existing solutions configure a decoder to send a message to an encoder requesting that it refresh a video by sending an intra-coded picture (e.g., IRAP picture). However, a problem with this technique is that the intra-coded pictures in the bitstream are typically much larger in size when compared to the inter-coded pictures of the bitstream. Such large IRAP pictures, as stated above, may cause whatever delay exists to be extended, as well as network congestion. The bitrate spikes from the IRAP pictures may be mitigated by encoding the IRAP pictures at a lower quality by increasing a quantization parameter (QP) value of the codec. However, encoding the IRAP pictures in this manner will typically have a negative impact on the quality. Additionally, a similar “bitrate spike” problem exists for solutions where periodic intrapictures (e.g., IRAP pictures) are signaled in the bitstream. Moreover, the overall bitrate of the bitstream will increase for a given quality because it is less efficient to compress (i.e., encode) intra-pictures than it is to compress inter-coded pictures.

[0091] GDR is also considered as a solution for recovery from packet loss. However, the drawback of using GDR is that the refresh is delayed. This is because a GDR-based refresh, as described above, is spread across multiple frames (i.e., pictures). Additionally, the overall compression efficiency is reduced when using GDR because periodic intra-coded blocks are still inserted in the bitstream to allow for the video to be refreshed.

[0092] Accordingly, embodiments of the present disclosure address these and other issues. Particularly, when refreshing a video of the bitstream, embodiments of the present disclosure configure an encoder to use a lower resolution when encoding a refresh picture than it does when encoding inter-coded pictures. Then, RPR may be used to predict from the refresh picture when encoding the inter-coded picture(s) that follow the refresh picture.

[0093] In one embodiment, for example, an indication to refresh the video stream (e.g., an intra request) is sent to the encoder. Upon receipt, the encoder encodes a refresh picture at a first resolution, and the following inter-coded picture(s) at a second, different resolution. The indication may be sent by a receiving device comprising the decoder, or by a network node that is operatively connected to the decoder using L4S marking.

[0094] In another embodiment, the present disclosure encodes periodic IRAP pictures at a first resolution and one or more inter-coded pictures following the IRAP pictures at a second resolution. The first resolution is lower than the second resolution, which as seen later in more detail, achieves a more uniform bit allocation between the pictures in the bitstream.

[0095] Accordingly, the present disclosure addresses the need to sufficiently handle bitrate spikes on a per-picture basis for large intra-coded pictures. Such bitrate spikes may, for instance, occur from:

[0096] • a need to refresh a video where non-periodic intra-coded pictures are requested from the encoder / transmitter;

[0097] • a need to provide a tune-in option such as by inserting periodic intra-coded pictures in the bitstream;

[0098] • a need to efficiently handle a sudden break of temporal redundancy, which can lead to inter-coded pictures being encoded with multiple intra blocks; and

[0099] • a need to change the resolution to provide a better quality tradeoff given the available bandwidth.

[0100] To handle these and other issues, the present embodiments use RPR to predict from a picture with a different resolution. Particularly, according to the present embodiments: • Intra-coded pictures are encoded at a resolution that is different (i.e., lower) than the resolution used to encode the inter-coded picture(s) that reference them in order to reduce bitrate spikes; and

[0101] • RPR is used to change the resolution of a picture when needed. For example, there may be a need to change the resolution of the pictures in a bitstream in order to maintain a desired number of bits per picture. Other reasons for such changes in resolution include, but are not limited to, allowing a temporary change in resolution due to available bandwidth, content complexity, and / or the availability of computing resources.

[0102] In one embodiment, according to the present disclosure, changing to low resolution for encoding intra-coded pictures to reduce bitrate spikes is done only in cases where the generated bits-quality make better trade off. In other cases, however, QP offset may be applied in addition to, or in lieu of, RPR. This may provide a more robust solution.

[0103] When using RPR for intra-coded pictures there may still be bitrate spikes in consecutive pictures. However, the present embodiments address this issue by applying QP offsets to deal with the bitrate spikes, and / or to introduce one or more additional intermediate resolutions for the inter-coded pictures that follow the “low-resolution” intra-coded picture.

[0104] The embodiments of the present disclosure provide advantages and benefits that conventional encoding / decoding systems and methods do not or cannot provide. For example, when compared to the state-of-the-art techniques, one advantage is that the present embodiments provide the ability to avoid bitrate spikes resulting from large intra-pictures or inter-pictures with a large proportion of intra-codec blocks. As stated above, and as explained in more detail below, this can be accomplished by reducing the resolution of selected pictures. This functionality is important because it allows applications to have more flexible control of the buffers in both the encoder and the decoder, which are required for low latency applications.

[0105] In another advantage, the techniques provided by the present embodiments provide a more robust solution to encoders. More particularly, encoders can use the resolution of a picture or group of pictures as an additional parameter (e.g., to QP, lambda, scaling matrices) to control the bits that are generated for each picture.

[0106] Encoder using RPR as a Response to an Indication to Refresh the Video

[0107] In a first embodiment of the present disclosure, an encoder uses RPR responsive to receiving an indication to refresh the video bitstream. In particular, the encoder encodes a first picture at a first resolution (e g., “resolution A”) to a CVS in the bitstream and then encodes a second picture at a second resolution B (e.g., “resolution B") to the CVS in the bitstream. According to the present disclosure, resolutions A and B are different.

[0108] The first picture at resolution A is encoded as a refresh picture, where at least a part of the picture is intra-coded. In one embodiment, the refresh picture fully refreshes the video. The refresh picture may, for example, comprise an IRAP picture. In yet another embodiment, the refresh picture initiates a refresh of the video, but the video is not fully refreshed until a subsequent picture following the refresh picture in decoding order is processed. In some embodiments, the refresh picture is a GDR picture. In other embodiments, however, the refresh picture is a set of pictures (e.g., the pictures from the GDR picture to the recovery point picture in a gradual decoding refresh).

[0109] The second picture, as stated above, follows the first picture in the decoding order and is encoded as an inter-coded picture that predicts from the first picture. In one embodiment, the second picture is encoded using RPR such that the reference picture is scaled prior to prediction.

[0110] In one preferred embodiment, at least one of the height and width of the first picture with resolution A is smaller than the corresponding height and width of the second picture with resolution B. For instance, consider an example where resolution A is full HD (i.e., 1920x1080 pixels), while resolution B is 4K (i.e., 3840x2160 pixels). Thus, resolution A is less than resolution B. By initiating a refresh of the video using an intra-picture encoded at a lower resolution (i.e., resolution A) for the refresh picture, and then increasing to resolution B to encode the subsequent inter-coded pictures, the present embodiments provide an even “perpicture” bitrate while still maintaining an overall good quality for the video. The refresh picture, which is at least partially intra-coded and less compression-efficient than the inter-coded pictures, can be encoded with a comparably lower bitrate while still maintaining a good quality (for that lower resolution). Then, the encoder can subsequently switch to encoding one or more of the inter-coded pictures that follow the refresh picture at a higher, more compression-efficient, resolution, such as resolution B or some other intermediate resolution between resolution A and resolution B.

[0111] For instance, consider a 5 Mbps bandwidth for a low latency video pipe that does not tolerate bitrate spikes at the picture level. For a 60 fps video, the size of each picture would be about 10 kB. If all the pictures in the video sequence were to be encoded at the higher resolution B (i.e., the 4K resolution), the first picture (i.e., the refresh picture) would be encoded with many undesirable artifacts and have poor quality because there are an insufficient number of bits. Therefore, the first picture would be insufficient for use as a reference picture for predicting subsequent pictures, and an encoder would have to encode multiple first pictures in order to achieve a sufficiently good quality for the video.

[0112] Now, consider a situation where, according to the present disclosure, the first picture is encoded at the lower resolution A (i.e., the full HD resolution) and the following inter-coded pictures are encoded at a higher resolution B (i.e., the 4K resolution). In these situations, the 10 kB may be sufficient enough with which to encode the first picture at a sufficiently good quality, thereby improving the quality of the subsequent inter-coded pictures predicted from the first picture. Figure 3 is a signaling diagram illustrating such an embodiment. More particularly, Figure 3 illustrates the communications between a sending device 200 (e.g., comprising an encoder) and a receiving device 300 (e.g., comprising a decoder). In this embodiment, the sending device 200 encodes a plurality of pictures at resolution B (i.e., the 4K resolution) and sends the encoded pictures to the receiving device 300 in the bitstream (boxes 42, 44, 46). As seen in Figure 3, the receiving device 300 receives pictures 1 and 3, but not picture 2. Therefore, responsive to detecting the lost packet carrying picture 2 (box 48), receiving device 300 requests that the sending device 200 refresh the video stream (box 50). Receiving device 300 may send the refresh request responsive to determining that decoding the missing / corrupted packet (i.e., picture 2) would result in a corrupted video or in undesirable artifacts appearing in the decoded video.

[0113] Responsive to receiving the refresh request, sending device 200 encodes and sends picture 4 to the receiving device (box 52). However, according to the present disclosure, sending device 200 does not encode picture 4 at the higher resolution B. Instead, sending device 200 encodes picture 4 as a refresh picture (e.g., an IRAP picture) at the lower resolution A (i.e., the full HD resolution) to send to the receiving device 300. Thereafter, sending device 200 resumes encoding one or more subsequent pictures (e.g., picture 5 and later pictures) at the higher resolution B to send to the receiving device 300 (box 54).

[0114] As stated above, encoding picture 4 as a refresh picture (e.g., an IRAP picture) at a resolution that is lower than that of subsequent pictures enables compression (i.e., encoding) with relatively good quality while simultaneously avoiding bitrate spikes. Additionally, in at least one embodiment, sending device 200 uses RPR to predict picture 5 from picture 4.

[0115] The embodiment of Figure 3 illustrates the change from resolution A to resolution B as a single “step.” However, the present disclosure is not so limited. In at least one embodiment, for example, the change of resolution may be performed in a stepwise manner over a number of pictures. Such a stepwise increase in resolution, as seen in more detail below, makes the change of resolution less noticeable to a user viewing the pictures. Additionally, a stepwise approach to increasing the resolution helps to maintain a predetermined bitrate per picture.

[0116] For instance, consider a situation where a refresh picture (e.g., an INTRA picture) is encoded at half the resolution of the inter-coded pictures that follow it (e.g., resolution A vs. resolution B). In these cases, the prediction for an inter-coded picture following the refresh picture in the decoding order would be worse than if the inter-coded picture and the refresh picture had the same resolution. This is because the refresh picture, at half the resolution of the following inter-coded pictures, does not have a sufficient number of bits to sufficiently support predicting the following inter-coded pictures. Therefore, the present embodiments employ a “stepwise” approach to changing the resolution. Specifically, the refresh picture is still encoded at the lower resolution. However, the inter-coded picture that immediately follows the refresh picture in decoding order is encoded at an intermediate resolution between that of the refresh picture and a third inter-coded picture appearing after the first and second pictures in the decoding order. By stepwise changing the resolution, as provided by the present embodiments, the relative predictions will be sufficiently accurate. Moreover, the step back to the original resolution (i.e., from the resolution of the inter-coded picture immediately following the refresh picture to the resolution of the next inter-coded picture in the decoding order) can be made without significantly diverting from a desired bitrate.

[0117] This is illustrated by graphs 60, 70, and 80 in Figures 4A-4C, which show three different cases of encoding. Pictures are indicated by rectangles labeled “I” for intra-coded picture (e.g., a refresh picture) and “B” for bi-directional predicted pictures (e.g., inter-coded pictures). The size of the rectangle illustrates the relative resolution of a picture, and the bars immediately above the “I” and “B” rectangles illustrate a bitrate for the corresponding picture.

[0118] In Figure 4A, the “I" picture has the same resolution as the “B” pictures that follow it in the decoding order. As previously described, this can cause a spike in the bitrate, as indicated by bar 62. In Figure 4B, the “I” picture is encoded at half the resolution of the “B” pictures that follow it. This approach makes it possible to encode the “I” picture with a QP and bitrate that is similar to those used to encode the “B” pictures that preceded it. However, the “B” picture immediately following the “I” picture in Figure 4B would be unable to accurately predict from the “I” picture. This also causes a bitrate spike for this picture, albeit smaller than the bitrate hike of Figure 4A where the “I” and “B” pictures had the same resolution.

[0119] In Figure 4C, the “I” picture has half the resolution of the “B” pictures that precede it in the decoding order. However, rather than encode the “B” picture 84 immediately following the “I” picture at the same resolution of the preceding “B” pictures, embodiments of the present disclosure employ the previously described “stepwise” approach and encode this “intermediate B picture” 84 at a resolution that is higher than the resolution used to encode the “I” picture, but lower than the resolution used to encode next “B” picture 86 in the decoding order. Because the “I” picture and the “intermediate B” picture 84 are closer in resolution, and because the intermediate “B” picture 84 and the next “B” picture 86 are closer in resolution, prediction from the “I” picture is enhanced. Specifically, bitrate spikes do not occur at all, or are at least significantly reduced, as shown by bar 82. Then, because of the better prediction and smaller resolution, the bitrate becomes about the same as the first B-pictures.

[0120] Figures 5A-5D are graphs 90, 100, 1 10, and 120 illustrating how the present embodiments configure a device to control the bits-per-picture using RPR. In more detail, graph 90 in Figure 5A shows an anchor, which uses a typical QP offset such as used in Joint Video Experts Team (JVET) Common Test Conditions (CTC) for low delay configuration. As seen in Figure 5A, the size of the intra-coded picture (POC 64) is significantly larger than the other pictures (i.e., POC 60, 62, 66, 68). This may require the provisioning of a larger rate control buffer. Graph 100 in Figure 5B shows how QP offset +6 helps to reduce the size of an intracoded picture by 2 but leads to increase in a consecutive picture size which compensates for worse prediction. Figure 5C illustrates a situation that is similar to that shown in Figure 5B, but with the intra-coded picture being coded with a reduced resolution (by 2x in both dimensions). Similar to Figure 5B above, Figure 50 illustrates an increase in a consecutive picture size which compensates for inadequate prediction. Figure 5D illustrates a graph 120 showing how gradual resolution changes applied on an intra-coded picture and the inter-coded pictures following it maintains the low size of the pictures and avoids the large bitrate peaks.

[0121] In one embodiment, the change in resolution is used together with other adaptive rate control tools to adapt the rate of the pictures. Such other adaptive rate control tools may be configured to change a QP (e.g., either on the block, slice, or picture level), change lambda values, and reduce a frame rate.

[0122] In another embodiment, the refresh indication to refresh the video is a request to send an intra-coded picture. For instance, the decoder (or other receiving device) can send a full intra request (FIR) message to the sending device (e.g., encoder) as specified in IETF RFC 5104 entitled “Codec Control Messages in the RTP Audio- Visual Profile with Feedback (AVPF)” published February 2008. Alternatively, the decoder (or other receiving device) can send a picture loss indication (PLI) message, or a slice loss indication (SLI) message, to the sending device (e.g., encoder) as specified in IETF RFC4585 entitled “Extended RTP Profile for Realtime Transport Control Protocol (RTCP)-Based Feedback (RTP / AVPF)” and published July 2006. Both IETF RFC 5104 and IETF RFC4585 are incorporated herein by reference in their entireties. In another example, the receiver requests the intra-coded picture in a DASH or HTTP live streaming (HLS) session.

[0123] In yet another embodiment, the indication is not to refresh the bitstream, but a request to send a CVS that starts with a refresh picture (e.g., an IRAP or GDR picture) at a first resolution A followed by a second inter-coded picture at a resolution B. As above, resolutions A and B are different, and the second picture at resolution B predicts from the refresh picture at resolution A. That is, the refresh picture is a reference picture to the second inter-coded picture. In this embodiment, the bitstream does not necessarily comprise coded video prior to the first picture sent to the receiving device.

[0124] In still another embodiment the indication to refresh the video is not sent from a receiving device 300 comprising the decoder, but from a node in network 10. The indication may for instance be signaled using L4S marking.

[0125] In at least one embodiment, no indication is sent to the sending device. More particularly, the sending device 200 (e.g., the encoder) encodes a first picture and a second picture. However, rather than begin encoding pictures responsive to receiving an explicit indication, the sending device 200 in this embodiment is configured to autonomously decide (i.e., without first receiving an indication) to encode the first picture with a resolution that is lower than the resolution used to encode the second picture.

[0126] Accordingly, an encoder configured according to the present disclosure may perform all or a subset of the following functions to encode pictures to a CVS in a bitstream.

[0127] • Receive an indication to refresh the video of the bitstream or receive an indication to start encoding video to a bitstream (i.e., to send a CVS). The indication may be a request to send an intra-coded picture, such as an IRAP picture or a GDR picture. The indication may be sent from the receiving device 200 or another network node, each of which may comprise or is associated with a decoder.

[0128] • Encode a first picture to the CVS. The first picture is encoded as a refresh picture where at least a part of the picture is intra-coded. The refresh picture may, for example, be an IRAP picture or a GDR picture. The first picture is encoded with a first resolution A, and the encoding of the first picture may be performed in response to receiving the indication.

[0129] • Encode a second picture to the CVS. The second picture follows the first picture in a decoding order and is encoded as an inter-coded picture that predicts from the first picture. The second picture is encoded with a second resolution B that is different from resolution A. Further, resolution A may be lower than resolution B where at least one of the height and the width of resolution A is smaller than the corresponding height or width of the second picture at resolution B. RPR may be used for predicting from the first picture and encoding the second picture at resolution B may also be performed as a response to receiving the indication.

[0130] • Encode a third picture to the CVS. The third picture follows the first and second pictures in decoding order and is encoded as an inter-coded picture predicted from at least one of the first and second pictures. The third picture is encoded with a third resolution “C”, where resolution C is different from both resolution A and resolution B. RPR may be used for predicting from the first picture and / or the second picture.

[0131] A decoder configured according to the present embodiments may perform all or a subset of the following functions to decode pictures from a CVS in a bitstream.

[0132] • Detect that a video packet has been lost or detect that one or more pictures are corrupt.

[0133] • Send an indication to the encoder to refresh the video of the bitstream or send an indication to the encoder to start encoding video to a bitstream (i.e., to send a CVS). The indication may be a request to send an intra-coded picture, such as an IRAP picture or a GDR picture. The sending of the indication may be done as a response to detecting that a video packet was lost or to detecting that one or more pictures are corrupt. • Receive a bitstream comprising a CVS.

[0134] • Obtain a first coded picture from the CVS.

[0135] • Decode the first coded picture. In this embodiment, the first coded picture has been encoded as a refresh picture, which as stated above, at least a part of the picture is intra-coded. The refresh picture may, for instance, be an IRAP picture or a GDR picture, and has a first resolution A.

[0136] • Obtain a second coded picture from the CVS.

[0137] • Decode the second coded picture. The second coded picture follows the first coded picture in decoding order and is decoded as an inter-coded picture predicted from the first coded picture. The second coded picture has a second resolution B that is different from resolution A. Additionally, resolution A may be lower than resolution B, and at least one of the height and the width of the first coded picture at resolution A is smaller than the corresponding height or width of the second coded picture at resolution B. As previously stated, RPR may be used for predicting from the first picture.

[0138] • Obtain a third coded picture from the CVS.

[0139] • Decode the third coded picture. The third coded picture follows the first and second coded pictures in the decoding order and is decoded as an inter-coded picture predicted from at least one of the first and second coded pictures. The third coded picture has a third resolution C that is different from both resolution A and resolution B. Additionally, RPR may be used to predict from the first and / or second coded pictures.

[0140] Indication

[0141] In at least one embodiment, the refresh indication sent to the sending device 200 by receiving device 300 may comprise one or more of the following:

[0142] • A request to send an intra-coded picture, e.g., an IRAP picture, to directly refresh the video;

[0143] • A request to gradually refresh the video (e.g., a request to send a GDR picture);

[0144] • A specified highest resolution;

[0145] • A specified lowest resolution;

[0146] • A request or allowance to use RPR for prediction;

[0147] • A preferred bitrate per picture;

[0148] • A preferred bitrate per second;

[0149] • A maximum bitrate per picture;

[0150] • A maximum bitrate per second;

[0151] • A request or an assumption that the encoder sets the resolution based on a specific bitrate; • A request to have a lower resolution for intra pictures and a higher resolution for inter-coded pictures;

[0152] • A request to start with a low resolution and then increase to a higher resolution; and

[0153] • A request to start with a low resolution and then incrementally increase to a higher resolution over a number of pictures.

[0154] Periodic intra refresh with RPR

[0155] As stated above, embodiments of the present disclosure mitigate bitrate spikes. For example, when an inter-coded picture is lost (e.g., one or more incoming packets at the are dropped and / or corrupted), a sending device configured according to the present embodiments refreshes the video at the next intra-coded picture. As previously described, the sending device can accomplish this function either autonomously or in response to receiving an indication from receiving device 300 or a network node. However, to achieve successful prediction from the intra-coded pictures, at least some of the inter-coded pictures use RPR.

[0156] For example, in one embodiment, an encoder configured according to the present disclosure (e.g., at sending device 200) periodically encodes one or more intra-pictures (e.g., IRAP pictures) to the CVS of the bitstream. Such pictures may be encoded at regular timed intervals (e.g., every 10 seconds), autonomously as needed, and / or at variable positions in the bitstream (e.g. at scene cuts). Regardless of the periodicity, however, the encoder in this embodiment encodes the intra-coded pictures at a resolution that is lower than the resolution at which it encodes the inter-coded pictures to the CVS of the bitstream.

[0157] Figure 6 illustrates one such embodiment. As seen in Figure 6, CVS 130 comprises a plurality of IRAP pictures 132 and one or more inter-coded pictures 134 disposed between the IRAP pictures 132. Further, as indicated by the different sizes of the pictures 132, 134, the IRAP pictures 132 are encoded with a resolution that is lower than the resolution at which it encodes the inter-coded pictures 134.

[0158] In one embodiment, the intra-coded pictures (e.g., IRAP pictures 132) are decoded by the receiving device 300 but are never displayed. That is, the decoder at receiving device 300 only uses the intra-coded pictures in the CVS as reference pictures for prediction, but does not output the decoded intra-coded pictures for further processing and / or display. However, because the lower resolution video (i.e., the lower resolution intra-coded pictures such as IRAP pictures 132) are not displayed, the present embodiments improve the overall quality of the video.

[0159] Additionally, as seen in Figure 7, for example, the encoder may be configured to encode an inter-coded picture 136 appearing immediately after an IRAP picture 132 in CVS 130 at a reduced resolution. By way of example only, the encoder at the sending device 200 may encode an IRAP picture 132 at half the normal resolution of the inter-coded pictures 134. Then, the encoder may encode an inter-coded picture appearing immediately after the IRAP picture 132 at an intermediate resolution that is higher than that of IRAP picture 132 and lower than that of an inter-coded picture 134. As stated above, this “stepwise” approach to changing the resolutions of the pictures encoded to a CVS of the bitstream helps mitigate spikes in the bitrate.

[0160] Figure 8 is a flow diagram illustrating a method 140 for encoding pictures into a CVS 130 according to embodiments of the present disclosure. For illustrative purposes only, method 140 is implemented by a sending device 200, such as a network node in cloud 14. However, those of ordinary skill in the art will readily appreciate that method 140 may be implemented at devices other than a network node, such as computing device 20 seen in Figure 1 , for example.

[0161] As seen in Figure 8, sending device 200 receives a refresh indication to refresh a video prior to encoding a first picture (box 142). In response to receiving the refresh indication, sending device 200 generates a CVS 130 in a bitstream (box 144) by encoding a first picture to the CVS in the bitstream at a first resolution (box 144a) and a second picture to the CVS in the bitstream at a second resolution (box 144b). In this embodiment, the first picture is encoded as a refresh picture and at least part of the first picture is intra-coded. Additionally, the first resolutions is different from the second resolution. Further, the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture.

[0162] In at least one embodiment, sending device 200 encodes a third picture to the CVS in the bitstream at a third resolution that is different from both the first and second resolutions (box 144c). The third picture follows the first and the second pictures in the decoding order and is encoded as an inter-coded picture by predicting from one or both of the first and second pictures. So encoded, the sending device 200 sends the CVS of the bitstream to a receiving device 300 (box 146).

[0163] In one embodiment, the refresh indication is received from the receiving device.

[0164] In another embodiment, the refresh indication is signaled by a network node.

[0165] In one embodiment, the refresh indication is based on Low Latency, Low Loss, and Scalable (L4S) marking.

[0166] In one embodiment, the bitstream is autonomously generated independently of receiving a refresh indication.

[0167] In one embodiment, the CVS in the bitstream appears after an initial CVS in the bitstream.

[0168] In one embodiment, the second picture is encoded using reference picture resampling (RPR) such that a reference picture used to predict the second picture is scaled prior to encoding the second picture.

[0169] In one embodiment, the reference picture is the first picture.

[0170] In one embodiment, a change from the first resolution to the second resolution when encoding the first and second pictures adapts a bitrate or a quality of at least one of the first picture, the second picture, and one or more subsequent pictures. In one embodiment, the bitrate or the quality of the at least one of the first picture, the second picture, and the one or more subsequent pictures is further adapted based on one or more adaptive rate control tools.

[0171] In one embodiment, the one or more adaptive rate control tools are configured to change a quantization parameter (QP) on one of a block level, a slice level, and a picture level, change one or more lambda values, and reduce a frame rate.

[0172] In one embodiment, the first picture is periodically encoded at a resolution that is lower than a resolution of inter-coded pictures in the bitstream.

[0173] In one embodiment, a position at which the first picture is encoded in the bitstream is variable.

[0174] In one embodiment, the receiving device comprises a decoder.

[0175] In another embodiment, however, the receiving device comprises a network node.

[0176] Figure 9 is a flow diagram illustrating a method 150 for decoding pictures from a CVS 130 according to embodiments of the present disclosure. It can be noted here that this embodiment of the present disclosure implements method 150 at a receiving device 300 that comprises a decoder. However, this is for illustrative purposes only. Those of ordinary skill in the art will readily understand that the receiving device 300 implementing method 150 may be a device such as computing device 20 in a network.

[0177] As seen in Figure 9, receiving device 300 detects that a video packet of the video has been lost or that one or more pictures in the video are corrupt (box 152). Responsive to the detection, receiving device 300 sends a refresh indication to refresh the video of the bitstream to the sending device 200 (box 154). The receiving device 300 then receives a bitstream from sending device 200 (box 156). In this embodiment, the bitstream comprises the CVS 130 that includes the first and second pictures.

[0178] So received, receiving device 300 decodes first and second pictures in the CVS of the bitstream (box 158). In more detail, receiving device 300 decodes a first picture from a CVS 130 in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution (box 158a). Receiving device 300 also decodes a second picture from CVS 130 in the bitstream (box 158b). In this embodiment, the second picture follows the first picture in decoding order, has a second resolution that is different from the first resolution, and is decoded as an inter-coded picture predicted from the first picture. In some embodiments, receiving device 300 is also configured to decode a third picture from the CVS 130 in the bitstream (box 158c). In these embodiments, the third picture follows the first and the second pictures in the decoding order, is decoded as an inter-coded picture by predicting from one or both of the first and second pictures, and is encoded at a third resolution that is different from both the first and second resolutions.

[0179] So decoded, receiving device 300 outputs the decoded pictures for further processing, such as for display or storage, for example (box 160). However, in at least one embodiment, receiving device 300 is precluded from sending a decoded first picture (e.g., a low-resolution intra-coded picture) to a display device for display to a user (box 162).

[0180] In one embodiment, outputting the decoded second picture comprises sending the decoded second picture to a display device for display.

[0181] In another embodiment, however, outputting the decoded second picture comprises sending the decoded second picture to a storage device for storage.

[0182] In one embodiment, the refresh indication is sent prior to receiving the CVS that includes the first and second pictures.

[0183] Regardless of whether the present embodiments are implemented by sending device 200 or receiving device 300, however, the refresh indication, in at least one embodiment, comprises a request for one of a full intra request (FIR) message, a picture loss indication (PLI) message, and a slice loss indication (SLI) message.

[0184] In one embodiment, the refresh indication requests the first picture in a Real-time Transport Protocol (RTP) session, a Dynamic Adaptive Streaming over Hypertext Transport Protocol (HTTP) (DASH) session or a HTTP Live Streaming (HLS) session.

[0185] In one embodiment, the refresh indication comprises a request for the sending device to refresh a video of the bitstream.

[0186] In one embodiment, the refresh indication comprises a request to begin encoding the video to the CVS in the bitstream.

[0187] In one embodiment, the refresh indication comprises a request to send an intra-coded picture.

[0188] In one embodiment, the refresh indication comprises a request to send the CVS in the bitstream.

[0189] In one embodiment, the refresh picture is an Intra Random Access Pictures (IRAP) picture.

[0190] In one embodiment, the refresh picture is a Gradual Decoding Refresh (GDR) picture.

[0191] In one embodiment, the second picture is predicted from the first picture using Reference Picture Resampling (RPR).

[0192] In one embodiment, the third picture is predicted from one or both of the first and second pictures using RPR.

[0193] In one embodiment, the first resolution is lower than the second resolution.

[0194] In one embodiment, the second resolution is lower than the third resolution.

[0195] In one embodiment, at least one of a height and a width of the first picture with the first resolution is smaller than a corresponding height and width of the second picture with the second resolution.

[0196] In one embodiment, the refresh indication comprises one or more of a request to send an intra-coded picture that directly refreshes the video, a request to gradually refresh the video, a specified highest resolution, a specified lowest resolution, a request or grant to use RPR for prediction, a preferred bitrate per picture, a preferred bitrate per second, a maximum bitrate per picture, a maximum bitrate per second, a request for the encoder to set a resolution based on a specified bitrate, an indication indicating an assumption that the encoder sets a resolution based on a specified bitrate, a request to encode intra-pictures at a resolution that is lower than the resolution for encoding inter-coded pictures, a request to encode inter-pictures at a resolution that is higher than the resolution for encoding intra-coded pictures, a request to start the encoding at an initial resolution and increase to a successive resolution that is higher than the initial resolution, and a request to start the encoding at the initial resolution and incrementally increase the resolution over a number of pictures to the successive resolution.

[0197] In one embodiment, the first picture fully refreshes the video.

[0198] In one embodiment, the first picture initiates a refresh of the video, and wherein a subsequent picture following the first picture in the decoding order completes the refresh of the video.

[0199] In one embodiment, the first picture comprises a plurality of refresh pictures.

[0200] In one embodiment, the plurality of refresh pictures comprises a plurality of pictures comprising a Gradual Decoding Refresh (GDR) picture and a recovery point picture.

[0201] In one embodiment, the plurality of refresh pictures further comprises one or more pictures between the GDR picture and the recovery point picture.

[0202] An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0203] Figure 10 is a functional block diagram illustrating some exemplary components of a sending device 200 configured according to embodiments of the present disclosure. As seen in Figure 10, sending device 200 comprises a physical computing node having physical circuitry and can be configured to implement one or more of the encoding functions, as previously described. In some cases, there are multiple sending devices 200 configured to implement one or more of the encoding functions. Thus, according to the present disclosure, there may be multiple sending devices 200 distributed throughout the communication network (e.g., the cloud) configured to execute the previously described functions.

[0204] As seen in Figure 10, sending device 200 comprises communication circuitry 202, processing circuitry 204, an encoder 206, and memory 208. The communication circuitry 202 comprises the hardware required for communicating directly or indirectly with one or more other network nodes, as well as with a receiving device 300, such as a user device. In this regard, the communication circuitry 202 may comprise a Network Interface Circuit (NIC) (e.g., an ETHERNET or similar interface). In some embodiments, the sending device 200 may be configured for wireless communications, and thus, communication circuitry 202 would comprise the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, sending device 200 may also be coupled to one or more antenna (not shown). In other embodiments, however, sending device 200 is configured to communicate via a wire interface.

[0205] The processing circuitry 204 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the sending device 200. In accordance with the present disclosure, processing circuitry 204 may comprise the encoder 208 and can be configured by software to perform one or more of the methods herein described including method 140 seen in Figure 8.

[0206] Memory 208 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 204 for operation. Memory 208 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 208 stores a computer program 210 comprising executable instructions that configure the processing circuit 204 in the sending device 200 to perform one or more of the methods herein described including method 140 seen in Figure 8. A computer program 210 in this regard may comprise one or more code modules corresponding to the means or units described above.

[0207] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 210 for configuring the processing circuitry 204 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 210 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0208] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0209] Embodiments of the present disclosure further include a carrier containing such a computer program 210. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0210] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0211] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by sending device 200. This computer program product may be stored on a computer readable recording medium.

[0212] Figure 1 1 is a functional block diagram illustrating some exemplary components of a receiving device 300 configured according to embodiments of the present disclosure. As seen in Figure 1 1 , receiving device 300 comprises a physical computing node having physical circuitry and can be configured to implement one or more of the decoding functions, as previously described. In some cases, there are multiple receiving devices 300 configured to implement one or more of the decoding functions. Thus, according to the present disclosure, there may be multiple receiving devices 300 distributed throughout the communication network (e.g., the cloud) configured to execute the previously described decoding functions.

[0213] As seen in Figure 11 , receiving device 300 comprises communication circuitry 302, processing circuitry 304, a decoder 306, and memory 308. The communication circuitry 302 comprises the hardware required for communicating directly or indirectly with one or more other network nodes, as well as with a sending device 200. In this regard, the communication circuitry 302 may comprise a Network Interface Circuit (NIC) (e.g., an ETHERNET or similar interface). In some embodiments, the receiving device 300 may be configured for wireless communications, and thus, communication circuitry 302 would comprise the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, receiving device 300 may also be coupled to one or more antenna (not shown). In other embodiments, however, receiving device 300 is configured to communicate via a wire interface.

[0214] The processing circuitry 304 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the receiving device 300. In accordance with the present disclosure, processing circuitry 304 may comprise the decoder 308 and can be configured by software to perform one or more of the methods herein described including method 150 seen in Figure 9. Memory 308 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 304 for operation. Memory 308 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 308 stores a computer program 310 comprising executable instructions that configure the processing circuit 304 in the receiving device 300 to perform one or more of the methods herein described including method 150 seen in Figure 9. A computer program 310 in this regard may comprise one or more code modules corresponding to the means or units described above.

[0215] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 310 for configuring the processing circuitry 304 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 310 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0216] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. As stated above, a computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0217] Embodiments of the present disclosure further include a carrier containing such a computer program 310. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0218] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0219] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by receiving device 300. This computer program product may be stored on a computer readable recording medium.

[0220] The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended embodiments are intended to be embraced therein.

Claims

CLAIMS1 . A method (140), implemented by a sending device (200) having an encoder (206), for encoding pictures to a coded video sequence (CVS) in a bitstream (130), the method comprising: generating (144) the CVS in the bitstream, the generating comprising: encoding (144a) a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded; and encoding (144b) a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture; and sending (146) the CVS of the bitstream to a receiving device (20, 300).

2. The method of claim 1 , further comprising: receiving (142) a refresh indication to refresh a video prior to encoding the first picture; and generating the CVS in the bitstream responsive to receiving the refresh indication.

3. The method of claim 2, wherein receiving the refresh indication comprises one or both of: receiving a refresh indication from the receiving device; and receiving a refresh indication signaled by a network node (200).

4. The method of any of claims 2-3, wherein the refresh indication is based on Low Latency, Low Loss, and Scalable (L4S) marking.

5. The method of any of claims 1 -4, wherein the CVS in the bitstream appears after an initial CVS in the bitstream.

6. The method of any of claims 1-5, wherein generating the CVS in the bitstream further comprises encoding (144c) a third picture to the CVS in the bitstream at a third resolution that is different from both the first and second resolutions, wherein the third picture: follows the first and the second pictures in the decoding order; and is encoded as an inter-coded picture by predicting from one or both of the first and second pictures.

7. The method of any of claims 1-6, wherein the second picture is encoded using reference picture resampling (RPR) such that a reference picture used to predict the second picture isscaled prior to encoding the second picture and wherein the first picture is the reference picture used to predict the second picture.

8. The method of any of claims 1-7, wherein a change from the first resolution to the second resolution when encoding the first and second pictures adapts a bitrate or a quality of at least one of the first picture, the second picture, and one or more subsequent pictures.

9. The method of claim 8, wherein the bitrate or the quality of the at least one of the first picture, the second picture, and the one or more subsequent pictures is further adapted based on one or more adaptive rate control tools, wherein the one or more adaptive rate control tools are configured to: change a quantization parameter (QP) value on at least one of a block level, a slice level, and a picture level; change one or more lambda values; and / or reduce a frame rate.

10. The method of any of claims 1-9, wherein the first picture is periodically encoded at a resolution that is lower than a resolution of inter-coded pictures in the bitstream and / or a position at which the first picture is encoded in the bitstream is variable.11 . The method of any of claims 1-10, wherein the receiving device comprises one or both of a decoder and a network node.

12. A method (150), implemented by a receiving device (300) having a decoder (306), for decoding pictures from a coded video sequence (CVS) in a bitstream, the method comprising: decoding (158a) a first picture from the CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution; decoding (158b) a second picture from the CVS in the bitstream, wherein the second picture: follows the first picture in decoding order; has a second resolution that is different from the first resolution; and is decoded as an inter-coded picture predicted from the first picture; and outputting (160) the decoded second picture.

13. The method of claim 12, wherein outputting the decoded second picture further comprises one or both of: sending the decoded second picture to a display device for display; and sending the decoded second picture to a storage device for storage.

14. The method of any of claims 12-13, further comprising receiving (156) the bitstream from a sending device, wherein the bitstream comprises the CVS that includes the first and second pictures.

15. The method of any of claims 12-14, further comprising sending (154) a refresh indication to the sending device.

16. The method of claim 15, wherein the refresh indication is sent prior to receiving the CVS that includes the first and second pictures.

17. The method of any of claims 12-16, further comprising: detecting (152) that a video packet of the video has been lost or that one or more pictures in the video are corrupt; and sending (154) the refresh indication to refresh the video of the bitstream to the sending device responsive to the detecting.

18. The method of any of claims 12-17, further comprising decoding (158c) a third picture from the CVS in the bitstream, wherein the third picture: follows the first and the second pictures in the decoding order; is decoded as an inter-coded picture by predicting from one or both of the first and second pictures; and is encoded at a third resolution that is different from both the first and second resolutions.

19. The method of any of claims 12-18, further comprising precluding sending (162) the decoded first picture to the display device for display.

20. The method of any of claims 1-19, wherein the refresh indication comprises a request for one of: a full intra request (FIR) message; a picture loss indication (PLI) message; and a slice loss indication (SLI) message.21 . The method of any of claims 1-20, wherein the refresh indication requests the first picture in a Real-time Transport Protocol (RTP) session, a Dynamic Adaptive Streaming over Hypertext Transport Protocol (HTTP) (DASH) session or a HTTP Live Streaming (HLS) session.

22. The method of any of claims 1-21 , wherein the refresh picture is an Intra Random Access Pictures (IRAP) picture or a Gradual Decoding Refresh (GDR) picture.

23. The method of any of claims 1-22, wherein one or both of: the second picture is predicted from the first picture using Reference Picture Resampling (RPR); and the third picture is predicted from one or both of the first and second pictures using RPR.

24. The method of any of claims 1-23, wherein one or more of: the first resolution is lower than the second resolution; the second resolution is lower than the third resolution; and at least one of a height and a width of the first picture with the first resolution is smaller than a corresponding height and width of the second picture with the second resolution.

25. The method of any of claims 1-24, wherein the refresh indication comprises one or more of: a request for the sending device to refresh a video of the bitstream; a request to begin encoding the video to the CVS in the bitstream; a request to send an intra-coded picture; a request to send the CVS in the bitstream; a request to send an intra-coded picture that directly refreshes the video; a request to gradually refresh the video; a specified highest resolution; a specified lowest resolution; a request or grant to use RPR for prediction; a preferred bitrate per picture; a preferred bitrate per second; a maximum bitrate per picture; a maximum bitrate per second; a request for the encoder to set a resolution based on a specified bitrate; an indication indicating an assumption that the encoder sets a resolution based on a specified bitrate; a request to encode intra- pictures at a resolution that is lower than the resolution for encoding inter-coded pictures; a request to encode inter-pictures at a resolution that is higher than the resolution for encoding intra-coded pictures; a request to start the encoding at an initial resolution and increase to a successive resolution that is higher than the initial resolution; anda request to start the encoding at the initial resolution and incrementally increase the resolution over a number of pictures to the successive resolution.

26. The method of any of claims 1-25, wherein the first picture fully refreshes the video or the first picture initiates a refresh of the video, and wherein a subsequent picture following the first picture in the decoding order completes the refresh of the video.

27. The method of any of claims 1-26, wherein the first picture comprises a plurality of refresh pictures comprising a Gradual Decoding Refresh (GDR) picture and a recovery point picture.

28. A sending device (200) for encoding pictures to a coded video sequence (CVS) in a bitstream (130), the sending device configured to: generate (144) the CVS in the bitstream wherein, to generate the CVS, the sending device is configured to: encode (144a) a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded; and encode (144b) a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture; and send (146) the CVS of the bitstream to a receiving device.

29. The sending device of claim 28, wherein the encoder is further configured to perform the method according to any one of claims 2-11 and 15-27.

30. A sending device (200) for encoding pictures to a coded video sequence (CVS) in a bitstream (130), the sending device comprising: communications circuitry (202) configured to communicate with a receiving device (300); and processing circuitry (204) operatively connected to the communications circuitry and configured to: generate (144) the CVS in the bitstream wherein, to generate the CVS, the processing circuitry is configured to: encode (144a) a first picture to the CVS in the bitstream at a first resolution, wherein the first picture is encoded as a refresh picture and wherein at least part of the first picture is intra-coded; andencode (144b) a second picture to the CVS in the bitstream at a second resolution that is different from the first resolution, wherein the second picture follows the first picture in decoding order and is encoded as an inter-coded picture by predicting from the first picture; and send (146) the CVS of the bitstream to a receiving device (300).31 . The sending device of claim 30, wherein the processing circuitry is further configured to perform the method according to any one of claims 2-11 and 15-27.

32. A computer program (210) comprising instructions that, when executed on processing circuitry of a sending device (200), causes the sending device to perform the method according to any of claims 1-11 and 15-27.

33. A carrier comprising the computer program (210) of claim 32, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

34. A non-transitory computer-readable storage medium (208) comprising a computer program (210) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (204) in a sending device (200), causes the sending device to perform the method of any of claims 1-1 1 and 15-27.

35. A receiving device (300) having a decoder (306) for decoding pictures from a coded video sequence (CVS) in a bitstream (130), the receiving device configured to: decode (158a) a first picture from a CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution; decode (158b) a second picture from the CVS in the bitstream, wherein the second picture: follows the first picture in decoding order; has a second resolution that is different from the first resolution; and is decoded as an inter-coded picture predicted from the first picture; and output (160) the decoded second picture.

36. The receiving device of claim 35, wherein the receiving device is further configured to perform the method according to any one of claims 12-27.

37. A receiving device (300) for decoding pictures from a coded video sequence (CVS) in a bitstream (130), the receiving device comprising: communications circuitry (302) configured to communicate with a sending device (200); andprocessing circuitry (304) operatively connected to the communications circuitry and configured to: decode (158a) a first picture from the CVS in the bitstream, wherein the first picture was encoded as a refresh picture and has a first resolution; decode (158b) a second picture from the CVS in the bitstream, wherein the second picture: follows the first picture in decoding order; has a second resolution that is different from the first resolution; and is decoded as an inter-coded picture predicted from the first picture; and output (160) the decoded second picture.

38. The receiving device of claim 37, wherein the processing circuitry is further configured to perform the method according to any one of claims 12-27.

39. A computer program (310) comprising instructions that, when executed on processing circuitry (304) of a receiving device (300) having a decoder (306), causes the receiving device to perform the method according to any of claims 12-27.

40. A carrier comprising the computer program of claim 39, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

41. A non-transitory computer-readable storage medium (308) comprising a computer program (310) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (304) of a receiving device (300) having a decoder (306), causes the receiving device to perform the method of any of claims 12-27.

Images