Intra coded picture refresh with reference picture resampling (RPR)

Abstract

The encoder (206) encodes pictures into a coded video sequence (CVS) of a bitstream (130). More specifically, the encoder encodes an intra-coded picture at a lower resolution than a resolution used to encode one or more inter-coded pictures that follow the intra-coded picture in the CVS. The encoded pictures are then sent to the decoder (306). Upon receipt, the decoder decodes the pictures from the CVS, where the intra-coded picture is encoded at a lower resolution than a resolution used to encode the one or more inter-coded pictures that follow the intra-coded picture in the CVS.

Description

Technical Field

[0001] This application generally relates to video encoding and decoding technologies, and more specifically to refresh technologies for refreshing video in a bitstream. Background Technology

[0002] Using a video codec, an encoder compresses a video stream into an encoded video bitstream before sending the bitstream over a network to a receiving device. This compression process is commonly referred to as "encoding." Upon reception, the receiving device uses a video codec to decompress the encoded video bitstream before forwarding the video for further processing and / or display to the user. This decompression process is commonly referred to as "decoding." Typically, a video stream can be encoded and decoded using any required or desired video codec. However, some examples of widely deployed video codecs include those configured to operate according to H.264 / Advanced Video Coding (AVC), H.265 / High-Efficiency Video Coding (HEVC), and / or H.266 / Various Video Coding (VVC), each of which was jointly developed and standardized by the Moving Picture Experts Group (MPEG) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T).

[0003] However, regardless of the specific standard, video codecs typically support both intra-frame coded pictures and inter-frame coded pictures. Intra-frame coded pictures can be predicted based on samples of the same picture only, while inter-frame coded pictures can also be predicted based on previously decoded pictures called "reference pictures." Inter-frame coded pictures can be further divided into "P-pictures" and bidirectional "B-pictures." A "P-picture" can only be predicted from one reference picture at a time for each coding block, while a bidirectional "B-picture" can be predicted from up to two reference pictures simultaneously for each coding block.

[0004] During the encoding process, a conventional video codec determines the difference between the pixel data of a first image (e.g., a reference image) and the pixel data of a second subsequent image that uses the first image as a reference image. This difference is often referred to as the "residual" and is transformed to the frequency domain by the encoder. The transformed residual is then quantized and entropy-coded before being passed to the decoder along with one or more necessary entropy-coded prediction parameters (e.g., prediction mode and motion vectors). The quantization level is controlled by the quantization parameter (QP) value. The decoder performs the reverse process. That is, the decoder applies inverse quantization and inverse transform to the received bitstream to obtain the residual. After obtaining the residual, the decoder adds it to intra-frame prediction or inter-frame prediction to reconstruct the image. Summary of the Invention

[0005] This disclosure provides methods and corresponding apparatus for encoding and decoding CVS bitstreams. More specifically, this embodiment encodes / decodes refresh images at a resolution lower than the resolution of one or more inter-coded images(s) used to encode the following refresh image. Reducing the refresh image resolution in this way avoids, or at least significantly reduces, the bitrate spikes associated with large intra-coded images and / or inter-coded images with large proportions of intra-coder blocks.

[0006] Therefore, in a first aspect, this disclosure provides a method implemented by an encoder for encoding an image into a coded video sequence (CVS) in a bitstream. The method includes generating the CVS in the bitstream. In this aspect, the method includes: encoding a first image into a CVS in the bitstream at a first resolution, wherein the first image is encoded as a refresh image, and wherein at least a portion of the first image is intra-coded; encoding a second image into a CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in decoding order, and the second image is encoded as an inter-coded image by prediction from the first image; and transmitting the CVS of the bitstream to a receiving device.

[0007] In a second aspect, this disclosure provides a method implemented by a decoder for decoding images from a coded video sequence (CVS) in a bitstream. In this embodiment, the method includes: decoding a first image from the CVS in the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; decoding a second image from the CVS in the bitstream, wherein the second image follows the first image in decoding order, the second image has a second resolution different from the first resolution, and the second image is decoded as an inter-frame coded image predicted based on the first image. After such decoding, the method further includes outputting the decoded second image.

[0008] In a third aspect, this disclosure provides an encoder for encoding images into coded video sequences (CVS) in a bitstream. In this embodiment, the encoder is configured to generate a CVS in the bitstream. To generate a CVS, the encoder is configured to: encode a first image into a CVS in the bitstream at a first resolution, wherein the first image is encoded as a refreshed image and wherein at least a portion of the first image is intra-coded; and encode a second image into a CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in decoding order and is encoded as an inter-coded image by prediction from the first image. After such encoding, the encoder transmits the CVS of the bitstream to a receiving device.

[0009] In a fourth aspect, this disclosure provides an encoder for encoding images into coded video sequences (CVS) in a bitstream. In this aspect, the encoder includes communication circuitry configured to communicate with a receiving device, and processing circuitry operatively connected to the communication circuitry. In this aspect, the processing circuitry is configured to generate a CVS in the bitstream by: encoding a first image into a CVS in the bitstream at a first resolution, wherein the first image is encoded as a refresh image and wherein at least a portion of the first image is intra-frame encoded; and encoding a second image into a CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in decoding order, and the second image is encoded as an inter-frame encoded image by prediction from the first image. Once encoded, the encoder of this aspect transmits the CVS of the bitstream to the receiving device.

[0010] In a fifth aspect, this disclosure provides a non-transitory computer-readable storage medium including a computer program stored thereon, the computer program including executable instructions that, when executed by processing circuitry in an encoder, cause the encoder to implement the method of the first aspect.

[0011] In a sixth aspect, this disclosure provides a decoder for decoding images from a coded video sequence (CVS) in a bitstream. In this aspect, the decoder is configured to: decode a first image from the CVS in the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; decode a second image from the CVS in the bitstream, wherein the second image follows the first image in decoding order, the second image has a second resolution different from the first resolution, and the second image is decoded as an inter-frame coded image predicted based on the first image. After such decoding, the decoder outputs the decoded second image.

[0012] In a seventh aspect, this disclosure provides a decoder for decoding images from a coded video sequence (CVS) in a bitstream. In this aspect, the decoder includes communication circuitry configured to communicate with a transmitting device, and processing circuitry operatively connected to the communication circuitry. The processing circuitry is configured to: decode a first image from the CVS in the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; decode a second image from the CVS in the bitstream, wherein the second image follows the first image in decoding order, the second image has a second resolution different from the first resolution, and the second image is decoded as an inter-frame coded image predicted based on the first image. After such decoding, the processing circuitry outputs the decoded second image.

[0013] In an eighth aspect, this disclosure provides a non-transitory computer-readable storage medium including a computer program stored thereon, the computer program including executable instructions that, when executed by processing circuitry in a decoder, cause the decoder to implement the method of the second aspect. Attached Figure Description

[0014] Figure 1 This is a functional block diagram illustrating a communication network configured according to embodiments of the present disclosure.

[0015] Figure 2 An image of a periodic intra-frame random access point (IRAP) in a bitstream according to an embodiment of the present disclosure is shown.

[0016] Figure 3 This is a signaling diagram illustrating signaling for refreshing an image of a bitstream according to an embodiment of this disclosure.

[0017] Figure 4A This is a graph showing the results of resampling intra-coded images without using reference image resampling (RPR) according to embodiments of the present disclosure.

[0018] Figure 4B This is a graph showing the results of using RPR on intra-frame coded pictures according to embodiments of the present disclosure.

[0019] Figure 4C This is a graph showing the results of using the 2-step RPR method on intra-frame coded pictures according to embodiments of the present disclosure.

[0020] Figure 5A , Figure 5B , Figure 5C and Figure 5D This is a graph illustrating different methods for processing intra-frame sizes according to embodiments of the present disclosure.

[0021] Figure 6 The illustration shows periodic IRAP images in a bitstream according to an embodiment of the present disclosure, wherein the IRAP images are encoded at a lower resolution than the inter-frame coded images between them.

[0022] Figure 7 A step-by-step method for encoding a CVS according to an embodiment of the present disclosure is shown, wherein periodic IRAP images are encoded into a CVS bitstream at a lower resolution than the inter-frame coded images set between them, and wherein an inter-frame coded image that immediately follows an IRAP image is encoded into a CVS including an IRAP image at an intermediate resolution between the resolution of the IRAP image and at least one other inter-frame coded image in the CVS.

[0023] Figure 8This is a flowchart illustrating a method for encoding an image into CVS according to an embodiment of the present disclosure.

[0024] Figure 9 This is a flowchart illustrating a method for decoding an image from CVS according to an embodiment of the present disclosure.

[0025] Figure 10 This is a functional block diagram illustrating some exemplary components of a transmitting device (e.g., an encoder) configured according to embodiments of the present disclosure.

[0026] Figure 11 This is a functional block diagram illustrating some exemplary components of a receiving device (e.g., a decoder) configured according to embodiments of the present disclosure. Detailed Implementation

[0027] This application claims the benefit of U.S. Provisional Application No. 63 / 632614, filed April 11, 2024, the entire disclosure of which is incorporated herein by reference.

[0028] This disclosure provides a technique and corresponding apparatus for encoding a refresh picture at a lower resolution than the resolution of one or more inter-coded pictures(s) used to encode a follow-up refresh picture. Reducing the refresh picture resolution in this way avoids, or at least significantly reduces, the bitrate spikes associated with large intra-coded pictures and / or inter-coded pictures with large proportions of intra-codec blocks.

[0029] More specifically, this disclosure provides a technique and corresponding apparatus for encoding images into a bitstream of a coded video sequence (CVS) and decoding images from the CVS. More specifically, when refreshing the video bitstream, a transmitting device (e.g., an encoder) configured according to this embodiment encodes the refresh image at a resolution lower than that used to encode one or more inter-frame coded images following the refresh image. When encoding the inter-frame coded images, the transmitting device can use reference image resampling (RPR) to predict based on the refresh image. A receiving device (e.g., a decoder) decodes the coded image from the CVS in the bitstream. The receiving device is specifically configured to request the refresh image from the transmitting device and, upon receiving the bitstream, use the refresh image as a reference image to decode the image from the CVS.

[0030] Now turn to the attached diagram. Figure 1 This is a functional block diagram illustrating a communication network 10 configured according to an embodiment of the present disclosure. Figure 1As shown, network 10 includes an access network 12 that communicatively connects client device 300 (e.g., HMD, mobile phone, or computer) to network nodes 200 (e.g., server nodes) deployed in cloud network 14. In some embodiments, computing device 20 may be placed between client device 300 and server node 200 and configured to perform at least some processing functions of client device 300.

[0031] Access network 12 can be any type of communication network (e.g., Wi-Fi, Ethernet, WLAN, WCDMA, LTE, etc.) and is used to connect subscriber devices, such as client device 300, to one or more service provider nodes, such as network node 200. Cloud network 14 provides such subscriber devices with “on-demand” availability of computing resources (e.g., memory, data storage, processing power, etc.) without requiring the user to directly and actively manage those resources. According to embodiments of this disclosure, such resources include, but are not limited to, one or more conversational services, cloud gaming applications, or XR applications running on network node 200. XR applications, described in more detail below, may include, for example, those used in combination with gaming applications and / or simulation applications.

[0032] Video encoding As mentioned earlier, video codecs are used to compress (i.e., encode) images into a coded video sequence (CVS) in a bitstream. For example, in Figure 1 In this context, a video codec executing on network node 200 can encode images of a video (e.g., pictures generated by a game engine) for transmission to client device 300 for decoding and display to the user. Some more widely known and used video codecs are developed and standardized by MPEG and ITU-T, and include, but are not limited to, H.264 / AVC, H.265 / HEVC, and H.266 / VVC.

[0033] AVC, HEVC, and VVC images are identified by their Picture Order Count (POC) value. The POC determines the output order of the decoded images. Furthermore, according to the H.264 / AVC and H.265 / HEVC video coding standards, images are typically divided into one or more Slice Network Abstraction Layer (NAL) units. A slice can be a complete image or a portion of an image. Additionally, each NAL unit is essentially a data packet, facilitating the packaging of NAL units into network packets. In this disclosure, the terms "image" and "frame" are used interchangeably.

[0034] Random Access Point (RAP) image In AVC, HEVC, VVC, and most other modern video codecs, random access point pictures, often referred to as "key pictures" by those skilled in the art, are typically used in the video bitstream for three reasons: 1. As the start of a bitstream; 2. Refresh the video; or 3. Tune the decoder to the bitstream, for example, when switching channels in a broadcast television scenario, or when moving to different parts of the video in a video-on-demand (VOD) scenario.

[0035] Random Access Point (RAP) pictures are typically intra-frame coded, meaning that RAP pictures are predicted only from themselves, not from other pictures. However, this prediction results in much lower compression efficiency compared to cases where inter-picture prediction is allowed. Therefore, Intra-Frame Random Access Point (IRAP) pictures tend to produce unwanted bitrate spikes in the bitstream, which can lead to jitter and increased overall latency.

[0036] VVC specifically supports two types of IRAP images. These are: 1. Instantaneous Decoding and Refreshing (IDR) of images; and 2. Clean Random Access (CRA) images.

[0037] IRAP pictures can have "tail pictures" and "preamble pictures." Tail pictures follow their associated IRAP pictures in both decoding and output order, while preamble pictures follow their associated IRAP pictures in decoding order but precede them in output order. In VVC, there are two types of preamble pictures: Random Access Decodable Preamble (RADL) pictures, which are correctly decoded when decoding begins with their associated IRAP pictures, and Random Access Skip Preamble (RASL) pictures, which may not be correctly decoded when decoding begins with their associated IRAP pictures. IDR pictures may have associated RADL pictures instead of RASL pictures, thus ensuring that all pictures in the bitstream that follow the IDR pictures in decoding order are correctly decoded. However, CRA pictures may also have RASL pictures. In some cases, such as when decoding begins with a CRA picture, it may be necessary to discard such preamble pictures.

[0038] Gradual decoding and refresh In some cases, Gradual Decode Refresh (GDR) pictures can be used as an alternative to using IRAP pictures. GDR pictures are recommended for low-latency video scenarios, refreshing only a portion of the video at a time across a series of pictures. This allows for a much smoother bitrate, but at the cost of a slight decrease in compression efficiency and a longer tuning time. Another drawback of using GDR for real-time streaming is that recovery of lost packets can be delayed. This is because the refresh is distributed across multiple frames (i.e., pictures). Furthermore, overall compression efficiency is reduced when using GDR because periodic intra-frame coded blocks are inserted into the bitstream to allow the video to be refreshed. However, despite these drawbacks, GDR is optional for supporting H.264 / AVC and H.265 / HEVC, but mandatory for decoders supporting H.266 / VVC.

[0039] Encoded video sequence In VVC and HEVC, a coded video sequence (CVS) is a sequence of access units (AUs) followed by zero or more AUs, up to (but not including) the AU that begins the next CVS in decoding order. In VVC, the AU that begins the CVS may include IRAP pictures, such as IDR or CRA pictures, or it may include GDR pictures. However, in HEVC, the AU that begins the CVS is an IDR picture. However, regardless of the specific standard, in the context of this disclosure, CVS is broadly defined as a sequence of coded video frames (i.e., pictures) in a bitstream.

[0040] Reference image resampling (RPR) A new feature of VVC is Reference Picture Resampling (RPR). RPR allows the spatial resolution (i.e., width and height) of a picture encoded into a CVS to be changed midway through the bitstream without having to encode the IRAP picture into the CVS. In VVC, the spatial resolution of a picture can be modified at the picture level and is therefore signaled in the Picture Parameter Set (PPS). In contrast, both AVC and HEVC signal the spatial resolution of a picture at the sequence level in the Sequence Parameter Set (SPS). It is worth noting that neither AVC nor HEVC currently supports RPR functionality.

[0041] When the current image and the reference image have different spatial resolutions, RPR enables the encoder to predict the current image using the reference image by scaling the reference image to the same spatial resolution as the current image before prediction occurs. This scaling is done at the block level.

[0042] As defined in the context of this disclosure, RPR functionality is not limited to the RPR functionality of VVC. Rather, RPR is more generally defined as reference picture resampling based on codecs such as VVC and AV1, as well as reference picture resampling functionality that may be provided by future video codecs.

[0043] Low-latency video coding Low-latency video coding is a critical requirement for many video applications, including interactive services, cloud gaming, and extended reality (XR) services. As defined in this disclosure, XR includes real-time virtual reality (VR), augmented reality (AR), and mixed reality (MR). Various factors can affect the latency of encoded video bitstreams. These factors include those related to the reference image and those related to rate control and buffer management.

[0044] Reference image Video used for broadcast television and video-on-demand (VOD) streaming is typically encoded using a group of pictures (GOP) structure. This structure references both past and future pictures to achieve a sufficient level of prediction with high compression. To achieve this, encoders often use bidirectional prediction pictures (i.e., B-pictures), which predict from both reference pictures at once. However, while this functionality can aid in prediction and compression, using future pictures for prediction also increases latency. Therefore, referencing future pictures is not recommended for low-latency video coding. Similarly, specialized encoders used in broadcast and VOD scenarios often provide a look-ahead feature to anticipate upcoming pictures. However, this feature also significantly increases latency and is therefore not recommended for low-latency video coding.

[0045] Rate control and buffer management The number of bits required to compress video images is highly dependent on the video content. Low-motion scenes with little detail are relatively easy to compress, while scenes with a lot of motion and detailed structure are more complex and require more bits. Therefore, broadcast and VOD services typically use variable bitrate (VBR) to maintain constant video quality. To handle variable bitrate, the encoding and decoding buffers must be large enough. However, this increases latency. To achieve lower latency with smaller buffers, video can be encoded using constant bitrate (CBR). CBR is less efficient than VBR and is not recommended for broadcast and VOD services. Instead, CBR is recommended for live video services.

[0046] Rate control algorithms typically modify the QP value for each image, slice, or block to achieve the desired bit rate. Alternatively or additionally, the λ value associated with the rate control algorithm can be changed.

[0047] Video streaming MPEG-DASH MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), specified in ISO / IEC 23009-1 dated August 2022, is an adaptive bitrate (ABR) streaming technology, the entire contents of which are incorporated herein by reference. Using MPEG-DASH, multimedia files are segmented into one or more segments and delivered to a client using the Hypertext Transfer Protocol (HTTP) (typically via Transmission Control Protocol (TCP)). An MPEG-DASH session is established using a Media Presentation Description (MPD), which describes segment information including timing, Uniform Resource Locator (URL), and media characteristics such as video resolution and bitrate. Based on Extensible Markup Language (XML), the MPD can be static (e.g., for movies) or dynamic (e.g., for live content).

[0048] Typically, ABR's VOD service modifies both the frame rate and resolution of the video in chunks or segments (as in MPEG-DASH) to obtain an appropriate bitrate based on available network bandwidth. Specifically, each segment begins with an intra-frame encoded picture to facilitate switching between different resolutions. For each segment, multiple different bitrates are typically pre-coded, with each bitrate corresponding to a certain quality level in a bitrate ladder. Below is an example of a bitrate ladder.

[0049] Table 1 The length of a segment is typically on the order of 6 to 20 seconds. Based on the available network bandwidth for a given segment duration, a segment with a bit rate matching the available bandwidth is requested / selected in terms of quality. Furthermore, different qualities Q1-Q5 can be encoded using different codecs.

[0050] Low-latency video streaming When streaming video with low latency, it is crucial to use a transport protocol that meets the latency requirements. A popular transport protocol for achieving latency below 500 milliseconds is the Real-Time Transport Protocol (RTP), which typically runs on User Datagram Protocol (UDP). The standard for RTP, RFC 3550 entitled "RTP: A Transport Protocol for Real-Time Applications" (July 2003), is explicitly incorporated herein by reference. TCP is generally discouraged because it requires retransmission of lost packets, which rapidly increases latency. AVC, HEVC, and VVC all have RTP payload formats to properly package the underlying bitstream compressed using the codecs in RTP.

[0051] Congestion control algorithms are essential for achieving low latency even when network conditions change. Congestion occurs when the transmitted bit rate exceeds the available bit rate capacity on a given transmission path. Video streaming applications can employ congestion control to achieve robust performance and avoid congestion crashes (i.e., a state where the network is stable but throughput is low).

[0052] Packet loss and packet loss hiding Nodes in current networks typically have large "jitter" buffers. This helps nodes avoid dropping packets when facing congestion. However, this is often referred to as "buffer bloat." For real-time streaming applications, this means the most common form of packet loss is "late loss." Late loss occurs when a packet successfully reaches the receiver, but the time taken to reach the receiver exceeds an acceptable latency period. In these cases, the jitter buffer simply discards the "late loss" packets.

[0053] Different real-time video applications employ different strategies for handling packet loss. Consider, for example, video conferencing applications. In such applications, it's generally believed that discarding all frames until error-free frames can be decoded provides the best user experience. This is because the video may not exhibit much motion. Therefore, freezing the video during a video conference is often preferable to capturing decoding artifacts, which distort objects in the video, such as faces, when displayed. On the other hand, cloud gaming applications typically involve a lot of motion. In this case, a game "freeze" can negatively impact gameplay. Therefore, the most common strategy for handling such freezes is to allow decoding errors and then let the decoder hide those errors as much as possible.

[0054] However, packet loss concealment methods differ between decoder implementations, with some methods handling errors better than others. For example, hardware-based decoding is often preferred for battery-constrained devices. While this extends battery life, it also limits the concealment methods that an application can use for a specific hardware platform, such as those offered by a hardware decoder.

[0055] Recover from loss In-frame request In the event of packet loss, the decoder cannot decode error-free video and therefore requires time to recover. Therefore, decoders in real-time video streaming applications typically recover from packet loss by requesting the encoder to generate intra-frame pictures. In RTP-based applications, this is achieved by the receiving device sending Real-Time Transmission Control Protocol (RTCP) feedback messages, including negative acknowledgments, missing picture indications (NACK-PLI), or missing slice indications (NACK-SLI). Once the decoder receives the intra-frame pictures, it is able to recover decoding error-free.

[0056] However, the requested intraframe images typically create spikes in the number of packets transmitted. Therefore, there is a risk of additional packet loss due to intraframe images. This additional packet loss causes the decoder to send a second intraframe image request to the encoder, which in turn leads to further packet loss, causing the decoder to send even more intraframe image requests to the encoder. This can create a negative cycle of requesting intraframe images to correct packet loss, which only results in additional packet loss.

[0057] L4S (Low Latency, Low Loss, Scalable) L4S was specified in RFC 3168, “The Addition of Explicit Congestion Notification (ECN) to IP,” dated September 2001, the entire contents of which are explicitly incorporated herein by reference. As described in RFC 3168, L4S is a network feature for early network congestion indication. It is a technique for reducing queuing latency, providing low latency for IP flows, and delivering higher performance throughput. L4S uses Explicit Congestion Notification (ECN) tags to indicate congestion in the network to hosts and nodes.

[0058] Periodic intra-frame images and progressive decoding refresh For example, video bitstream (e.g.) Figure 2The bitstream 30 shown may contain periodic intra-coded pictures for tuning into the bitstream or recovering from lost or corrupted frames. Such periodic intra-coded pictures are commonly referred to as "keyframes" by those skilled in the art. In AVC, HEVC, and VVC, such pictures are called Intra-Random Access Point (IRAP) pictures 32. Figure 2 As shown, bitstream 30 typically includes a sequence of inter-frame coded images 34 between periodic IRAP images 32.

[0059] AVC, HEVC, and VVC also support the use of Progressive Decode Refresh (GDR) to tune into the bitstream at inter-coded pictures. With GDR, each picture from a tuning point is typically refreshed by encoding a new region of the picture with intra-coded blocks, until the entire picture is refreshed after several pictures. A major advantage of using GDR is that it allows for a more uniform bitrate compared to using periodic intra-pictures (e.g., intra-coded blocks), which typically require significantly more bits for a given quality. In VVC, GDR is canonically supported using the GDR picture type, and in AVC and HEVC, it is optionally supported using Recovery Point Supplement Enhancement Information (SEI) messages. In VVC, the pictures in the video bitstream from the next IRAP / GDR picture to the next IRAP / GDR picture are called a sequence.

[0060] Periodic intraframes are typically not used in real-time video streaming applications. This is because, even without packet loss, intraframes usually require sending more unnecessary data. Furthermore, recovery from packet loss is delayed compared to using forced intraframes, as the decoder will have to wait for the next periodic intraframe.

[0061] Resolution change Typical VOD services use ABR (Advanced Background Streaming) to control the quality of delivery to the end user while mitigating the impact of varying bandwidth in the transport channel. In ABR streaming, multiple presentations of the same content are generated at a server (e.g., a network node, content delivery network (CDN), etc.). However, the presentation versions differ in codec resolution and bitrate. In some cases, the decision to obtain a specific resolution for playback depends on the client device. In other cases, such as in broadcast scenarios where content is encoded at a selected resolution, the encoder needs to choose whichever resolution is best for encoding performance. The same applies to low-latency applications using a one-to-one distribution model (i.e., content provided only to a user is specific to that user and will not be redistributed to other users). An example of this is cloud gaming content, where each game session will differ based on user interaction (e.g., input).

[0062] In all these video distribution scenarios, where the video resolution changes based on available bandwidth (either via the client device or the encoding device), the video can be seamlessly decoded on the client device, but only if the resolution change in the resulting bitstream does not affect the decoding process. In most cases, this requires the encoder to insert intraframe pictures to signal the decoder that subsequent pictures should be encoded at the new resolution. However, intraframe pictures are not always necessary. For example, the VVCRPR feature allows resolution changes without inserting intraframe pictures.

[0063] While useful, existing techniques for changing resolution mid-bitstream are problematic. For example, for ABR services, each segment is encoded with intra-frame images so that the decoder can switch to the new resolution of the new segment. However, encoding intra-frame images is less efficient than encoding inter-frame images. This inefficiency only increases the total bitrate and can therefore lead to unwanted bitrate spikes in the bitstream.

[0064] Current solutions require the encoder to either use all intra-frame images, which produces large bitrate spikes, or to use GDR, as mentioned earlier, which splits the intra-frame image into several images. The latter provides ultra-low latency at the cost of longer tuning time. However, for some applications, GDR may be too much, and a gentler control over the size of the intra-frame image may be more beneficial.

[0065] RPR was developed for VVC to allow changing the resolution of images in the bitstream without inserting new intra-coded frames. However, there is no explicit guidance in the existing literature describing how to determine the use of RPR, especially in the case of low-latency video, its impact on the management of rate control buffers under low-latency constraints, and its impact on video coding performance and quality. For cases specifically addressing video coding performance and quality, there is no clear discussion or solution on how to mitigate the increased bit latency when encoding intra-frame images at lower resolutions. There also appears to be no discussion on the impact of this on the coding performance of images referencing such intra-frame images. Furthermore, existing solutions do not seem sufficient to handle video refresh after packet loss or video corruption.

[0066] More specifically, existing solutions configure the decoder to send messages to the encoder requesting it to refresh the video by sending intra-coded images (e.g., IRAP images). However, the problem with this technique is that intra-coded images in a bitstream are typically much larger than inter-coded images in the bitstream. As mentioned above, such large IRAP images can amplify any existing latency and cause network congestion. Encoding IRAP images at a lower quality by increasing the quantization parameter (QP) value of the codec can mitigate bitrate spikes from IRAP images. However, encoding IRAP images in this way generally has a negative impact on quality.

[0067] Furthermore, a similar "bitrate spike" problem exists in solutions that signal periodic intra-frame images (e.g., IRAP images) within the bitstream. Additionally, for a given quality, the total bitrate of the bitstream will increase because compressing (i.e., encoding) intra-frame images is less efficient than compressing inter-frame encoded images.

[0068] GDR is also considered a solution for recovering from packet loss. However, a drawback of using GDR is refresh latency. This is because, as mentioned above, GDR-based refresh unfolds across multiple frames (i.e., images). Furthermore, overall compression efficiency is reduced when using GDR because periodic intra-frame coded blocks are still inserted into the bitstream to allow video refresh.

[0069] Therefore, embodiments of this disclosure address these and other problems. Specifically, when refreshing a video bitstream, embodiments of this disclosure configure the encoder to use a lower resolution when encoding refreshed images than when encoding inter-frame coded images. Then, when encoding one or more inter-frame coded images that follow the refreshed images, RPR can be used for prediction based on the refreshed images.

[0070] For example, in one embodiment, an instruction to refresh the video stream (e.g., an intra-frame request) is sent to the encoder. Upon reception, the encoder encodes the refreshed image at a first resolution and encodes subsequent inter-frame(s) images at a different second resolution. This instruction can be sent by a receiving device including a decoder, or by a network node operatively connected to the decoder using L4S tags.

[0071] In another embodiment, this disclosure encodes periodic IRAP images at a first resolution and encodes one or more inter-frame coded images following the IRAP images at a second resolution. The first resolution is lower than the second resolution, which, as will be seen in more detail later, achieves a more uniform bit distribution among images in the bitstream.

[0072] Therefore, this disclosure addresses the need to adequately handle bit rate spikes on a per-frame basis for large intra-coded images. For example, such bit rate spikes may occur when: The video needs to be refreshed, which involves requesting non-periodic intra-coded images from the encoder / transmitter; Tuning options need to be provided, such as by inserting periodic intra-frame coded images into the bitstream; The need for efficient handling of sudden interruptions due to temporal redundancy may necessitate encoding inter-frame coded images using multiple intra-frame blocks; and Given the available bandwidth, the resolution needs to be changed to provide a better quality trade-off.

[0073] To address these and other issues, this embodiment uses RPR to make predictions from images with different resolutions. Specifically, according to this embodiment: Intra-coded images are encoded at a resolution different from (i.e., lower than) that of the inter-coded images(s) used to reference them, in order to reduce bit rate spikes; and RPR is used to change the resolution of an image when needed. For example, it may be necessary to change the resolution of images in a bitstream in order to maintain the desired number of bits per image. Other reasons for such resolution changes include, but are not limited to, allowing temporary changes in resolution due to available bandwidth, content complexity, and / or the availability of computing resources.

[0074] In one embodiment, according to this disclosure, the intra-frame coded image is switched to a lower resolution to reduce bitrate spikes only if a better trade-off is made in the resulting bit-quality. However, in other cases, a QP offset can be applied in addition to, or instead of, RPR. This can provide a more robust solution.

[0075] When using RPR for intra-frame coded images, bit rate spikes may still exist in consecutive images. However, this embodiment addresses this issue by applying QP offset to handle bit rate spikes and / or introducing one or more additional intermediate resolutions for inter-frame coded images that follow “low-resolution” intra-frame coded images.

[0076] The embodiments disclosed herein offer advantages and benefits that conventional encoding / decoding systems and methods do not or cannot provide. For example, one advantage compared to the prior art is that this embodiment provides the ability to avoid bitrate spikes caused by large intra-frame or inter-frame images with a large proportion of intra-codec blocks. As described above and explained in more detail below, this can be achieved by reducing the resolution of the selected images. This functionality is important because it allows applications more flexible control over the buffers in both the encoder and decoder, which is essential for low-latency applications.

[0077] Another advantage is that the technology provided in this embodiment offers a more robust solution for the encoder. More specifically, the encoder can use the resolution of the image or group of images as an additional parameter (e.g., in addition to QP, λ, and the scaling matrix) to control the bits generated for each image.

[0078] The encoder uses RPR as a response to an instruction to refresh the video. In a first embodiment of this disclosure, the encoder uses RPR in response to receiving an instruction to refresh the video bitstream. Specifically, the encoder encodes a first image at a first resolution (e.g., "resolution A") into a CVS in the bitstream, and then encodes a second image at a second resolution B (e.g., "resolution B") into a CVS in the bitstream. According to this disclosure, resolutions A and B are different.

[0079] A first image with resolution A is encoded as a refresh image, wherein at least a portion of the image is intra-coded. In one embodiment, the refresh image completely refreshes the video. The refresh image may, for example, include an IRAP image. In yet another embodiment, the refresh image initiates a refresh of the video, but the video is not fully refreshed until subsequent images following the refresh image in decoding order have been processed. In some embodiments, the refresh image is a GDR image. However, in other embodiments, the refresh image is a collection of images (e.g., images from the GDR image to the recovery point image in a progressive decoding refresh).

[0080] As described above, the second image follows the first image in decoding order and is encoded as an inter-frame coded image predicted based on the first image. In one embodiment, the second image is encoded using RPR such that the reference image is scaled before prediction.

[0081] In a preferred embodiment, at least one of the height and width of a first image at resolution A is smaller than the corresponding height and width of a second image at resolution B. For example, consider an example where resolution A is full HD (i.e., 1920 × 1080 pixels) and resolution B is 4K (i.e., 3840 × 2160 pixels). Therefore, resolution A is smaller than resolution B. By initiating a video refresh by using an intra-frame image encoded at a lower resolution (i.e., resolution A) as the refresh image, and then increasing to resolution B to encode subsequent inter-frame encoded images, this embodiment provides a uniform "per-frame" bitrate while still maintaining good overall video quality. The refresh image (at least partially intra-frame encoded and less compressed than the inter-frame encoded image) can be encoded at a relatively low bitrate while still maintaining good quality (for the lower resolution). The encoder can then subsequently switch to encode one or more inter-frame encoded images following the refresh image at a higher, more compressed resolution, such as resolution B or some other intermediate resolution between resolution A and resolution B.

[0082] For example, consider a low-latency video pipeline with a bandwidth of 5 Mbps, which cannot tolerate image-level bitrate spikes. For a 60fps video, each image is approximately 10 kB in size. If all images in the video sequence are encoded at a high resolution (i.e., 4K resolution), the first image (the refresh image) will be encoded with many unwanted artifacts and will have poor quality due to insufficient bit count. Therefore, the first image is insufficient as a reference image for predicting subsequent images, and the encoder must encode multiple first images to achieve sufficiently good video quality.

[0083] Now consider a scenario where, according to this disclosure, the first image is encoded at a lower resolution A (i.e., full HD resolution), and subsequent inter-frame coded images are encoded at a higher resolution B (i.e., 4K resolution). In these cases, 10kB may be sufficient to encode the first image with sufficiently good quality, thereby improving the quality of subsequent inter-frame coded images predicted from the first image.

[0084] Figure 3 This is a signaling diagram illustrating this embodiment. More specifically, Figure 3 Communication between a transmitting device 200 (e.g., including an encoder) and a receiving device 300 (e.g., including a decoder) is illustrated. In this embodiment, the transmitting device 200 encodes multiple images at resolution B (i.e., 4K resolution) and sends the encoded images to the receiving device 300 in a bitstream (boxes 42, 44, 46). Figure 3As shown, receiving device 300 receives images 1 and 3, but not image 2. Therefore, in response to detecting a lost packet carrying image 2 (box 48), receiving device 300 requests sending device 200 to refresh the video stream (box 50). Receiving device 300 may send a refresh request in response to determining that decoding the lost / corrupted packet (i.e., image 2) will result in corrupted video or unwanted artifacts in the decoded video.

[0085] In response to receiving a refresh request, the transmitting device 200 encodes and transmits image 4 to the receiving device (box 52). However, according to this disclosure, the transmitting device 200 does not encode image 4 at a higher resolution B. Instead, the transmitting device 200 encodes image 4 at a lower resolution A (i.e., full HD resolution) as a refresh image (e.g., an IRAP image) to transmit to the receiving device 300. Thereafter, the transmitting device 200 continues to encode one or more subsequent images (e.g., image 5 and subsequent images) at a higher resolution B to transmit to the receiving device 300 (box 54).

[0086] As described above, encoding image 4 as a refreshed image (e.g., an IRAP image) at a resolution lower than that of subsequent images enables compression (i.e., encoding) with relatively good quality while avoiding bit rate spikes. Additionally, in at least one embodiment, the transmitting device 200 uses RPR to predict image 5 from image 4.

[0087] Figure 3 The embodiments illustrate a change from resolution A to resolution B as a single "step". However, this disclosure is not limited thereto. For example, in at least one embodiment, the change in resolution can be performed gradually across multiple images. As described in more detail below, this gradual increase in resolution makes the change in resolution less noticeable to a user viewing the images. Furthermore, the method of gradually increasing the resolution helps to maintain a predetermined bitrate for each image.

[0088] For example, consider a scenario where a refreshed image (e.g., an intra-frame image) is encoded at half the resolution of the inter-coded image that follows it (e.g., resolution A versus resolution B). In these cases, predictions for the inter-coded image that follows the refreshed image in the decoding order are worse than when the inter-coded image and the refreshed image have the same resolution. This is because the refreshed image, with a resolution half that of the subsequent inter-coded image, does not have enough bits to adequately support predictions for the subsequent inter-coded image. Therefore, this embodiment employs a "step-by-step" approach to change the resolution. Specifically, the refreshed image is still encoded at a lower resolution. However, the inter-coded image that immediately follows the refreshed image in the decoding order is encoded at an intermediate resolution between the refreshed image and the third inter-coded image that appears in the decoding order after the first and second images. As provided in this embodiment, by gradually changing the resolution, the relative prediction will be sufficiently accurate. Furthermore, a step back to the original resolution (i.e., from the resolution of the inter-coded image immediately following the refreshed image to the resolution of the next inter-coded image in the decoding order) can be performed without significantly deviating from the desired bit rate.

[0089] This is due to Figure 4A , Figure 4B and Figure 4C Graphs 60, 70, and 80 in the diagram illustrate three different coding scenarios. For intra-coded images (e.g., refresh images), the image is represented by a rectangle labeled "I," and for bidirectional prediction images (e.g., inter-coded images), the image is represented by a rectangle labeled "B." The size of the rectangle represents the relative resolution of the image, and the bar immediately above the "I" and "B" rectangles indicates the bit rate of the corresponding image.

[0090] exist Figure 4A In this context, the "I" image has the same resolution as the "B" image that follows it in the decoding sequence. As mentioned earlier, this can cause bit rate spikes, as shown in bar 62. Figure 4B In this approach, the encoding resolution of the "I" image is half the resolution of the following "B" image. This method makes it possible to encode the "I" image at a similar QP and bit rate as the "B" image preceding it. However, the image immediately following... Figure 4B The "B" image in the "I" image cannot be accurately predicted from the "I" image. This also causes a bitrate spike in the image, although it is less than... Figure 4A The bit rate is increased, where the "I" and "B" images have the same resolution.

[0091] exist Figure 4CIn this process, the resolution of the "I" image is half the resolution of the "B" image preceding it in the decoding sequence. However, instead of encoding the "B" image 84 immediately following the "I" image at the same resolution as the preceding "B" image, the embodiments of this disclosure employ a previously described "step-by-step" method, encoding the "intermediate B image" 84 at a resolution higher than that used to encode the "I" image, but lower than that used to encode the next "B" image 86 in the decoding sequence. Because the "I" image and the "intermediate B image" 84 are closer in resolution, and because the intermediate "B" image 84 and the next "B" image 86 are closer in resolution, prediction from the "I" image is enhanced. Specifically, bitrate spikes do not occur at all, or are at least significantly reduced, as shown in section 82. Then, due to better prediction and a smaller resolution, the bitrate becomes approximately the same as the first B image.

[0092] Figure 5A , Figure 5B , Figure 5C and Figure 5D Graphs 90, 100, 110, and 120 illustrate how this embodiment configures the device to use RPR control per-picture bit. More specifically, Figure 5A Graph 90 in the figure shows the anchor point, which uses a typical QP offset, such as the offset used for low-latency configurations in the Joint Video Experts Group (JVET) Common Test Conditions (CTC). Figure 5A As shown, the size of the intra-coded image (POC 64) is significantly larger than other images (i.e., POC 60, 62, 66, and 68). This may require a larger rate control buffer.

[0093] Figure 5B The curve in Figure 100 shows how QP offset +6 helps reduce the size of intra-coded pictures by a factor of 2, but results in an increase in the size of consecutive pictures, which compensates for the poor prediction. Figure 5C It shows something similar to Figure 5B The situation is illustrated, but the intra-frame coded image is encoded at a reduced resolution (2x in both dimensions). Similar to the above. Figure 5B , Figure 5C The increase in consecutive image size due to insufficient compensation for prediction is shown. Figure 5D A graph 120 is shown, which illustrates how a gradual change in resolution applied to intra-coded images and the following inter-coded images maintains small image sizes and avoids large bit rate spikes.

[0094] In one embodiment, resolution variation is used in conjunction with other adaptive rate control tools to adjust the image rate. These other adaptive rate control tools can be configured to change the QP (e.g., at the block, slice, or picture level), change the λ value, and reduce the frame rate.

[0095] In another embodiment, the refresh indication for refreshing the video is a request to send intra-coded pictures. For example, the decoder (or other receiving device) may send an intra-frame request (FIR) message to the sending device (e.g., an encoder), as specified in IETF RFC 5104, entitled “CodecControl Messages in the RTP Audio-Visual Profile with Feedback (AVPF)”, published February 2008. Alternatively, the decoder (or other receiving device) may send a Picture Loss Indication (PLI) message or a Slice Loss Indication (SLI) message to the sending device (e.g., an encoder), as specified in IETF RFC 4585, entitled “Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP / AVPF)”, published July 2006. Both IETF RFC 5104 and IETF RFC 4585 are incorporated herein by reference in their entirety. In another example, a receiver requests an intra-frame encoded image in a DASH or HTTP Live Streaming (HLS) session.

[0096] In another embodiment, the instruction is not to refresh the bitstream, but rather to request the transmission of a CVS (Continuous Frame View), which begins with a refreshed image (e.g., an IRAP or GDR image) at a first resolution A, followed by a second inter-frame coded image at resolution B. As mentioned above, resolutions A and B are different, and the second image at resolution B is predicted based on the refreshed image at resolution A. That is, the refreshed image is a reference image for the second inter-frame coded image. In this embodiment, the bitstream does not necessarily include the encoded video before the first image is sent to the receiving device.

[0097] In another embodiment, the instruction to refresh the video is not sent from the receiving device 300, which includes the decoder, but from a node in the network 10. This instruction may be signaled, for example, using an L4S tag.

[0098] In at least one embodiment, no instruction is sent to the transmitting device. More specifically, the transmitting device 200 (e.g., an encoder) encodes the first image and the second image. However, in this embodiment, the transmitting device 200 is configured to autonomously decide (i.e., not to receive an instruction first) to encode the first image at a resolution lower than that used for encoding the second image, rather than starting to encode the image in response to receiving an explicit instruction.

[0099] Therefore, the encoder configured according to this disclosure can perform all or a subset of the following functions to encode an image into a CVS in a bitstream.

[0100] The system receives an instruction to refresh the video bitstream, or an instruction to begin encoding the video into a bitstream (i.e., to send a CVS). This instruction may be a request to send intra-frame encoded images, such as IRAP or GDR images. This instruction can be sent from receiving device 200 or another network node, each of which may include or be associated with a decoder.

[0101] The first image is encoded into a CVS. The first image is encoded as a refresh image, wherein at least a portion of the image is intra-coded. The refresh image may be, for example, an IRAP image or a GDR image. The first image is encoded at a first resolution A, and the encoding of the first image can be performed in response to a received instruction.

[0102] The second image is encoded into the CVS. The second image follows the first image in the decoding order, and is encoded as an inter-frame coded image predicted based on the first image. The second image is encoded at a second resolution B, different from resolution A. Furthermore, resolution A can be lower than resolution B, wherein at least one of the height and width of the first image at resolution A is smaller than the corresponding height or width of the second image at resolution B. RPR can be used for prediction from the first image, and can also be performed to encode the second image at resolution B in response to a received instruction.

[0103] The third image is encoded into CVS. The third image follows the first and second images in decoding order, and is encoded as an inter-frame coded image predicted based on at least one of the first and second images. The third image is encoded at a third resolution "C", where resolution C is different from both resolution A and resolution B. RPR can be used for prediction based on the first image and / or the second image.

[0104] The decoder configured according to this embodiment can perform all or a subset of the following functions to decode images from CVS in the bitstream.

[0105] A video packet has been detected to be missing or one or more images have been corrupted.

[0106] The encoder is instructed to refresh the video bitstream, or to begin encoding the video into a bitstream (i.e., send a CVS). This instruction can be a request to send intra-coded frames, such as IRAP or GDR frames. Sending this instruction can also be a response to the detection of lost video packets or the detection of one or more corrupted frames.

[0107] Receives a bitstream including CVS.

[0108] Obtain the first encoded image from CVS.

[0109] Decode the first encoded image. In this embodiment, the first encoded image has been encoded as a refresh image, and as described above, at least a portion of this image is intra-frame coded. The refresh image may be, for example, an IRAP image or a GDR image, and has a first resolution A.

[0110] Obtain the second encoded image from CVS.

[0111] Decode the second coded image. The second coded image follows the first coded image in the decoding order, and is decoded into an inter-frame coded image predicted based on the first coded image. The second coded image has a second resolution B, different from resolution A. Furthermore, resolution A may be lower than resolution B, and at least one of the height and width of the first coded image at resolution A is smaller than the corresponding height and width of the second coded image at resolution B. As previously stated, RPR can be used for prediction based on the first image.

[0112] Obtain the third-encoded image from CVS.

[0113] Decode the third coded image. The third coded image follows the first and second coded images in decoding order, and is decoded into an inter-frame coded image predicted based on at least one of the first and second coded images. The third coded image has a third resolution C, different from resolutions A and B. Additionally, RPR can be used for prediction based on the first and / or second coded images.

[0114] instruct In at least one embodiment, the refresh instruction sent from the receiving device 300 to the transmitting device 200 may include one or more of the following: Request to send intra-frame encoded images, such as IRAP images, to directly refresh the video; Requests to gradually refresh the video (e.g., requests to send GDR images); The specified maximum resolution; The specified minimum resolution; Requests or permission to use RPR for prediction; Preferred bitrate per image; Preferred bit rate per second; Maximum bitrate per image; Maximum bit rate per second; The encoder's request or assumption to set the resolution based on a specific bit rate; Requests for intra-frame images with lower resolution and inter-frame coded images with higher resolution; Requests that start at a low resolution and then increase to a higher resolution; and Start with low resolution and then gradually increase the requests to higher resolutions across multiple images.

[0115] Use RPR for periodic intra-frame refresh As described above, embodiments of this disclosure mitigate bit rate spikes. For example, when inter-coded frames are lost (e.g., one or more incoming packets are dropped and / or corrupted at the receiving device), the transmitting device configured according to this embodiment refreshes the video at the next intra-coded frame. As previously stated, the transmitting device can perform this function autonomously or in response to an instruction received from the receiving device 300 or a network node. However, in order to achieve successful prediction from intra-coded frames, at least some inter-coded frames use RPR.

[0116] For example, in one embodiment, an encoder configured according to this disclosure (e.g., at transmitting device 200) periodically encodes one or more intra-frame images (e.g., IRAP images) into a CVS of the bitstream. Such images may be encoded at regular time intervals (e.g., every 10 seconds), autonomously as needed, and / or at variable locations in the bitstream (e.g., at scene transitions). However, regardless of the periodicity, the encoder in this embodiment encodes the intra-frame encoded images at a resolution lower than the resolution at which it encodes inter-frame encoded images into a CVS of the bitstream.

[0117] Figure 6 An example of such an embodiment is shown. Figure 6 As shown, CVS 130 includes a plurality of IRAP images 132 and one or more inter-frame coded images 134 located between the IRAP images 132. Furthermore, as indicated by the different sizes of images 132, 134, the IRAP images 132 are encoded at a resolution lower than the resolution at which they encode the inter-frame coded images 134.

[0118] In one embodiment, the intra-coded image (e.g., IRAP image 132) is decoded by the receiving device 300 but is never displayed. That is, the decoder at the receiving device 300 uses only the intra-coded image in the CVS as a reference image for prediction, and does not output the decoded intra-coded image for further processing and / or display. However, because lower-resolution video (i.e., lower-resolution intra-coded images, such as IRAP image 132) is not displayed, this embodiment improves the overall video quality.

[0119] In addition, such as Figure 7 As shown, for example, the encoder can be configured to encode the inter-frame coded image 136 that appears immediately after the IRAP image 132 in the CVS 130 at a reduced resolution. By way of example only, the encoder at the transmitting device 200 can encode the IRAP image 132 at half the normal resolution of the inter-frame coded image 134. The encoder can then encode the inter-frame coded image that appears immediately after the IRAP image 132 at an intermediate resolution that is higher than the resolution of the IRAP image 132 and lower than the resolution of the inter-frame coded image 134. As described above, this "stepwise" method of changing the image resolution of the CVS encoded into a bitstream helps to mitigate bitrate spikes.

[0120] Figure 8This is a flowchart illustrating a method 140 for encoding an image into a CVS 130 according to an embodiment of the present disclosure. For illustrative purposes only, method 140 is implemented by a transmitting device 200, such as a network node in cloud 14. However, those skilled in the art will readily understand that method 140 can be implemented on devices other than network nodes, such as... Figure 1 The computing device 20 shown.

[0121] like Figure 8 As shown, the transmitting device 200 receives a refresh instruction to refresh the video before encoding the first image (box 142). In response to receiving the refresh instruction, the transmitting device 200 generates CVS 130 in the bitstream (box 144) by encoding the first image as a CVS in the bitstream at a first resolution (box 144a) and encoding the second image as a CVS in the bitstream at a second resolution (box 144b). In this embodiment, the first image is encoded as a refreshed image, and at least a portion of the first image is intra-coded. Furthermore, the first resolution is different from the second resolution. Additionally, the second image follows the first image in the decoding order, and the second image is encoded as an inter-coded image by prediction from the first image.

[0122] In at least one embodiment, the transmitting device 200 encodes a third image into a CVS (box 144c) in the bitstream at a third resolution different from the first and second resolutions. The third image follows the first and second images in the decoding order, and the third image is encoded as an inter-frame coded image by prediction from one or both of the first and second images. After such encoding, the transmitting device 200 transmits the CVS of the bitstream to the receiving device 300 (box 146).

[0123] In one embodiment, a refresh instruction is received from the receiving device.

[0124] In another embodiment, the refresh instruction is signaled by a network node.

[0125] In one embodiment, the refresh indication is based on a low-latency, low-loss, and scalable (L4S) flag.

[0126] In one embodiment, the bitstream is generated autonomously, independent of receiving refresh instructions.

[0127] In one embodiment, the CVS in the bitstream appears after the initial CVS in the bitstream.

[0128] In one embodiment, the second image is encoded using reference image resampling (RPR), such that the reference image used to predict the second image is scaled before the second image is encoded.

[0129] In one embodiment, the reference image is the first image.

[0130] In one embodiment, when encoding the first and second images, the change from a first resolution to a second resolution adapts to the bit rate or quality of at least one of the first image, the second image, and one or more subsequent images.

[0131] In one embodiment, the bit rate or quality of at least one of the first image, the second image, and one or more subsequent images is further adapted based on one or more adaptive rate control tools.

[0132] In one embodiment, one or more adaptive rate control tools are configured to change the quantization parameter (QP) at one of the block, slice, and picture levels, change one or more λ values, and reduce the frame rate.

[0133] In one embodiment, the first image is periodically encoded at a resolution lower than that of the inter-coded images in the bitstream.

[0134] In one embodiment, the position where the first image in the bitstream is encoded is variable.

[0135] In one embodiment, the receiving device includes a decoder.

[0136] However, in another embodiment, the receiving device includes a network node.

[0137] Figure 9 This is a flowchart illustrating a method 150 for decoding an image from a CVS 130 according to an embodiment of the present disclosure. It may be noted here that this embodiment of the present disclosure implements method 150 at a receiving device 300 including a decoder. However, this is merely for illustrative purposes. Those skilled in the art will readily understand that the receiving device 300 implementing method 150 can be a device such as a computing device 20 in a network.

[0138] like Figure 9 As shown, the receiving device 300 detects that video packets in the video have been lost or that one or more images in the video have been corrupted (box 152). In response to this detection, the receiving device 300 sends a refresh instruction to the transmitting device 200 to refresh the video bitstream (box 154). The receiving device 300 then receives the bitstream from the transmitting device 200 (box 156). In this embodiment, the bitstream includes a CVS 130 containing the first and second images.

[0139] Upon receiving the data, the receiving device 300 decodes the first and second images from the CVS 130 in the bitstream (box 158). More specifically, the receiving device 300 decodes the first image from the CVS 130 in the bitstream, wherein the first image is encoded as a refresh image and has a first resolution (box 158a). The receiving device 300 also decodes the second image from the CVS 130 in the bitstream (box 158b). In this embodiment, the second image follows the first image in the decoding order, the second image has a second resolution different from the first resolution, and the second image is decoded as an inter-frame coded image predicted based on the first image. In some embodiments, the receiving device 300 is also configured to decode a third image from the CVS 130 in the bitstream (box 158c). In these embodiments, the third image follows the first and second images in the decoding order, the third image is decoded as an inter-frame coded image by prediction from one or both of the first and second images, and the third image is encoded at a third resolution different from the first and second resolutions.

[0140] After such decoding, the receiving device 300 outputs the decoded image for further processing, such as for display or storage (box 160). However, in at least one embodiment, the receiving device 300 is prevented from sending the decoded first image (e.g., a low-resolution intra-coded image) to a display device for display to the user (box 162).

[0141] In one embodiment, outputting the decoded second image includes sending the decoded second image to a display device for display.

[0142] However, in another embodiment, outputting the decoded second image includes sending the decoded second image to a storage device for storage.

[0143] In one embodiment, a refresh instruction is sent before receiving the CVS, which includes the first and second images.

[0144] However, regardless of whether the current embodiment is implemented by the transmitting device 200 or the receiving device 300, in at least one embodiment, the refresh indication includes a request for one of the following: an intra-frame request (FIR) message, a picture loss indication (PLI) message, and a slice loss indication (SLI) message.

[0145] In one embodiment, the refresh indicator requests the first image in a Real-time Transport Protocol (RTP) session, a Dynamic Adaptive Streaming (DASH) session over Hypertext Transfer Protocol (HTTP), or an HTTP Live Streaming (HLS) session.

[0146] In one embodiment, the refresh instruction includes requesting the sending device to refresh the video bitstream.

[0147] In one embodiment, the refresh instruction includes a request to begin encoding video into a bitstream using CVS.

[0148] In one embodiment, the refresh instruction includes a request to send an intra-frame encoded image.

[0149] In one embodiment, the refresh instruction includes sending a CVS request in the bitstream.

[0150] In one embodiment, the refreshed image is an Intra-Random Access Image (IRAP) image.

[0151] In one embodiment, refreshing the image is a Progressive Decoding Refresh (GDR) image.

[0152] In one embodiment, a second image is predicted from a first image using reference image resampling (RPR).

[0153] In one embodiment, RPR is used to predict a third image from one or both of the first and second images.

[0154] In one embodiment, the first resolution is lower than the second resolution.

[0155] In one embodiment, the second resolution is lower than the third resolution.

[0156] In one embodiment, at least one of the height and width of the first image with a first resolution is smaller than the corresponding height and width of the second image with a second resolution.

[0157] In one embodiment, a refresh instruction includes one or more of the following: a request to send a direct refresh of intra-coded frames of video, a request to gradually refresh video, a specified maximum resolution, a specified minimum resolution, a request or authorization to use RPR for prediction, a preferred bitrate per frame, a preferred bitrate per second, a maximum bitrate per frame, a maximum bitrate per second, a request to set the resolution of the encoder based on a specified bitrate, an instruction to instruct the encoder to set the resolution based on a specified bitrate, a request to encode intra-frame frames at a resolution lower than the resolution of the inter-coded frames, a request to encode inter-frame frames at a resolution higher than the resolution of the intra-coded frames, a request to start encoding at an initial resolution and increase to subsequent resolutions higher than the initial resolution, and a request to start encoding at an initial resolution and gradually increase the resolution to subsequent resolutions over multiple frames.

[0158] In one embodiment, the first image completely refreshes the video.

[0159] In one embodiment, the first image initiates a video refresh, and subsequent images following the first image in the decoding order complete the video refresh.

[0160] In one embodiment, the first image includes multiple refreshed images.

[0161] In one embodiment, the multiple refresh images include multiple images, including progressive decode refresh (GDR) images and recovery point images.

[0162] In one embodiment, the plurality of refresh images also includes one or more images between the GDR image and the recovery point image.

[0163] The device can perform any of the methods described herein by implementing any functional components, modules, units, or circuits. For example, in one embodiment, the device includes a corresponding loop or circuit configured to perform the steps shown in the method diagram. In this regard, the loop or circuit may include a loop dedicated to performing specific functional processing and / or one or more microprocessors combined with memory. For example, the circuit may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), application-specific digital logic, etc. The processing circuitry may be configured to execute program code stored in memory, which may include one or more types of memory, such as read-only memory (ROM), random access memory, cache memory, flash memory devices, optical storage devices, etc. In several embodiments, the program code stored in the memory may include program instructions for executing one or more telecommunications and / or data communication protocols, and instructions for executing one or more techniques described herein. In embodiments employing memory, the memory stores program code that, when executed by one or more processors, performs the techniques described herein.

[0164] Figure 10 This is a functional block diagram illustrating some exemplary components of a transmitting apparatus 200 configured according to embodiments of the present disclosure. For example... Figure 10 As shown, the transmitting device 200 includes a physical computing node with physical circuitry and can be configured to implement one or more encoding functions, as previously described. In some cases, there are multiple transmitting devices 200 configured to implement one or more encoding functions. Therefore, according to this disclosure, there may be multiple transmitting devices 200 distributed throughout a communication network (e.g., a cloud) configured to perform the functions previously described.

[0165] like Figure 10As shown, the transmitting device 200 includes communication circuitry 202, processing circuitry 204, encoder 206, and memory 208. Communication circuitry 202 includes the hardware required for direct or indirect communication with one or more other network nodes and receiving device 300 (e.g., a user device). In this respect, communication circuitry 202 may include network interface circuitry (NIC) (e.g., Ethernet or a similar interface). In some embodiments, transmitting device 200 may be configured for wireless communication; therefore, communication circuitry 202 will include radio frequency (RF) circuitry required for transmitting and receiving signals through a wireless communication channel. In these cases, transmitting device 200 may also be coupled to one or more antennas (not shown). However, in other embodiments, transmitting device 200 is configured to communicate via a wired interface.

[0166] Processing circuitry 204 includes one or more microprocessors, hardware, firmware, or combinations thereof that control the overall operation of transmitting device 200. According to this disclosure, processing circuitry 204 may include encoder 208 and may be software-configurable to perform one or more methods described herein, including... Figure 8 Method 140 is shown.

[0167] Memory 208 includes both volatile and non-volatile memory for storing computer program code and data required for the operation of processing circuitry 204. Memory 208 may include any tangible, non-transitory computer-readable storage medium for storing data, including electronic, magnetic, optical, electromagnetic, or semiconductor data storage devices. Memory 208 stores a computer program 210 including executable instructions that configure the processing circuitry 204 in the transmitting device 200 to perform one or more methods described herein, including... Figure 8 The method 140 shown. In this respect, the computer program 210 may include one or more code modules corresponding to the components or units described above.

[0168] Typically, computer program instructions and configuration information are stored in non-volatile memory, such as ROM, erasable programmable read-only memory (EPROM), or flash memory. Temporary data generated during operation may be stored in volatile memory, such as random access memory (RAM). In some embodiments, computer program 210 for configuring processing circuitry 204 as described herein may be stored in removable memory, such as portable optical disc, portable digital video disc, or other removable media. Computer program 210 may also be contained in a carrier, such as electronic signals, optical signals, radio signals, or computer-readable storage media.

[0169] Those skilled in the art will also understand that the embodiments herein also include corresponding computer programs. The computer program includes instructions that, when executed on at least one processor of the device, cause the device to perform any of the corresponding processes described above. In this respect, the computer program may include one or more code modules corresponding to the aforementioned components or units.

[0170] Embodiments of this disclosure also include a carrier containing such a computer program 210. The carrier may include one of an electrical signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0171] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer-readable (storage or recording) medium, the computer program product including instructions that, when executed by a processor of a device, cause the device to perform as described above.

[0172] The embodiments also include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the transmitting device 200 executes the computer program product. The computer program product may be stored on a computer-readable recording medium.

[0173] Figure 11 This is a functional block diagram illustrating some exemplary components of a receiving device 300 configured according to embodiments of the present disclosure. For example... Figure 11 As shown, the receiving device 300 includes a physical computing node with physical circuitry and can be configured to implement one or more decoding functions, as previously described. In some cases, there are multiple receiving devices 300 configured to implement one or more decoding functions. Therefore, according to this disclosure, there can be multiple receiving devices 300 distributed throughout a communication network (e.g., a cloud) configured to perform the previously described decoding functions.

[0174] like Figure 11 As shown, the receiving device 300 includes communication circuitry 302, processing circuitry 304, decoder 306, and memory 308. Communication circuitry 302 includes the hardware necessary for communicating directly or indirectly with one or more other network nodes and with the transmitting device 200. In this respect, communication circuitry 302 may include network interface circuitry (NIC) (e.g., Ethernet or a similar interface). In some embodiments, the receiving device 300 may be configured for wireless communication; therefore, communication circuitry 302 will include radio frequency (RF) circuitry necessary for transmitting and receiving signals through a wireless communication channel. In these cases, the receiving device 300 may also be coupled to one or more antennas (not shown). However, in other embodiments, the receiving device 300 is configured to communicate via a wired interface.

[0175] Processing circuitry 304 includes one or more microprocessors, hardware, firmware, or combinations thereof that control the overall operation of receiving device 300. According to this disclosure, processing circuitry 304 may include decoder 308 and may be software-configurable to perform one or more methods described herein, including... Figure 9 Method 150 is shown.

[0176] Memory 308 includes both volatile and non-volatile memory for storing computer program code and data required for the operation of processing circuitry 304. Memory 308 may include any tangible, non-transitory computer-readable storage medium for storing data, including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 308 stores a computer program 310 including executable instructions that configure the processing circuitry 304 in receiving device 300 to perform one or more methods described herein, including... Figure 9 The method 150 shown. In this respect, the computer program 310 may include one or more code modules corresponding to the components or units described above.

[0177] Typically, computer program instructions and configuration information are stored in non-volatile memory, such as ROM, erasable programmable read-only memory (EPROM), or flash memory. Temporary data generated during operation may be stored in volatile memory, such as random access memory (RAM). In some embodiments, the computer program 310 for configuring the processing circuitry 304 as described herein may be stored in removable memory, such as a portable optical disc, portable digital video disc, or other removable medium. The computer program 310 may also be contained in a carrier, such as an electronic signal, optical signal, radio signal, or computer-readable storage medium.

[0178] Those skilled in the art will also understand that the embodiments herein also include corresponding computer programs. As described above, the computer program includes instructions that, when executed on at least one processor of the device, cause the device to perform any of the corresponding processes described above. In this respect, the computer program may include one or more code modules corresponding to the aforementioned components or units.

[0179] Embodiments of this disclosure also include a carrier containing such a computer program 310. The carrier may include one of an electrical signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0180] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer-readable (storage or recording) medium, the computer program product including instructions that, when executed by a processor of a device, cause the device to perform as described above.

[0181] The embodiments also include a computer program product including program code portions for performing the steps of any of the embodiments herein when the receiving device 300 executes the computer program product. The computer program product may be stored on a computer-readable recording medium.

[0182] Of course, this embodiment can be implemented in ways other than those specifically set forth herein without departing from the characteristics described herein. Therefore, this embodiment is to be considered illustrative rather than restrictive in all respects, and all variations within the meaning and equivalent scope of the appended embodiments are intended to be included therein.

Claims

1. A method (140) implemented by a transmitting device (200) having an encoder (206) for encoding a picture into a encoded video sequence (CVS) in a bitstream (130), the method comprising: Generating (144) the CVS in the bitstream, the generation comprising: Encoding (144a) a first image into the CVS in the bitstream at a first resolution, wherein the first image is encoded as a refresh image, and wherein at least a portion of the first image is intra-frame encoded; and The second image is encoded (144b) into the CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in decoding order, and the second image is encoded as an inter-frame coded image by prediction from the first image; and The CVS of the bit stream is sent (146) to the receiving device (20, 300).

2. The method according to claim 1, further comprising: Before encoding the first image, receive (142) a refresh instruction to refresh the video; and In response to receiving the refresh instruction, the CVS is generated in the bitstream.

3. The method according to claim 2, wherein, Receiving the refresh instruction includes one or both of the following: Receive the refresh instruction from the receiving device; and Receive the refresh instruction signaled by the network node (200).

4. The method according to any one of claims 2 to 3, wherein, The refresh indication is based on the Low Latency, Low Loss, and Scalable (L4S) flag.

5. The method according to any one of claims 1 to 4, wherein, The CVS in the bitstream appears after the initial CVS in the bitstream.

6. The method according to any one of claims 1 to 5, wherein, Generating the CVS in the bitstream further includes encoding a third image (144c) into the CVS in the bitstream at a third resolution different from both the first and second resolutions, wherein the third image: Follow the first image and the second image in the decoding order; and The image is encoded as an inter-frame coded image by prediction from one or both of the first and second images.

7. The method according to any one of claims 1 to 6, wherein, The second image is encoded using Reference Image Resampling (RPR), such that a reference image used to predict the second image is scaled before the second image is encoded, and wherein the first image is the reference image used to predict the second image.

8. The method according to any one of claims 1 to 7, wherein, When encoding the first image and the second image, the change from the first resolution to the second resolution is adapted to the bit rate or quality of at least one of the first image, the second image, and one or more subsequent images.

9. The method according to claim 8, wherein, The bitrate or quality of at least one of the first image, the second image, and the one or more subsequent images is further adapted based on one or more adaptive rate control tools, wherein the one or more adaptive rate control tools are configured to: Change the quantization parameter (QP) value at at least one of the block level, slice level, and image level; Change one or more lambda values; and / or Reduce the frame rate.

10. The method according to any one of claims 1 to 9, wherein, The first image is periodically encoded at a resolution lower than that of the inter-frame encoded images in the bitstream, and / or the position of the first image encoded in the bitstream is variable.

11. The method according to any one of claims 1 to 10, wherein, The receiving device includes one or both of a decoder and a network node.

12. A method (150) implemented by a receiving device (300) having a decoder (306) for decoding images from a coded video sequence (CVS) in a bitstream, the method comprising: Decode (158a) the first image of the CVS from the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; Decode (158b) the second image of the CVS from the bitstream, wherein the second image: Follow the first image in the decoding order; It has a second resolution that is different from the first resolution; and It is decoded into an inter-frame coded image predicted based on the first image; as well as Output (160) the decoded second image.

13. The method according to claim 12, wherein, The output of the decoded second image also includes one or both of the following: The decoded second image is sent to a display device for display; and The decoded second image is sent to a storage device for storage.

14. The method according to any one of claims 12 to 13, further comprising receiving (156) the bit stream from the transmitting device, wherein the bit stream includes the CVS comprising the first picture and the second picture.

15. The method according to any one of claims 12 to 14, further comprising sending a (154) refresh instruction to the transmitting device.

16. The method according to claim 15, wherein, The refresh instruction is sent before receiving the CVS, which includes the first image and the second image.

17. The method according to any one of claims 12 to 16, further comprising: The detection (152) indicates that the video group of the video has been lost or one or more images in the video have been corrupted; and In response to the detection, the refresh instruction (154) is sent to the transmitting device to refresh the video of the bitstream.

18. The method according to any one of claims 12 to 17, further comprising decoding (158c) a third image of the CVS from the bitstream, wherein the third image: Follow the first image and the second image in the decoding order; The image is decoded into an inter-frame coded image by prediction from one or both of the first and second images; and Encoded at a third resolution, different from both the first and second resolutions.

19. The method according to any one of claims 12 to 18, further comprising preventing the transmission (162) of the decoded first image to the display device for display.

20. The method according to any one of claims 1 to 19, wherein, The refresh instruction includes a request for one of the following: In-Frame Request (FIR) message; Image Loss Indication (PLI) message; and Slice Loss Indication (SLI) message.

21. The method according to any one of claims 1 to 20, wherein, The refresh instruction requests the first image in a Real-time Transport Protocol (RTP) session, a Dynamic Adaptive Streaming (DASH) session based on Hypertext Transfer Protocol (HTTP), or an HTTP Live Streaming (HLS) session.

22. The method according to any one of claims 1 to 21, wherein, The refreshed image is an Intra-Frame Random Access Image (IRAP) image or a Gradual Decoding Refresh (GDR) image.

23. The method according to any one of claims 1 to 22, wherein, One or both of the following: Predict the second image from the first image using reference image resampling (RPR); and The third image is predicted from one or both of the first and second images using RPR.

24. The method according to any one of claims 1 to 23, wherein, One or more of the following: The first resolution is lower than the second resolution; The second resolution is lower than the third resolution; and At least one of the height and width of the first image having the first resolution is smaller than the corresponding height and width of the second image having the second resolution.

25. The method according to any one of claims 1 to 24, wherein, The refresh instruction includes one or more of the following: A request from the transmitting device to refresh the video bitstream; The request to begin encoding the video into the CVS in the bitstream; Request to send an intra-frame encoded image; The CVS request is sent in the bitstream; Send a request to directly refresh the intra-frame encoded images of the video; Gradually refresh the video request; The specified maximum resolution; The specified minimum resolution; Requests or authorizations for using RPR for prediction; Preferred bitrate per image; Preferred bit rate per second; Maximum bitrate per image; Maximum bit rate per second; In response to the encoder's request to set the resolution based on a specified bit rate; An indication that the encoder sets its resolution based on a specified bit rate; A request to encode intra-frame images at a resolution lower than that of inter-frame encoded images; A request to encode inter-frame images at a resolution higher than that of intra-frame encoded images; A request to start encoding at an initial resolution and increase to subsequent resolutions higher than the initial resolution; and The request to start encoding at the initial resolution and increment the resolution to the subsequent resolutions across multiple images.

26. The method according to any one of claims 1 to 25, wherein, The first image completely refreshes the video, or the first image initiates a refresh of the video, wherein, in the decoding order, subsequent images after the first image complete the refresh of the video.

27. The method according to any one of claims 1 to 26, wherein, The first image includes multiple refresh images, which include Progressive Decoding Refresh (GDR) images and recovery point images.

28. A transmitting apparatus (200) for encoding images into a encoded video sequence (CVS) in a bitstream (130), the transmitting apparatus being configured to: (144) of the CVS is generated in the bitstream, wherein, In order to generate the CVS, the sending device is configured to: The first image is encoded (144a) into the CVS in the bitstream at a first resolution, wherein the first image is encoded as a refresh image, and wherein at least a portion of the first image is intra-frame encoded; and The second image is encoded (144b) into the CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in the decoding order, and the second image is encoded as an inter-frame coded image by prediction from the first image; as well as The CVS of the bit stream is sent (146) to the receiving device.

29. The transmitting device according to claim 28, wherein, The encoder is also configured to perform the method according to any one of claims 2 to 11 and 15 to 27.

30. A transmitting apparatus (200) for encoding images into a encoded video sequence (CVS) in a bitstream (130), the transmitting apparatus comprising: A communication circuit (202) configured to communicate with a receiving device (300); as well as Processing circuit (204), operably connected to the communication circuit and configured to: The CVS is generated (144) in the bitstream, wherein, in order to generate the CVS, the processing circuit is configured as follows: Encoding (144a) a first image into the CVS in the bitstream at a first resolution, wherein the first image is encoded as a refresh image, and wherein at least a portion of the first image is intra-frame encoded; and The second image is encoded (144b) into the CVS in the bitstream at a second resolution different from the first resolution, wherein the second image follows the first image in decoding order, and the second image is encoded as an inter-frame coded image by prediction from the first image; and The CVS of the bit stream is sent (146) to the receiving device (300).

31. The transmitting device according to claim 30, wherein, The processing circuit is also configured to perform the method according to any one of claims 2 to 11 and 15 to 27.

32. A computer program (210) comprising instructions which, when executed on the processing circuitry of a transmitting device (200), cause the transmitting device to perform the method according to any one of claims 1 to 11 and 15 to 27.

33. A carrier comprising the computer program (210) of claim 32, wherein, The carrier is one of electronic signals, optical signals, radio signals, or computer-readable storage media.

34. A non-transitory computer-readable storage medium (208) comprising a computer program (210) stored thereon, the computer program comprising executable instructions which, when executed by a processing circuit (204) in a transmitting device (200), cause the transmitting device to perform the method according to any one of claims 1 to 11 and 15 to 27.

35. A receiving device (300) having a decoder (306) for decoding images from a encoded video sequence (CVS) in a bitstream (130), the receiving device being configured to: Decode (158a) the first image of CVS from the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; Decode (158b) the second image of the CVS from the bitstream, wherein the second image: Follow the first image in the decoding order; It has a second resolution that is different from the first resolution; and It is decoded into an inter-frame coded image predicted based on the first image; as well as Output (160) the decoded second image.

36. The receiving device according to claim 35, wherein, The receiving device is further configured to perform the method according to any one of claims 12 to 27.

37. A receiving apparatus (300) for decoding images from a encoded video sequence (CVS) in a bitstream (130), the receiving apparatus comprising: A communication circuit (302) configured to communicate with a transmitting device (200); as well as Processing circuit (304), operably connected to the communication circuit and configured to: Decode (158a) the first image of the CVS from the bitstream, wherein the first image is encoded as a refresh image and has a first resolution; Decode (158b) the second image of the CVS from the bitstream, wherein the second image: Follow the first image in the decoding order; It has a second resolution that is different from the first resolution; and It is decoded into an inter-frame coded image predicted based on the first image; as well as Output (160) the decoded second image.

38. The receiving device according to claim 37, wherein, The processing circuit is also configured to perform the method according to any one of claims 12 to 27.

39. A computer program (310) comprising instructions, which, when executed on a processing circuit (304) of a receiving device (300) having a decoder (306), causes the receiving device to perform the method according to any one of claims 12 to 27.

40. A carrier comprising the computer program of claim 39, wherein, The carrier is one of electronic signals, optical signals, radio signals, or computer-readable storage media.

41. A non-transitory computer-readable storage medium (308) comprising a computer program (310) stored thereon, the computer program comprising executable instructions which, when executed by a processing circuit (304) of a receiving device (300) having a decoder (306), cause the receiving device to perform the method according to any one of claims 12 to 27.

Images