Reference picture management in layered video coding - Patents.com

Abstract

To disclose a video coding mechanism.SOLUTION: The present mechanism includes receiving a bit stream having a reference picture list structure having a current picture and an inter-layer reference picture flag. The present mechanism determines, on the basis of the inter-layer reference picture flag, that an entry in the reference picture list structure related to the current picture is an inter-layer reference picture (ILRP) entry. The current picture is decoded on the basis of the inter-layer reference picture shown by an entry in the reference picture list structure when the entry is an ILRP entry. The current picture is transferred for display as a part of a decoding video sequence.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a divisional application of Japanese Patent Application No. JP2021-570948 filed on May 19, 2020, which is a continuation of International Application No. PCT / US2020 / 033595, entitled "Reference Picture Management In Layered Video Coding", filed on May 19, 2020 by Ye-Kui Wang et al., which claims priority to U.S. Provisional Patent Application No. 62 / 854,827, entitled "Reference Picture Management In Layered Video Coding", filed on May 30, 2019 by Ye-Kui Wang. All of the aforementioned patent applications are incorporated herein by reference in their entirety.

[0002] FIELD This disclosure relates generally to video coding, and more particularly to reference picture management when employed in inter-layer prediction in video coding. [Background technology]

[0003] The amount of video data required to depict even a relatively short video can be substantial, which can pose difficulties when the data is to be streamed or otherwise communicated across communication networks with limited bandwidth capacity. Thus, video data is typically compressed before being communicated across modern telecommunications networks. The size of the video can also be an issue when the video is stored on a storage device, since memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data prior to transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques are desirable that improve compression ratios with little sacrifice in image quality. Summary of the Invention [Means for solving the problem]

[0004] In one embodiment, the present disclosure includes a method implemented in a decoder, the method comprising: receiving, by a receiver of the decoder, a bitstream comprising a current picture and a reference picture list structure comprising an inter-layer reference picture flag; determining, by a processor of the decoder, based on the inter-layer reference picture flag, that an entry in the reference picture list structure associated with the current picture is an inter-layer reference picture (ILRP) entry; and, when the entry is an ILRP entry, decoding, by the processor, the current picture based on the inter-layer reference picture indicated by the entry in the reference picture list structure.

[0005] A video coding system may code a picture according to inter prediction. In inter prediction, a picture is coded with reference to another picture. The picture being coded is called a current picture, and a picture used as a reference is called a reference picture. Some video coding systems keep track of the reference pictures by employing a reference picture list. Some video coding systems employ scalable video coding. In scalable video coding, a video sequence is coded as a base layer and one or more enhancement layers. In this context, a picture may be divided into parts that reside in different layers. For example, a picture of a lower layer version may be of lower quality than a picture of a higher layer version. Furthermore, a picture of a lower layer version may be smaller (e.g., have a smaller width and / or height) than a picture of a higher layer version. When using layers, a picture in a current layer may be coded according to intra prediction (without a reference picture), may be coded according to inter prediction with reference to a reference picture in the same layer, or may be coded according to inter-layer prediction with reference to a reference picture in a different layer. Some reference picture management systems may not be configured to manage reference pictures used in inter-layer prediction. This example includes a mechanism for managing a reference picture list when employing inter-layer prediction. The reference picture list may be configured to include an entry for each reference picture used by the coded video. A flag may then be employed to indicate whether each entry includes an inter-predicted reference picture or an inter-layer reference picture. In one example, a group of reference picture flags for a syntax structure that includes a reference picture list may be indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i]. Additionally, an ILRP layer indicator may be used to indicate which layer includes the indicated inter-layer reference picture.The decoder can then use the reference picture flag and the ILRP layer indicator to select an appropriate inter-layer reference picture to perform inter-layer prediction. Thus, the disclosed mechanism creates additional functionality in the encoder and / or decoder. Furthermore, the disclosed mechanism may increase coding efficiency, which may reduce processor, memory, and / or network resource usage in the encoder and / or decoder.

[0006] Optionally, in any of the aforementioned aspects, another implementation form of the aspect provides that the reference picture list structure further comprises an ILRP layer indicator, and the method further comprises a step of determining, by the processor, a layer of the inter-layer reference picture based on the ILRP layer indicator when the entry is an ILRP entry.

[0007] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the reference picture list structure is denoted as ref_pic_list_struct(listIdx, rplsIdx), where listIdx identifies a reference picture list, rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure.

[0008] Optionally, in any of the aforementioned aspects, another implementation form of the aspect provides that the inter-layer reference picture flag is indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i], where when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 1, and when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 0.

[0009] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that when the entry is not an ILRP entry, the current picture is further decoded by the processor according to intra-layer prediction based on the intra-layer reference picture indicated by the entry in the reference picture list structure.

[0010] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that ref_pic_list_struct( listIdx, rplsIdx ) and inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] are included in the bitstream in a sequence parameter set (SPS).

[0011] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the inter-layer reference picture is in the same access unit (AU) as the current picture, and the inter-layer reference picture is associated with a lower layer identifier than the current picture.

[0012] In one embodiment, the present disclosure includes a method implemented in an encoder, the method comprising the steps of: encoding a current picture into a bitstream by a processor of the encoder, where the current picture is encoded according to inter-layer prediction based on an inter-layer reference picture; encoding a reference picture list structure into the bitstream by the processor, where the reference picture list structure comprises a plurality of entries for a plurality of reference pictures including an entry associated with the current picture and indicates the inter-layer reference picture; encoding an inter-layer reference picture flag into the bitstream by the processor, where the inter-layer reference picture flag indicates that the entry associated with the current picture is an ILRP entry; and storing the bitstream by a memory coupled to the processor for communication towards a decoder.

[0013] A video coding system may code a picture according to inter prediction. In inter prediction, a picture is coded with reference to another picture. The picture being coded is called a current picture, and a picture used as a reference is called a reference picture. Some video coding systems keep track of the reference pictures by employing a reference picture list. Some video coding systems employ scalable video coding. In scalable video coding, a video sequence is coded as a base layer and one or more enhancement layers. In this context, a picture may be divided into parts that reside in different layers. For example, a picture of a lower layer version may be of lower quality than a picture of a higher layer version. Furthermore, a picture of a lower layer version may be smaller (e.g., have a smaller width and / or height) than a picture of a higher layer version. When using layers, a picture in a current layer may be coded according to intra prediction (without a reference picture), may be coded according to inter prediction with reference to a reference picture in the same layer, or may be coded according to inter-layer prediction with reference to a reference picture in a different layer. Some reference picture management systems may not be configured to manage reference pictures used in inter-layer prediction. This example includes a mechanism for managing a reference picture list when employing inter-layer prediction. The reference picture list may be configured to include an entry for each reference picture used by the coded video. A flag may then be employed to indicate whether each entry includes an inter-predicted reference picture or an inter-layer reference picture. In one example, a group of reference picture flags for a syntax structure that includes a reference picture list may be indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i]. Additionally, an ILRP layer indicator may be used to indicate which layer includes the indicated inter-layer reference picture.The decoder can then use the reference picture flag and the ILRP layer indicator to select an appropriate inter-layer reference picture to perform inter-layer prediction. Thus, the disclosed mechanism creates additional functionality in the encoder and / or decoder. Furthermore, the disclosed mechanism may increase coding efficiency, which may reduce processor, memory, and / or network resource usage in the encoder and / or decoder.

[0014] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides further comprising encoding, by the processor, an ILRP layer indicator into the bitstream, the ILRP layer indicator indicating a layer of the inter-layer reference picture.

[0015] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the reference picture list structure is denoted as ref_pic_list_struct(listIdx, rplsIdx), where listIdx identifies a reference picture list, rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure that returns an entry based on listIdx and rplsIdx.

[0016] Optionally, in any of the aforementioned aspects, another implementation form of the aspect provides that the inter-layer reference picture flag is indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i], where when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 1, and when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 0.

[0017] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that ref_pic_list_struct( listIdx, rplsIdx ) and inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] are encoded in the bitstream within the SPS.

[0018] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that ref_pic_list_struct(listIdx, rplsIdx) and inter_layer_ref_pic_flag[listIdx][rplsIdx][i] are encoded in a header associated with the current picture.

[0019] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the inter-layer reference picture is in the same AU as the current picture, and the inter-layer reference picture is associated with a lower layer identifier than the current picture.

[0020] In one embodiment, the present disclosure includes a video coding device comprising a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform the method of any of the aforementioned aspects.

[0021] In one embodiment, the present disclosure includes a non-transitory computer-readable medium comprising a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, causes the video coding device to perform a method of any of the aforementioned aspects.

[0022] In one embodiment, the present disclosure includes a decoder comprising: receiving means for receiving a bitstream comprising a current picture, a reference picture list structure, and an inter-layer reference picture flag; determining means for determining, based on the inter-layer reference picture flag, whether an entry in the reference picture list structure associated with the current picture is an ILRP entry; decoding means for decoding the current picture according to inter-layer prediction based on the inter-layer reference picture indicated by the entry in the reference picture list structure when the entry is an ILRP entry; and forwarding means for forwarding the current picture for display as part of a decoded video sequence.

[0023] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the decoder is further configured to perform the method of any of the aforementioned aspects.

[0024] In one embodiment, the present disclosure includes an encoder comprising: encoding means for encoding a current picture into a bitstream, where the current picture is encoded according to inter-layer prediction based on an inter-layer reference picture; encoding a reference picture list structure into the bitstream, where the reference picture list structure comprises a plurality of entries for a plurality of reference pictures including an entry associated with the current picture and indicates the inter-layer reference picture; and encoding an inter-layer reference picture flag into the bitstream, where the inter-layer reference picture flag indicates that the entry associated with the current picture is an ILRP entry, as well as storage means for storing the bitstream for communication towards a decoder.

[0025] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the encoder is further configured to perform the method of any of the aforementioned aspects.

[0026] For clarity, any one of the above embodiments may be combined with any one or more of the other embodiments above to produce new embodiments within the scope of the present disclosure.

[0027] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

[0028] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts, in which: [Brief description of the drawings]

[0029] [Figure 1] 4 is a flowchart of an exemplary method for coding a video signal. [Diagram 2]1 is a schematic diagram of an example coding and decoding (codec) system for video coding. [Diagram 3] FIG. 1 is a schematic diagram illustrating an exemplary video encoder. [Figure 4] FIG. 2 is a schematic diagram illustrating an exemplary video decoder. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example of unidirectional inter prediction. [Figure 6] FIG. 1 is a schematic diagram illustrating an example of bidirectional inter prediction. [Figure 7] FIG. 2 is a schematic diagram illustrating an example of layer-based prediction. [Figure 8] FIG. 2 is a schematic diagram illustrating an exemplary reference picture list structure. [Figure 9] 1 is a schematic diagram illustrating an example bitstream including coding tool parameters for supporting inter-layer prediction. [Figure 10] 1 is a schematic diagram of an example video coding device. [Figure 11] 1 is a flowchart of an example method for encoding a video sequence into a bitstream according to inter-layer prediction. [Figure 12] 1 is a flowchart of an example method for decoding a video sequence from a bitstream when employing inter-layer prediction. [Figure 13] 1 is a schematic diagram of an example system for coding a video sequence of images in a bitstream when employing inter-layer prediction; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Although exemplary implementations of one or more embodiments are provided below, it should be understood at the outset that the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The present disclosure should in no way be limited to the exemplary implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations shown and described herein, but may be amended within the scope of the appended claims together with their full scope of equivalents.

[0031] The following terms are defined as follows, unless used in the present specification to the contrary. In particular, the following definitions are intended to provide additional clarity to the present disclosure. However, terms may be explained differently in different contexts. Therefore, the following definitions should be considered as supplemental and not as limiting any other definitions given to such terms in the present specification.

[0032] A bitstream is a sequence of bits that includes video data that is compressed for transmission between an encoder and a decoder. An encoder is a device configured to employ an encoding process to compress video data into a bitstream. A decoder is a device configured to employ a decoding process to reconstruct video data from the bitstream for display. A picture is an array of luma samples and / or an array of chroma samples that make up a frame or a field of a frame. For clarity of explanation, a picture being encoded or decoded can be referred to as a current picture. A reference picture is a picture that includes reference samples that can be used when coding other pictures by reference with inter-prediction and / or inter-layer prediction. A reference picture list is a list of reference pictures used for inter-prediction and / or inter-layer prediction. Some video coding systems refer to two picture lists that can be denoted as Reference Picture List 1 and Reference Picture List 0. A reference picture list structure is an addressable syntax structure that includes multiple reference picture lists. A layer is a group of pictures that all relate to similar value characteristics, such as similar size, quality, resolution, signal-to-noise ratio, capacity, etc. A layer identifier (ID) is an item of data associated with a picture and indicating that the picture is part of a indicated layer. Inter-prediction is a mechanism for coding samples of a current picture by reference to indicated samples in a reference picture different from the current picture, but the reference picture and the current picture are in the same layer. In an inter-layer context, inter-prediction can also be referred to as intra-layer prediction, since both the current picture and the reference picture are in the same layer. Inter-layer prediction is a mechanism for coding samples of a current picture by reference to indicated samples in a reference picture, but the current picture and the reference picture are in different layers and therefore have different layer IDs. An inter-layer reference picture is a reference picture used for inter-layer prediction.Some video coding systems may require that a current picture and an associated inter-layer reference picture are included in the same access unit (AU). A reference picture list structure entry is an addressable location in a reference picture list structure that indicates a reference picture associated with the reference picture list. An inter-layer reference picture (ILRP) entry is an entry that contains a reference picture used for inter-layer prediction. An inter-layer reference picture flag is data that indicates that a reference picture in an entry of a reference picture list structure is an inter-layer reference picture. An ILRP layer indicator is data that indicates the layer associated with the inter-layer reference picture referenced by the current picture. A slice header is a part of a coded slice that includes data elements related to all video data in a tile represented in the slice. A sequence parameter set (SPS) is a parameter set that includes data related to a sequence of pictures. An AU is a set of one or more coded pictures that are associated with the same display time (e.g., the same picture order count) for output from a decoded picture buffer (DPB) (e.g., for display to a user). A decoded video sequence is a sequence of pictures that has been reconstructed by a decoder in preparation for presentation to a user.

[0033] The following acronyms are used in this specification: Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Joint Video Experts Team (JVET), Motion-Constrained Tile Set (MCTS), Maximum Transfer Unit (MTU), Network Abstraction Layer (NAL), Picture Order Count (POC), Random Access Decodable Leading (RADL) picture, Random Access Skipped Leading (RASL) picture, Raw Byte Sequence Payload (RBSP), Reference Picture List (RPL), Sequence Parameter Set (SPS), Video Coding Layer (VCL), Video Coding Layer (VCL), Video Streaming Layer (VST ... Layer), Versatile Video Coding (VVC), and Working Draft (WD) are used.

[0034] Many video compression techniques may be employed to reduce the size of video files with minimal loss of data. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or remove data redundancy in a video sequence. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded unidirectionally predictive (P) or bidirectionally predictive (B) slice of a picture may be coded by employing spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame and / or an image, and a reference picture may be referred to as a reference frame and / or a reference image. Spatial or temporal prediction results in a predictive block representing the image block. The residual data represents pixel differences between the original image block and the predictive block. Thus, inter-coded blocks are coded according to a motion vector pointing to a block of reference samples forming the predictive block, and the residual data indicating the difference between the coded block and the predictive block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain. These result in residual transform coefficients, which may be quantized. The quantized transform coefficients may be initially arranged in a two-dimensional array. The quantized transform coefficients may be scanned to generate a one-dimensional vector of transform coefficients. To achieve even further compression, entropy coding may be applied.Such video compression techniques are described in more detail below.

[0035] To ensure that the encoded video can be accurately decoded, the video is encoded and decoded according to a corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), and Multiview Video Coding Plus Depth (MVC+D), as well as three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC Joint Video Experts Team (JVET) has begun developing a video coding standard called Versatile Video Coding (VVC). VVC is included in the Working Drafts (WDs) including JVET-N1001-v6.

[0036] A video coding system may code a picture according to inter prediction. In inter prediction, a picture is coded by reference to another picture. The picture being coded is called a current picture, and a picture used as a reference is called a reference picture. Some video coding systems keep track of reference pictures by employing a reference picture list. Furthermore, some video coding systems employ scalable video coding. In scalable video coding, a video sequence is coded as a base layer and one or more enhancement layers. In this context, a picture may be divided into multiple pictures that exist in different layers. For example, a picture of a lower layer version may be of lower quality than a picture of a higher layer version. Furthermore, a picture of a lower layer version may be smaller (e.g., have a smaller width and / or height) than a picture of a higher layer version. When using layers, a picture in a current layer may be coded according to intra prediction (without a reference picture), may be coded according to inter prediction with reference to a reference picture in the same layer, or may be coded according to inter-layer prediction with reference to a reference picture in a different layer. However, reference picture lists may not be designed to describe reference pictures across multiple layers.

[0037] An exemplary mechanism for managing a reference picture list when employing inter-layer prediction is disclosed herein. For example, a reference picture list may be configured to include an entry for each reference picture used by the coded video. A flag may then be employed to indicate whether each entry includes an inter-predicted reference picture (for referencing within the same level) or an inter-layer reference picture (for referencing between layers). In one example, a group of reference picture flags for a syntax structure including a reference picture list may be indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i]. The inter_layer_ref_pic_flag[listIdx][rplsIdx][i] flag may be set equal to 1 when the i-th entry in the reference picture structure is an ILRP entry, or may be set to 0 when the i-th entry in the reference picture structure is not an ILRP entry. Furthermore, an ILRP layer indicator may be used to indicate which layer includes the inter-layer reference picture indicated by the entry of the reference picture structure. A decoder may then use the reference picture flag and the ILRP layer indicator to select an appropriate inter-layer reference picture to perform inter-layer prediction. Thus, the disclosed mechanism creates additional functionality in the encoder and / or decoder. Furthermore, the disclosed mechanism may increase coding efficiency, which may reduce processor, memory, and / or network resource usage in the encoder and / or decoder.

[0038] 1 is a flow chart of an exemplary operational method 100 of coding a video signal. Specifically, a video signal is encoded in an encoder. The encoding process compresses the video signal by employing various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted towards a user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to an end user. The decoding process generally mimics the encoding process exactly to allow the decoder to consistently reconstruct the video signal.

[0039] In step 101, a video signal is input into the encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. The video file may include both audio and video components. The video component includes a series of image frames that, when viewed in sequence, give the visual impression of motion. The frames include pixels that are represented in terms of light, referred to herein as luma components (or luma samples), and color, referred to herein as chroma components (or color samples). In some examples, the frames may also include depth values ​​to support three-dimensional displays.

[0040] In step 103, the video is partitioned into blocks. Partitioning involves subdividing pixels in each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame may first be partitioned into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels by 64 pixels). CTUs include both luma and chroma samples. A coding tree may be employed to partition the CTUs into blocks and then recursively subdivide the blocks until a configuration that supports further encoding is obtained. For example, the luma component of a frame may be subdivided until each block contains relatively uniform illumination values. Furthermore, the chroma component of a frame may be subdivided until each block contains relatively uniform color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

[0041] In step 105, various compression mechanisms are employed to compress the image blocks partitioned in step 103. For example, inter-prediction and / or intra-prediction may be employed. Inter-prediction is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Thus, a block representing an object in a reference frame does not need to be described repeatedly in adjacent frames. In particular, an object such as a table may remain in a constant position across multiple frames. Thus, the table may be described once and adjacent frames may refer back to the reference frame. A pattern matching mechanism may be employed to match objects across multiple frames. Furthermore, a moving object may be represented across multiple frames, for example, due to object movement or camera movement. As a specific example, a video may show a car moving across the screen across multiple frames. A motion vector may be employed to describe such movement. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in one frame to the coordinates of the object in the reference frame. Thus, inter-prediction may encode an image block in a current frame as a set of motion vectors indicating an offset from a corresponding block in a reference frame.

[0042] Intra prediction encodes blocks within a common frame. Intra prediction exploits the fact that luma and chroma components tend to cluster together within a frame. For example, green fragments in a portion of a tree tend to be located adjacent to similar fragments of green. Intra prediction employs multiple directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional mode indicates that the current block is similar / same as samples of neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks (e.g., planes) along a row / column may be interpolated based on neighboring blocks at the edge of the row. Planar mode effectively indicates a smooth transition of light / color across the row / column by employing a relatively constant gradient in changing values. DC mode is employed for boundary smoothing, indicating that the block is similar / same as the average value associated with samples of all neighboring blocks associated with the angular direction of the directional prediction mode. Thus, intra prediction blocks can represent image blocks as various prediction mode values ​​that are related, rather than actual values. Additionally, inter-prediction blocks can represent image blocks as motion vector values ​​rather than actual values. In either case, the prediction blocks may not exactly represent the image blocks in some cases. Any differences are stored in the residual blocks. To further compress the file, a transform may be applied to the residual blocks.

[0043] In step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. Block-based prediction as described above may result in the generation of blocky images at the decoder. Furthermore, the block-based prediction scheme may code a block and then reconstruct the coded block for later use as a reference block. The in-loop filtering scheme applies noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to the block / frame iteratively. These filters mitigate such blocking artifacts so that the coded file can be accurately reconstructed. Furthermore, these filters mitigate artifacts in the reconstructed reference block so that the artifacts are less likely to cause additional artifacts in subsequent blocks being coded based on the reconstructed reference block.

[0044] Once the video signal has been partitioned, compressed, and filtered, the resulting data is encoded in a bitstream in step 109. The bitstream includes the data described above, as well as any signaling data desired to support proper video signal reconstruction at the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in a memory for transmission to the decoder upon request. The bitstream may also be broadcast and / or multicast to multiple decoders. Creation of the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 may occur continuously and / or simultaneously across many frames and blocks. The order shown in FIG. 1 is presented for clarity and simplicity of explanation and is not intended to limit the video coding process to a particular order.

[0045] The decoder receives the bitstream and starts the decoding process in step 111. Specifically, the decoder employs an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. The decoder employs syntax data from the bitstream to determine a partition for the frame in step 111. The partition should match the result of the block partition in step 103. The entropy encoding / decoding as employed in step 111 is now described. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible options based on the spatial arrangement of values ​​in the input image. Signaling the exact choice may employ a number of bins. A bin, as used herein, is a binary value (e.g., a bit value that may change depending on the context) that is treated as a variable. Entropy coding allows the encoder to discard any options that are not clearly viable for a particular case, leaving a set of acceptable options. Each acceptable option is then assigned a codeword. The length of the codeword is based on the number of allowable options (e.g., one bin for two options, two bins for three to four options, etc.). The encoder then encodes the codeword for the selected option. This scheme reduces the size of the codeword when the codeword is as large as desired to uniquely indicate a choice from a small subset of allowable options, as opposed to uniquely indicating a choice from a potentially large set of all possible options. The decoder then decodes the choices by determining the set of allowable options in a manner similar to the encoder. By determining the set of allowable options, the decoder can read the codeword and determine the selection made by the encoder.

[0046] In step 113, the decoder performs block decoding. Specifically, the decoder employs an inverse transform to generate a residual block. Then, the decoder employs the residual block and a corresponding prediction block to reconstruct an image block according to the partition. The prediction block may include both intra-prediction blocks and inter-prediction blocks, as generated in the encoder in step 105. The reconstructed image block is then placed in a frame of the reconstructed video signal according to the partition data determined in step 111. The syntax for step 113 may also be signaled in the bitstream via entropy coding as described above.

[0047] In step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to step 107 in the encoder. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and an SAO filter may be applied to the frames to remove blocking artifacts. Once the frames have been filtered, the video signal may be output to a display in step 117 for viewing by an end user.

[0048] 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functionality to support the implementation of the operational method 100. The codec system 200 is generalized to show components employed in both an encoder and a decoder. The codec system 200 receives and segments a video signal as described with respect to steps 101 and 103 in the operational method 100, which results in a segmented video signal 201. The codec system 200 then compresses the segmented video signal 201 into a coded bitstream when acting as an encoder as described with respect to steps 105, 107, and 109 in the method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream as described with respect to steps 111, 113, 115, and 117 in the operational method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown. In FIG. 2, the black lines indicate the movement of data to be encoded / decoded, and the dashed lines indicate the movement of control data that controls the operation of the other components. All of the components of the codec system 200 may reside in an encoder. A decoder may include a subset of the components of the codec system 200. For example, a decoder may include the intra-picture prediction component 217, the motion compensation component 219, the scaling and inverse transform component 229, the in-loop filter component 225, and a decoded picture buffer component 223. These components are now described.

[0049] The partitioned video signal 201 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree employs various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks may then be further subdivided into even smaller blocks. The blocks may be referred to as nodes on the coding tree. Larger parent nodes are partitioned into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node / coding tree. The partitioned blocks may be included in coding units (CUs) in some cases. For example, a CU may be a sub-part of a CTU that includes a luma block, a red differential chroma (Cr) block, and a blue differential chroma (Cb) block, along with corresponding syntax instructions for the CU. The partitioning modes may include a binary tree (BT), a triple tree (TT), and a quad tree (QT), which are employed to partition a node into two, three, or four child nodes, respectively, with shapes that vary depending on the partitioning mode employed. The segmented video signal 201 is forwarded to a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

[0050] The generic coder control component 211 is configured to make decisions related to the coding of images of a video sequence into a bitstream according to applicable constraints. For example, the generic coder control component 211 manages the optimization of bitrate / bitstream size versus reconstruction quality. Such decisions may be made based on storage space / bandwidth availability and image resolution requirements. The generic coder control component 211 also manages buffer utilization in light of transmission rate to mitigate buffer underrun and overrun problems. To manage these problems, the generic coder control component 211 manages segmentation, prediction, and filtering by other components. For example, the generic coder control component 211 may dynamically increase the amount of compression computation to increase resolution and increase bandwidth usage, or decrease the amount of compression computation to decrease resolution and bandwidth usage. Thus, the generic coder control component 211 controls other components of the codec system 200 to balance video signal reconstruction quality with bitrate importance. The generic coder control component 211 creates control data that controls the operation of other components. Control data is also encoded in the bitstream and forwarded to the header formatting and CABAC component 231 for signaling parameters for decoding at the decoder.

[0051] The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to perform temporal prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0052] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by the motion estimation component 221 is a process of generating motion vectors that estimate motion for a video block. A motion vector may indicate, for example, the displacement of a coded object relative to a predictive block. A predictive block is a block that is found to closely match a block to be coded in terms of pixel differences. A predictive block is sometimes referred to as a reference block. Such pixel differences may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metrics. HEVC employs several coded objects including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU may be partitioned into CTBs, which may then be partitioned into CBs for inclusion in a CU. A CU may be coded as a prediction unit (PU), which includes predictive data, and / or a transform unit (TU), which includes transformed residual data for the CU. The motion estimation component 221 generates the motion vectors, PUs, and TUs by using rate-distortion analysis as part of a rate-distortion optimization process. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for a current block / frame, and may select the reference block, motion vector, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of the video reconstruction (e.g., the amount of data loss due to compression) with the coding efficiency (e.g., the size of the final encoding).

[0053] In some examples, the codec system 200 may calculate values ​​for sub-integer pixel positions of a reference picture stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Thus, the motion estimation component 221 may perform motion search for full-pixel positions and fractional pixel positions and output motion vectors with fractional pixel accuracy. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of the predictive blocks of the reference pictures. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header formatting and CABAC component 231 for encoding, and to the motion compensation component 219.

[0054] The motion compensation performed by the motion compensation component 219 may involve fetching or generating a predictive block based on a motion vector determined by the motion estimation component 221. Again, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the predictive block to which the motion vector points. A residual video block is then formed by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values. In general, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for both the chroma and luma components. The predictive block and the residual block are forwarded to the transform scaling and quantization component 213.

[0055] The partitioned video signal 201 is also sent to an intra-picture estimation component 215 and an intra-picture prediction component 217. As with the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated, but are illustrated separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict a current block relative to blocks in a current frame, as an alternative to the inter-prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames, as described above. Specifically, the intra-picture estimation component 215 determines an intra-prediction mode to be used to encode the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode for encoding the current block from the tested multiple intra-prediction modes. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0056] For example, the intra picture estimation component 215 calculates rate-distortion values ​​using a rate-distortion analysis for various intra prediction modes tested, and selects the intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (i.e., error) between the coding block and the original uncoded block that was coded to generate the coding block, as well as the bit rate (e.g., number of bits) used to generate the coding block. The intra picture estimation component 215 calculates a ratio from the distortion and rate for various coding blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block. In addition, the intra picture estimation component 215 may be configured to code the depth blocks of the depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

[0057] The intra picture prediction component 217 may generate a residual block from the prediction block based on a selected intra prediction mode determined by the intra picture estimation component 215 when implemented on an encoder, or may read the residual block from the bitstream when implemented on a decoder. The residual block includes value differences between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra picture estimation component 215 and the intra picture prediction component 217 may operate on both luma and chroma components.

[0058] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block to generate a video block comprising residual transform coefficient values. A wavelet transform, an integer transform, a subband transform, or other types of transforms may also be used. The transform may convert the residual information from a pixel value domain to a transform domain, such as a frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information, such that different frequency information is quantized with different granularity, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan of a matrix containing the quantized transform coefficients, which are forwarded to the header formatting and CABAC component 231 to be encoded in the bitstream.

[0059] The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transformation, and / or quantization to reconstruct a residual block in the pixel domain, for example, for later use as a reference block that may become a predictive block for another current block. The motion estimation component 221 and / or the motion compensation component 219 may compute a reference block by adding the residual block back to the corresponding predictive block for use in motion estimation of a later block / frame. A filter is applied to the reconstructed reference block to mitigate artifacts that occur during the scaling, quantization, and transformation. Such artifacts may potentially cause inaccurate predictions (and generate additional artifacts) when subsequent blocks are predicted.

[0060] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or to the reconstructed image block. For example, the transformed residual block from the scaling and inverse transform component 229 may be combined with the corresponding prediction block from the intra-picture prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter may then be applied to the reconstructed image block. In some examples, the filter may be applied to the residual block instead. As with the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are shown separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to a particular spatial region and include multiple parameters to adjust how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference block to determine where such filters should be applied and sets the corresponding parameters. Such data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be applied in the spatial / pixel domain (e.g., to reconstructed pixel blocks) or in the frequency domain, depending on the example.

[0061] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or predictive blocks are stored in the decoded picture buffer component 223 for later use in motion estimation, as described above. When operating as a decoder, the decoded picture buffer component 223 stores the reconstructed and filtered blocks and forwards them towards the display as part of the output video signal. The decoded picture buffer component 223 may be any memory device capable of storing predictive blocks, residual blocks, and / or reconstructed image blocks.

[0062] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission towards the decoder. In particular, the header formatting and CABAC component 231 generates various headers for encoding control data, such as general control data and filter control data. In addition, prediction data, including intra prediction and motion data, and residual data in the form of quantized transform coefficient data are all encoded in the bitstream. The final bitstream contains all information desired by the decoder to reconstruct the partitioned original video signal 201. Such information may also include an intra prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, an indication of the most probable intra prediction mode, an indication of partition information, etc. Such data may be encoded by employing entropy coding. For example, the information may be encoded by employing Context-Adaptive Variable Length Coding (CAVLC), CABAC, Syntax-Based Context-Adaptive Binary Arithmetic Coding (SBAC), Probabilistic Interval Partitioning Entropy (PIPE) coding, or another entropy coding technique. Following entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0063] 3 is a block diagram illustrating an example video encoder 300. The video encoder 300 may be employed to perform the encoding functions of the codec system 200 and / or to perform steps 101, 103, 105, 107, and / or 109 of the method of operation 100. The encoder 300 segments an input video signal, resulting in a segmented video signal 301, which is substantially similar to the segmented video signal 201. The segmented video signal 301 is then compressed and encoded into a bitstream by components of the encoder 300.

[0064] Specifically, the partitioned video signal 301 is forwarded to an intra-picture prediction component 317 for intra prediction. The intra-picture prediction component 317 may be substantially similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction based on a reference block in a decoded picture buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction block and the residual block from the intra-picture prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transforming and quantizing the residual block. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction block are forwarded (together with associated control data) to an entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231 .

[0065] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. An in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters as described with respect to the in-loop filter component 225. The filtered block is then stored in the decoded picture buffer component 323 for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially similar to the decoded picture buffer component 223.

[0066] 4 is a block diagram illustrating an example video decoder 400. The video decoder 400 may be employed to perform the decoding functions of the codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of the method of operation 100. The decoder 400 receives a bitstream, e.g., from the encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0067] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to perform an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may employ header information to provide a context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding a video signal, such as general control data, filter control data, partition information, motion data, prediction data, and quantized transform coefficients from the residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 for reconstruction into the residual block. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0068] The reconstructed residual block and / or predictive block are forwarded to an intra-picture prediction component 417 for reconstruction into an image block based on an intra-prediction operation. The intra-picture prediction component 417 may be similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 employs a prediction mode to identify the location of a reference block in a frame and applies the residual block to the result to reconstruct an intra-predicted image block. The reconstructed, intra-predicted image block and / or residual block, as well as the corresponding inter-prediction data, are forwarded to a decoded picture buffer component 423 via an in-loop filter component 425, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, the residual block, and / or the predictive block, and such information is stored in the decoded picture buffer component 423. The reconstructed image block from the decoded picture buffer component 423 is forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 employs a motion vector from a reference block to generate a prediction block, and applies a residual block to the result to reconstruct an image block. The resulting reconstructed block may also be forwarded to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks, which may be reconstructed into frames via the partition information. Such frames may also be placed into a sequence. The sequence is output to a display as a reconstructed output video signal.

[0069] FIG. 5 is a schematic diagram illustrating an example of unidirectional inter prediction 500, as performed, for example, in block compression step 105, block decoding step 113, motion estimation component 221, motion compensation component 219, motion compensation component 321, and / or motion compensation component 421 to determine a motion vector (MV).

[0070] The unidirectional inter prediction 500 employs a reference picture 530 having a reference block 531 to predict a current block 511 in a current picture 510. The reference picture 530 may be located temporally after the current picture 510 as shown (e.g., as a subsequent reference picture), but in some examples, may be located temporally before the current picture 510 (e.g., as a preceding reference picture). The current picture 510 is an exemplary frame / picture being coded / decoded at a particular time. The current picture 510 includes an object in the current block 511 that matches an object in the reference block 531 of the reference picture 530. The reference picture 530 is a picture employed as a reference for coding the current picture 510, and the reference block 531 is a block in the reference picture 530 that includes an object that is also included in the current block 511 of the current picture 510.

[0071] The current block 511 is any coding unit being coded / decoded at a specified point in the coding process. The current block 511 may be a whole partitioned block or a sub-block when employing an affine inter prediction mode. The current picture 510 is away from the reference picture 530 by some temporal distance (TD) 533. The TD 533 indicates the amount of time between the current picture 510 and the reference picture 530 in the video sequence and may be measured in units of pictures. The prediction information for the current block 511 may reference the reference picture 530 and / or the reference block 531 by a reference index indicating the direction and temporal distance between the pictures. Over the time period represented by the TD 533, an object in the current block 511 moves from one position in the current picture 510 to another position in the reference picture 530 (e.g., the position of the reference block 531). For example, the object may move along a motion trajectory 513, which is the direction of the object's movement in time. The motion vector 535 represents the direction and magnitude of the object's movement along the motion trajectory 513 over the TD 533. Thus, the coded motion vector 535, the reference block 531, and the residual comprising the difference between the current block 511 and the reference block 531 provide sufficient information to reconstruct the current block 511 and to identify its position in the current picture 510.

[0072] FIG. 6 is a schematic diagram illustrating an example of bidirectional inter prediction 600, as may be performed to determine the MV, e.g., in the block compression step 105, the block decoding step 113, the motion estimation component 221, the motion compensation component 219, the motion compensation component 321, and / or the motion compensation component 421.

[0073] Bidirectional inter prediction 600 is similar to unidirectional inter prediction 500, but employs a pair of reference pictures to predict a current block 611 in a current picture 610. Thus, the current picture 610 and the current block 611 are substantially similar to the current picture 510 and the current block 511, respectively. The current picture 610 is temporally positioned between a previous reference picture 620 that occurs before the current picture 610 in the video sequence and a subsequent reference picture 630 that occurs after the current picture 610 in the video sequence. The previous reference picture 620 and the subsequent reference picture 630 are otherwise substantially similar to the reference picture 530.

[0074] A current block 611 is aligned to a previous reference block 621 in a previous reference picture 620 and to a subsequent reference block 631 in a subsequent reference picture 630. Such alignment indicates that an object moves from a position in the previous reference block 621 to a position in the subsequent reference block 631 along a motion trajectory 613 and through the current block 611 over the course of the video sequence. The current picture 610 is separated from the previous reference picture 620 by some previous temporal distance (TD0) 623 and separated from the subsequent reference picture 630 by some subsequent temporal distance (TD1) 633. TD0 623 indicates the amount of time in picture units between the previous reference picture 620 and the current picture 610 in the video sequence. TD1 633 indicates the amount of time in picture units between the current picture 610 and the subsequent reference picture 630 in the video sequence. Thus, an object moves from a preceding reference block 621 to a current block 611 along a motion trajectory 613 over the time period indicated by TD0 623. An object also moves from a current block 611 to a subsequent reference block 631 along a motion trajectory 613 over the time period indicated by TD1 633. Prediction information for a current block 611 may reference a preceding reference picture 620 and / or a preceding reference block 621, and a subsequent reference picture 630 and / or a subsequent reference block 631, by a pair of reference indexes indicating the direction and temporal distance between the pictures.

[0075] A previous motion vector (MV0) 625 represents a direction and a magnitude of an object's movement along a motion trajectory 613 (e.g., between a previous reference picture 620 and a current picture 610) over TD0 623. A subsequent motion vector (MV1) 635 represents a direction and a magnitude of an object's movement along a motion trajectory 613 (e.g., between a current picture 610 and a subsequent reference picture 630) over TD1 633. Thus, in bidirectional inter prediction 600, a current block 611 may be coded and reconstructed by employing a previous reference block 621 and / or a subsequent reference block 631, MV0 625, and MV1 635.

[0076] 7 is a schematic diagram illustrating an example of layer-based prediction 700, as may be performed to determine MV, for example, in block compression step 105, block decoding step 113, motion estimation component 221, motion compensation component 219, motion compensation component 321, and / or motion compensation component 421. Layer-based prediction 700 is similar to unidirectional inter prediction 500 and / or bidirectional inter prediction 600, but is also performed between pictures in different layers.

[0077] Layer-based prediction 700 is applied between pictures 711, 712, 713, and 714 and pictures 715, 716, 717, and 718 in different layers. In the illustrated example, pictures 711, 712, 713, and 714 are part of layer N+1 732, and pictures 715, 716, 717, and 718 are part of layer N 731. A layer, such as layer N 731 and / or layer N+1 732, is a group of pictures that are all associated with similar values ​​of characteristics, such as similar size, quality, resolution, signal-to-noise ratio, capacity, etc. In the illustrated example, layer N+1 732 is associated with a larger image size than layer N 731. Thus, in this example, pictures 711, 712, 713, and 714 in layer N+1 732 have a larger picture size (e.g., larger height and width, and therefore more samples) than pictures 715, 716, 717, and 718 in layer N 731. However, such pictures may be separated between layer N+1 732 and layer N 731 by other characteristics. Although only two layers, namely layer N+1 732 and layer N 731, are illustrated, a set of pictures may be separated into any number of layers based on associated characteristics. Layer N+1 732 and layer N 731 may also be indicated by a layer ID. A layer ID is an item of data associated with a picture that indicates that the picture is part of the indicated layer. Thus, each picture 711-718 may be associated with a corresponding layer ID to indicate which layer N+1 732 or layer N 731 contains the corresponding picture.

[0078] Pictures 711-718 in different layers 731-732 are configured to be displayed alternatively. Thus, pictures 711-718 in different layers 731-732 may share the same picture order count and may be included in the same AU. As used herein, an AU is a set of one or more coded pictures that are associated with the same display time for output from the DPB. For example, if a smaller picture is desired, the decoder may decode and display picture 715 at the current display time, or if a larger picture is desired, the decoder may decode and display picture 711 at the current display time. Thus, pictures 711-714 in higher layer N+1 732 contain substantially the same image data as corresponding pictures 715-718 in lower layer N 731 (despite differences in picture size). Specifically, picture 711 contains substantially the same image data as picture 715, picture 712 contains substantially the same image data as picture 716, and so on.

[0079] Pictures 711-718 may be coded by reference to other pictures 711-718 in the same layer N 731 or N+1 732. Coding a picture with reference to another picture in the same layer results in inter prediction 723, which is substantially similar to unidirectional inter prediction 500 and / or bidirectional inter prediction 600. Inter prediction 723 is indicated by a solid arrow. For example, picture 713 may be coded by employing inter prediction 723 using one or two of pictures 711, 712, and / or 714 in layer N+1 732 as references, where one picture is referenced for unidirectional inter prediction and / or two pictures are references for bidirectional inter prediction. Furthermore, picture 717 may be coded by employing inter prediction 723 using one or two of pictures 715, 716, and / or 718 in layer N 731 as references, where one picture is referenced for unidirectional inter prediction and / or two pictures are references for bidirectional inter prediction. When a picture is used as a reference for another picture in the same layer when performing inter prediction 723, the picture may be referred to as a reference picture. For example, picture 712 may be a reference picture used to code picture 713 according to inter prediction 723. Inter prediction 723 may also be referred to as intra-layer prediction in a multi-layer context. Thus, inter prediction 723 is a mechanism for coding samples of a current picture by reference to indicated samples in a reference picture different from the current picture, but the reference picture and the current picture are in the same layer.

[0080] Pictures 711-718 may also be coded by reference to other pictures 711-718 in different layers. This process is also known as inter-layer prediction 721 and is indicated by dashed arrows. Inter-layer prediction 721 is a mechanism for coding samples of a current picture by reference to indicated samples in reference pictures, where the current picture and the reference pictures are in different layers and therefore have different layer IDs. For example, a picture in a lower layer N 731 may be used as a reference picture for coding a corresponding picture in a higher layer N+1 732. As a specific example, picture 711 may be coded by reference to picture 715 by inter-layer prediction 721. In such a case, picture 715 is used as an inter-layer reference picture. An inter-layer reference picture is a reference picture used for inter-layer prediction 721. In most cases, inter-layer prediction 721 is constrained such that a current picture, such as picture 711, can only use inter-layer reference pictures that are in the same AU and in lower layers, such as picture 715. When multiple layers (e.g., three or more) are available, inter-layer prediction 721 can encode / decode the current picture based on multiple inter-layer reference pictures at lower levels than the current picture.

[0081] A video encoder may employ layer-based prediction 700 to encode pictures 711-718 via many different combinations and / or permutations of inter-prediction 723 and inter-layer prediction 721. For example, picture 715 may be coded according to intra prediction. Pictures 716-718 may then be coded according to inter-prediction 723 by using picture 715 as a reference picture. Furthermore, picture 711 may be coded according to inter-layer prediction 721 by using picture 715 as an inter-layer reference picture. Pictures 712-714 may then be coded according to inter-prediction 723 by using picture 711 as a reference picture. Thus, a reference picture may serve as both a single layer reference picture and an inter-layer reference picture for different coding mechanisms. By coding pictures of higher layer N+1 732 based on pictures of lower layer N 731, higher layer N+1 732 can avoid employing intra prediction, which has much lower coding efficiency than inter prediction 723 and inter-layer prediction 721. Thus, the poor coding efficiency of intra prediction may be limited to pictures with the smallest / lowest quality and therefore limited to coding a minimum amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures may be indicated in entries of a reference picture list included in a reference picture list structure.

[0082] 8 is a schematic diagram illustrating an example reference picture list structure (RPL structure) 800. The RPL structure 800 may be employed to store indications of reference pictures and / or inter-layer reference pictures used in unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700. Thus, the RPL structure 800 may be employed by the codec system 200, the encoder 300, and / or the decoder 400 when performing the method 100.

[0083] The RPL structure 800 is an addressable syntax structure that includes multiple reference picture lists, such as RPL0 811 and RPL1 812. The RPL structure 800 may be stored in an SPS and / or slice header of a bitstream depending on the example. The reference picture lists, such as RPL0 811 and RPL1 812, are lists of reference pictures used for inter prediction and / or inter-layer prediction. RPL0 811 and RPL1 812 may each include multiple entries 815. The reference picture list structure entries 815 are addressable locations in the RPL structure 800 that indicate the reference pictures associated with the reference picture lists, such as RPL0 811 and / or RPL1 812. Each entry 815 may include a picture order count (POC) value (or other pointer value) that references a picture used for inter prediction. Specifically, references to pictures used by unidirectional inter prediction 500 are stored in RPL0 811, and references to pictures used by bidirectional inter prediction 600 are stored in both RPL0 811 and RPL1 812. For example, bidirectional inter prediction 600 may use one reference picture indicated by RPL0 811 and one reference picture indicated by RPL1 812.

[0084] In a specific example, the RPL structure 800 may be denoted as ref_pic_list_struct( listIdx, rplsIdx ), where listIdx 821 identifies reference picture lists RPL0 811 and / or RPL1 812, and rplsIdx 825 identifies an entry 815 in the reference picture list. Thus, ref_pic_list_struct is a syntax structure that returns an entry 815 based on listIdx 821 and rplsIdx 825. An encoder may encode a portion of the RPL structure 800 for each non-intra coded slice in a video sequence. A decoder may then resolve the corresponding portion of the RPL structure 800 before decoding each non-intra coded slice in the coded video sequence.

[0085] As described above, a reference picture may be referenced for inter-prediction. Furthermore, a reference picture may be used as an inter-layer reference picture for inter-layer prediction. Thus, the RPL structure 800 is modified by including an ILRP flag 833 and an ILRP layer indicator 835 to support inter-layer prediction. The ILRP flag 833 is data that indicates whether a picture referenced by a corresponding entry 815 of the RPL structure 800 is an inter-layer reference picture used for inter-layer prediction. Thus, an encoder can use the ILRP flag 833 to indicate whether each entry 815 should be treated as an ILRP entry. An ILRP entry is any entry 815 that references an inter-layer reference picture used for inter-layer prediction. Furthermore, a decoder can use the ILRP flag 833 to determine whether each entry 815 in the RPL structure 800 is an ILRP entry. In a specific example, the ILRP flag 833 is denoted as inter_layer_ref_pic_flag[listIdx][rplsIdx][i], where i is a counter variable where each value indicates a corresponding entry 815. When the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 1. Additionally, when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 0.

[0086] As mentioned above, an inter-layer reference picture may be located in the same AU and have the same POC value as the current picture being coded / decoded. Thus, the RPL structure 800 may not be modified to add the POC value of the corresponding inter-layer reference picture. However, a decoder may not be able to infer which layer or layers of an AU contain the appropriate inter-layer reference picture for decoding the current picture. For this purpose, the ILRP layer indicator 835 is included. The ILRP layer indicator 835 is data indicating the layer associated with the inter-layer reference picture referenced by the current picture. Specifically, the ILRP layer indicator 835 may indicate one or more layers for each entry 815 that is an ILRP entry as indicated by the ILRP flag 833. Thus, an encoder may code the ILRP layer indicator 835 into the bitstream to indicate the layer of one or more inter-layer reference pictures associated with the entry 815. In addition, when the entry 815 is an ILRP entry as indicated by the ILRP flag 833, the decoder can determine the layer of the inter-layer reference picture based on the ILRP layer indicator 835. Thus, the addition of the ILRP flag 833 and the ILRP layer indicator 835 to the RPL structure 800 provides sufficient information to enable the RPL structure 800 to manage reference pictures for inter-layer prediction.

[0087] 9 is a schematic diagram illustrating an example bitstream 900 including coding tool parameters for supporting inter-layer prediction. For example, the bitstream 900 may be generated by the codec system 200 and / or the encoder 300 for decoding by the codec system 200 and / or the decoder 400. As another example, the bitstream 900 may be generated by the encoder in step 109 of the method 100 for use by the decoder in step 111. Furthermore, the bitstream 900 may be a coded video sequence that may be coded according to unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700. In addition, the bitstream 900 may be employed to communicate the RPL structure 800.

[0088] The bitstream 900 includes a sequence parameter set (SPS) 910, a number of picture parameter sets (PPS) 911, a number of slice headers 915, and image data 920. The SPS 910 includes sequence data common to all pictures in a video sequence included in the bitstream 900. Such data may include picture size determination, bit depth, coding tool parameters, bit rate constraints, etc. The PPS 911 includes parameters that apply to the entire picture. Thus, each picture in the video sequence may reference the PPS 911. Note that while each picture references the PPS 911, in some examples a single PPS 911 may include data for multiple pictures. For example, multiple similar pictures may be coded according to similar parameters. In such cases, a single PPS 911 may include data for such similar pictures. The PPS 911 may indicate available coding tools, quantization parameters, offsets, etc. for slices in the corresponding picture. The slice header 915 includes parameters that are specific to each slice in the picture. Thus, there may be one slice header 915 per slice in a video sequence. The slice header 915 may include slice type information, a picture order count (POC), a reference picture list, prediction weights, tile entry points, deblocking parameters, etc. Note that in some contexts, the slice header 915 may also be referred to as a tile group header. Note that in some examples, the bitstream 900 may also include a picture header, which is a syntax structure that includes parameters that apply to all slices in a single picture. For this reason, in some contexts, the picture header and the slice header 915 may be used interchangeably. For example, some parameters may be moved between the slice header 915 and the picture header depending on whether such parameters are common to all slices in a picture.

[0089] The image data 920 includes video data coded according to inter-, intra-, and / or inter-layer prediction, as well as corresponding transformed and quantized residual data. For example, a video sequence includes a number of pictures 923. A picture 923 is an array of luma samples and / or an array of chroma samples that make up a frame or a field of a frame. A frame is a complete image intended for full or partial display to a user at a corresponding instant in the video sequence. A picture 923 may be included in a single AU 921. An AU 921 is a coding unit configured to store all coded pictures 923 having the same picture order count and, optionally, one or more headers, such as slice header 915, that include parameters representing the coding mechanism employed to code the coded picture 923. Thus, an AU 921 may include a single picture 923 for each picture level included in the bitstream 900. A picture 923 includes one or more slices 925. A slice 925 may be defined as an integer number of complete tiles or an integer number of consecutive complete CTU rows (e.g., within a tile) of a picture 923 that are exclusively contained within a single NAL unit. A slice 925 is further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a group of samples of a predefined size that may be partitioned by a coding tree. A CTB is a subset of a CTU and contains the luma or chroma components of the CTU. The CTUs / CTBs are further divided into coding blocks based on the coding tree. The coding blocks may then be encoded / decoded according to a prediction mechanism.

[0090] The bitstream 900 includes various coding tool parameters for supporting inter-layer prediction. Specifically, the bitstream includes a reference picture list structure 931, an ILRP flag 933, and an ILRP layer indicator 935, which may be substantially similar to the RPL structure 800, the ILRP flag 833, and the ILRP layer indicator 835, respectively. The reference picture list structure 931, the ILRP flag 933, and the ILRP layer indicator 935 may be coded in the SPS 910, the slice header 915, and / or the corresponding picture header. Thus, an encoder can enumerate reference pictures in entries in the reference picture list structure 931, indicate which of the reference pictures are inter-layer reference pictures in the ILRP flag 933, and indicate which layer contains the associated inter-layer reference picture in the ILRP layer indicator 935. Further, the decoder may decode the SPS 910, slice header 915, and / or corresponding picture header to obtain a reference picture list structure 931, an ILRP flag 933, and an ILRP layer indicator 935. The decoder may then determine reference pictures for the current picture from entries in the reference picture list structure 931. The decoder may also determine which of the reference pictures are inter-layer reference pictures by employing the ILRP flag 933. Further, the decoder may determine one or more levels associated with an inter-layer reference picture by employing the ILRP layer indicator 935. Thus, the bitstream 900 is configured to provide sufficient information for managing reference pictures when performing a combination of intra prediction, inter prediction, and / or inter-layer prediction to code a video sequence.

[0091] The aforementioned information will now be described in more detail below in this specification. In order to change the spatial resolution of a coded picture in the middle of a bitstream, a reference picture resampling (RPR) mechanism may be employed. This resolution change may be achieved in a current picture even when the current picture is not intra-coded. To enable this function, the current picture may refer to one or more reference pictures for inter-prediction purposes, where the reference pictures adopt a spatial resolution different from that of the current picture. Thus, the encoding and decoding of the current picture may involve resampling of such reference pictures or parts of reference pictures. This function may also be referred to as adaptive resolution change (ARC). The resampling of reference pictures may be realized either at the picture level or at the coding block level.

[0092] Some implementations may benefit from RPR. For example, video telephony and video conferencing may employ rate adaptation. The rate of the coded video may be adapted to changing network conditions. When network conditions worsen and less bandwidth is available, the encoder may adapt by encoding pictures at lower resolution pictures. As another example, a change of active speaker in a multi-party video conference may benefit from RPR. In a multi-party video conference, the active speaker may be shown at a larger video size than the video for the other participants of the conference. As the active speaker changes, the picture resolution for each participant may also be adjusted. When the active speaker changes more frequently, the RPR / ARC mechanism becomes more beneficial over time. In another example, a quick start in streaming may benefit from RPR. A streaming application may buffer up to some length of decoded pictures before starting to display. Starting the bitstream at a lower resolution may allow the application to buffer enough pictures to start displaying more quickly. Once display has begun, the resolution may then be increased. In another example, adaptive stream switching in streaming may benefit from RPR. Dynamic Adaptive Streaming over HTTP (DASH) employs a feature called @mediaStreamStructureId. This feature enables switching between different representations at open group of pictures (GOP) random access points with non-decodable leading pictures, sometimes called clean random access (CRA) pictures with associated RASL pictures in HEVC. For example, two different representations of the same video may have different bitrates and the same spatial resolution while they have the same value of @mediaStreamStructureId. In such a case, switching between the two representations at CRA pictures with associated RASL pictures may be performed.RASL pictures associated with CRA pictures can be decoded with acceptable quality, which allows seamless switching. ARC / RPR allows the @mediaStreamStructureId feature to be employed when switching between DASH representations with different spatial resolutions.

[0093] ARC / RPR may be implemented by employing layered video coding, sometimes referred to as scalable video coding and / or video coding with scalability. Scalability in video coding may be supported by using multi-layer coding techniques. A multi-layer bitstream comprises a base layer (BL) and one or more enhancement layers (EL). Scalability may include spatial scalability, quality / signal-to-noise ratio (SNR) scalability, multi-view scalability, etc. When a multi-layer coding technique is employed, a picture (or a portion of a picture) may be coded (1) without using a reference picture (intra prediction), (2) with reference to one or more reference pictures in the same layer (inter prediction), or (3) with reference to one or more reference pictures in other layers (inter-layer prediction). A reference picture used for inter-layer prediction of a current picture is called an inter-layer reference picture (ILRP).

[0094] The H.26x video coding family may provide support for scalability in separated profiles from profiles for single-layer coding. Scalable video coding (SVC) is a scalable extension of AVC / H.264 that provides support for spatial, temporal, and quality scalability. For SVC, a flag is signaled in each macroblock (MB) in a picture to indicate whether the MB is predicted using a collocated block from a lower layer. Prediction from a collocated block may include texture modes, motion vector modes, and / or coding modes.

[0095] Scalable HEVC (SHVC) is an extension of HEVC / H.265 that provides support for spatial and quality scalability. Multi-view HEVC (MV-HEVC) is an extension of HEVC / H.265 that provides support for multi-view scalability. Three-dimensional (3D) HEVC (3D-HEVC) is an extension of HEVC / H.264 that provides support for 3D video coding. Temporal scalability can be employed in a single-layer HEVC codec. Multi-layer extensions of HEVC employ a mechanism whereby decoded pictures used for inter-layer prediction are taken only from the same AU and are treated as long-term reference pictures (LTRPs). Such pictures are assigned a reference index in a reference picture list together with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the value of a reference index to refer to an inter-layer reference picture in a reference picture list.

[0096] When an ILRP has a different spatial resolution than the current picture being encoded or decoded, spatial scalability may employ resampling of reference pictures or portions of reference pictures. Reference picture resampling may be realized either at the picture level or at the coding block level.

[0097] In a video codec specification, a picture may be identified for multiple purposes. For example, a picture may be identified for use as a reference picture in inter prediction, for output from the DPB, for scaling of motion vectors, for weighted prediction, etc. In some video coding systems, a picture may be identified by a picture order count (POC). Furthermore, a picture in the DPB may be marked as being used for short-term reference, used for long-term reference, or not used for reference. A picture may no longer be used for prediction once it has been marked as not used for reference. When such a picture is no longer needed for output, the picture may be removed from the DPB.

[0098] AVC may employ short-term and long-term reference pictures. When a picture is no longer needed as a prediction reference, the reference picture may be marked as unused for reference. The transition between these three statuses (short-term, long-term, and unused for reference) is controlled by a marking process of the decoded reference picture. For reference picture marking in such a system, an implicit sliding window process and an explicit memory management control operation (MMCO) process may be employed. The sliding window process marks a short-term reference picture as unused for reference when the number of reference frames is equal to the maximum number (max_num_ref_frames) in the SPS. The short-term reference pictures are stored in a first-in-first-out manner such that the most recently decoded short-term picture is kept in the DPB. The explicit MMCO process may include multiple MMCO commands. The MMCO command may mark one or more short-term or long-term reference pictures as not used for reference, may mark all pictures as not used for reference, or may mark the current reference picture or an existing short-term reference picture as long-term and may assign a long-term picture index to the long-term reference picture. In AVC, the reference picture marking operation and the process of outputting and removing pictures from the DPB are performed after the picture is decoded.

[0099] HEVC employs reference picture sets (RPS) for reference picture marking. With RPS, a complete set of reference pictures used by the current picture or any subsequent pictures is provided for each slice. Thus, the complete set of all pictures to be kept in the DPB for use by the current or future pictures is signaled in the RPS. This differs from the AVC scheme, where only relative changes in the DPB are signaled. When the RPS maintains the exact status of reference pictures in the DPB, it does not need to store information from pictures earlier in the decoding order.

[0100] In AVC, picture marking and buffering operations (both outputting and removing the decoded picture from the DPB) may be applied after the current picture is decoded. In HEVC, the RPS is first decoded from the slice header of the current picture. Then, picture marking and buffering operations may be applied before decoding the current picture.

[0101] VVC adopts reference picture list 0 and reference picture list 1 for reference picture management. With this approach, the reference picture list for a picture is directly constructed without using the reference picture list initialization process and the reference picture list modification process. Moreover, reference picture marking is directly performed based on the two reference picture lists.

[0102] The above system has some problems: A VVC decoder should be able to derive inter-layer reference pictures to enable a multi-layer video codec with inter-layer prediction. To implement such a mechanism, the codec should adopt mechanisms for signaling the RPL, deriving the RPL, and performing reference picture marking in a multi-layer context.

[0103] This disclosure includes several techniques for reference picture management in layered video coding. This includes signaling of RPL, derivation of RPL, and reference picture marking. The description of the techniques is based on VVC, but applies to other layered video coding specifications. For example, this disclosure includes mechanisms for signaling of RPL in multi-layer video codecs with inter-layer prediction based on VVC. The following constraints may apply to the examples described below: All VCL NAL units that have the same layer ID and are associated with the same presentation time may comprise a picture. Furthermore, all VCL NAL units in a picture may have the same POC value. Such a POC value may also be referred to as the POC value of the picture.

[0104] The first example is summarized as follows: All pictures related to the same presentation time belong to one access unit. Pictures of different layers and in the same access unit have the same POC value. A first flag, such as inter_layer_ref_pics_flag, may be added to the SPS to specify whether ILRP is used for inter prediction of any coded picture in the CVS. When the first flag specifies that ILRP may be used for inter prediction of one or more coded pictures in the CVS, a second flag may be signaled for each entry in the RPL structure to specify whether the entry is an ILRP entry. When the second flag for an entry specifies that the entry is an ILRP entry, a first delta value may be signaled that specifies the difference -1 between the layer ID of the current picture and the picture referenced by the entry. In some examples, the first delta value specifies the difference -1 between the layer index of the layer containing the current picture and the layer index of the layer containing the picture referenced by the entry. At the start of decoding the current slice, the RPL may be configured by the decoder according to the RPL signaling in the bitstream. When an entry is an ILRP entry, the entry is derived to reference a picture with the same PicOrderCntVal as the current picture and with a layer ID equal to the layer ID of the current picture minus the first delta value minus 1. At the start of decoding the current picture, each ILRP is marked as used for long-term reference (if present). At the end of decoding the current picture, each ILRP is marked as used for short-term reference (if present). When decoding pictures of the same layer, a decoded picture may only be marked as not used for reference.

[0105] The method of implementing the above example is as follows. In one example, a method for decoding a video bitstream is disclosed. The bitstream comprises a plurality of layers, each layer comprising a plurality of pictures. One or more of the pictures belong to different layers and have the same presentation time, thus forming one access unit. The method comprises deriving a POC value for each picture. The POC values ​​of all pictures in one access unit are the same. Several RPLs are derived for a current slice. Each entry is associated with a flag that specifies whether the entry is an ILRP entry. The current slice is decoded based on the derived RPL. In one exemplary aspect, the bitstream includes an SPS flag that specifies whether the ILRP entry is used for inter prediction of any coded picture in the CVS. In one exemplary aspect, when an entry in the RPL is specified to be an ILRP entry, the bitstream includes a first delta value that specifies a difference between the layer ID of the current picture that includes the current slice and the picture referenced by the entry, minus one. In an example aspect, an ILRP entry references a picture that has the same PicOrderCntVal as the current picture and has a layer ID equal to the layer ID of the current picture minus the first delta value minus 1. In an example aspect, at the start of decoding of the current picture, each ILRP (if present) is marked as being used for long-term reference. At the end of decoding of the current picture, each ILRP (if present) is marked as being used for short-term reference.

[0106] A second way of implementing the above example is as follows: In one example, all pictures related to the same presentation time belong to one access unit. Each picture may be associated with an intra-layer POC value and an inter-layer POC value. Pictures in one CVS may have different inter-layer POC values. Pictures of different layers and in the same access unit may have the same intra-layer POC value but different inter-layer POC values. No picture may reference a picture of a higher layer and therefore a picture with a higher value layer ID. When decoding pictures of the same layer, the decoded picture may only be marked as not used for reference.

[0107] A third way of implementing the above example is as follows: In one example, all pictures related to the same presentation time belong to one access unit. Pictures within a CVS may have different POC values. Pictures within one CVS may be identified only by POC values. No picture may reference a picture of a higher layer, and therefore may not reference a picture with a higher value of layer ID. When decoding pictures of the same layer, the decoded picture may only be marked as not used for reference.

[0108] A fourth way of implementing the above example is as follows: Each access unit may contain only one picture. Thus, any two pictures related to the same presentation time but belonging to different layers may belong to two different access units. Pictures in one CVS may have different POC values. Pictures in one CVS may be identified only by the POC value. No picture may refer to a picture in a higher layer, and therefore may not refer to a picture with a higher value of layer ID. Regardless of whether the current picture and the reference picture belong to the same layer or not, the reference picture may be marked as not used for reference.

[0109] A first exemplary implementation of the above method is described below. An exemplary definition is as follows: An ILRP is a picture in the same access unit as the current picture, has a nuh_layer_id smaller than the nuh_layer_id of the current picture, and is marked as being used for long-term reference. A long-term reference picture (LTRP) is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture and is marked as being used for long-term reference. A reference picture is a picture that is a short-term reference picture or a long-term reference picture or an inter-layer reference picture. A reference picture contains samples that can be used for inter prediction in the decoding process of a subsequent picture in decoding order. A short-term reference picture (STRP) is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture, and is marked as being used for short-term reference.

[0110] An exemplary sequence parameter set RBSP syntax is as follows:

[0111] [Table 1]

[0112] An exemplary reference picture list structure syntax is as follows:

[0113] [Table 2]

[0114] Exemplary sequence parameter set RBSP semantics are as follows: to specify that LTRP is not used for inter prediction of any coded picture in the CVS, long_term_ref_pics_flag may be set equal to 0. to specify that LTRP may be used for inter prediction of one or more coded pictures in the CVS, long_term_ref_pics_flag may be set equal to 1. to specify that ILRP is not used for inter prediction of any coded picture in the CVS, inter_layer_ref_pics_flag may be set equal to 0. to specify that ILRP may be used for inter prediction of one or more coded pictures in the CVS, inter_layer_ref_pics_flag may be set equal to 1.

[0115] Exemplary generic slice header semantics are as follows: slice_type specifies the coding type of the slice according to the table below.

[0116] [Table 3]

[0117] When NalUnitType is a value of NalUnitType in the range IDR_W_RADL to CRA_NUT, inclusive, and the current picture is the first picture in an access unit, slice_type shall be equal to 2.

[0118] Exemplary reference picture list structure semantics are as follows: To specify that the i-th entry in the ref_pic_list_struct(listIdx,rplsIdx) syntax structure is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 1. To specify that the i-th entry in the ref_pic_list_struct(listIdx,rplsIdx) syntax structure is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 0. When not present, the value of inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is inferred to be equal to 0. To specify that the i-th entry in the ref_pic_list_struct( listIdx, rplsIdx ) syntax structure is a STRP entry, st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be set equal to 1. To specify that the i-th entry in the ref_pic_list_struct( listIdx, rplsIdx ) syntax structure is a LTRP entry, st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be set equal to 0. When inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 0 and st_ref_pic_flag[listIdx][rplsIdx][i] is not present, the value of st_ref_pic_flag[listIdx][rplsIdx][i] may be inferred to be equal to 1.

[0119] The variables NumLtrpEntries[ listIdx ][ rplsIdx ] may be derived as follows: For ( i = 0, NumLtrpEntries[ listIdx ][ rplsIdx ] = 0; i < num_ref_entries[ listIdx ][ rplsIdx ]; i++ ) if( !inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] && !st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] ) NumLtrpEntries[ listIdx ][ rplsIdx ]++

[0120] To specify that the i-th entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) has a value greater than or equal to 0, strp_entry_sign_flag[listIdx][rplsIdx][i] may be set equal to 1. To specify that the i-th entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) has a value less than 0, strp_entry_sign_flag[listIdx][rplsIdx][i] may be set equal to 0. When not present, the value of strp_entry_sign_flag[listIdx][rplsIdx][i] may be inferred to be equal to 1.

[0121] The list DeltaPocSt[listIdx][rplsIdx] may be derived as follows: for( i = 0; i < num_ref_entries[ listIdx ][ rplsIdx ]; i++ ) if( !inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] && st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] ) { (7-93) DeltaPocSt[ listIdx ][ rplsIdx ][ i ] = ( strp_entry_sign_flag[ listIdx ][ rplsIdx ][ i ] ) ? abs_delta_poc_st[ listIdx ][ rplsIdx ][ i ] : 0 - abs_delta_poc_st[ listIdx ][ rplsIdx ][ i ]

[0122] rpls_entry_layer_id_delta_minus1[listIdx][rplsIdx][i]+1 specifies the difference between the nuh_layer_id of the current picture and the picture referenced by the i-th entry. The value of rpls_entry_layer_id_delta_minus1[listIdx][rplsIdx][i] may be in the range 0 to 125, inclusive.

[0123] An exemplary decoding process for a coded picture is as follows: The decoding process operates as follows for the current picture CurrPic: NAL units are decoded. The following decoding process may use syntax from the following elements in the slice header layer or above: Variables and functions related to picture order count are derived. This may be called only for the first slice of a picture. At the start of the decoding process for each slice of a non-instantaneous decoding refresh (IDR) picture, a decoding process for reference picture list construction may be called for the derivation of reference picture list 0 (RefPicList

[0000] ) and reference picture list 1 (RefPicList

[0001] ). A decoding process for reference picture marking may be called, where reference pictures may be marked as not used for reference or used for long-term reference. This may be called only for the first slice of a picture. When the current picture is a CRA picture with NoIncorrectPicOutputFlag equal to 1 or a Gradual Random Access (GRA) picture with NoIncorrectPicOutputFlag equal to 1, a decoding process for generating unavailable reference pictures is invoked, and such a process may be invoked only for the first slice of the picture. PictureOutputFlag may be set as follows: PictureOutputFlag may be set equal to 0 if one of the following conditions is true: PictureOutputFlag may be set equal to 0 when the current picture is a RASL picture and NoIncorrectPicOutputFlag of the associated IRAP picture is equal to 1. PictureOutputFlag may be set equal to 0 when gra_enabled_flag is equal to 1 and the current picture is a GRA picture with NoIncorrectPicOutputFlag equal to 1.When gra_enabled_flag is equal to 1, the current picture is associated with a GRA picture with NoIncorrectPicOutputFlag equal to 1, and the current picture's PicOrderCntVal is less than the associated GRA picture's RpPicOrderCntVal, PictureOutputFlag may be set equal to 0. Otherwise, PictureOutputFlag is set equal to 1. After all slices of the current picture are decoded, the currently decoded picture is marked as used for short-term reference, and each ILRP entry in RefPicList

[0000] or RefPicList

[0001] is marked as used for short-term reference.

[0124] An exemplary decoding process for reference picture list construction is as follows: This process is invoked at the start of the decoding process for each slice of a non-IDR picture. A reference picture is addressed through a reference index, which is an index into a reference picture list. When decoding an intra-coded (I) slice, no reference picture list is used in decoding slice data. When decoding a unidirectional inter-predicted (P) slice, only reference picture list 0 (RefPicList

[0000] ) is used in decoding slice data. When decoding a bidirectional inter-predicted (B) slice, both reference picture list 0 and reference picture list 1 (RefPicList

[0001] ) are used in decoding slice data. At the start of the decoding process for each slice of a non-IDR picture, reference picture lists RefPicList

[0000] and RefPicList

[0001] are derived. The reference picture lists are used in marking reference pictures or in decoding slice data. For an I-slice of a non-IDR picture that is not the first slice of the picture, RefPicList

[0000] and RefPicList

[0001] may be derived for bitstream conformance checking purposes, but their derivation may not be required for decoding of the current picture or pictures that follow the current picture in decoding order. For a P-slice that is not the first slice of the picture, RefPicList

[0001] may be derived for bitstream conformance checking purposes, but may not be required for decoding of the current picture or pictures that follow the current picture in decoding order.

[0125] The reference picture lists RefPicList

[0000] and RefPicList

[0001] may be constructed as follows: for( i = 0 ; i < 2 ; i++ ) { for( j = 0, k = 0, pocBase = PicOrderCntVal; j < num_ref_entries[ i ][ RplsIdx[ i ] ]; j++) { if( !inter_layer_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { if( st_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ] = pocBase - DeltaPocSt[ i ][ RplsIdx[ i ] ][ j ] if (there is a reference picture picA in the DPB with the same nuh_layer_id as the current picture and PicOrderCntVal equal to RefPicPocList[i][j]) RefPicList[i][j] = picA else RefPicList[ i ][ j ] = "No reference pictures" (8-5) pocBase = RefPicPocList[ i ][ j ] } else { if( !delta_poc_msb_cycle_lt[ i ][ k ] ) { if(reference picA is in DPB, has the same nuh_layer_id as current picture and PicOrderCntVal & (MaxPicOrderCntLsb - 1) is equal to PocLsbLt[i][k]) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } else { if (reference picA with the same nuh_layer_id as the current picture and PicOrderCntVal equal to FullPocLt[i][k] is in the DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } k++ } } else { refPicLayerId = nuh_layer_id - rpls_entry_layer_id_delta_minus1[ i ][ RplsIdx[ i ] ][ j ] - 1 if (reference picture picA with nuh_layer_id equal to refPicLayerId and same PicOrderCntVal as current picture is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } } } }

[0126] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are referred to as active entries in RefPicList[i], and the other entries in RefPicList[i] are referred to as inactive entries in RefPicList[i]. It is possible for a particular picture to be referenced by both an entry in RefPicList

[0000] and an entry in RefPicList

[0001] . It is also possible for a particular picture to be referenced by more than one entry in RefPicList

[0000] or by more than one entry in RefPicList

[0001] . The active entries in RefPicList

[0000] and the active entries in RefPicList

[0001] collectively refer to all reference pictures that may be used for inter prediction of the current picture and one or more pictures that follow the current picture in decoding order. Inactive entries in RefPicList

[0000] and inactive entries in RefPicList

[0001] collectively refer to all reference pictures that are not used for inter-prediction of the current picture but may be used in inter-prediction for one or more pictures that follow the current picture in decoding order. There may be one or more entries in RefPicList

[0000] or RefPicList

[0001] that are not equal to any reference picture because no corresponding picture exists in the DPB. Each inactive entry in RefPicList

[0000] or RefPicList

[0000] that is not equal to any reference picture should be ignored. For each active entry in RefPicList

[0000] or RefPicList

[0001] that is not equal to any reference picture, an accidental picture loss should be inferred.

[0127] For bitstream conformance, the following constraints should be applied: For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] should not be less than NumRefIdxActive[i]. The picture referenced by each active entry in RefPicList

[0000] or RefPicList

[0001] should be present in the DPB and should have a TemporalId less than or equal to the TemporalId of the current picture. The picture referenced by each entry in RefPicList

[0000] or RefPicList

[0001] should not be the current picture. The STRP entries in RefPicList

[0000] or RefPicList

[0001] of a slice of a picture and the LTRP entries in RefPicList

[0000] or RefPicList

[0001] of the same slice or different slices of the same picture should not reference the same picture. There should be no LTRP entries in RefPicList

[0000] or RefPicList

[0001] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of a picture referenced by the entry is greater than or equal to 224. Let setOfRefPics be the set of unique pictures referenced by all entries in RefPicList

[0000] with the same nuh_layer_id as the current picture and all entries in RefPicList

[0001] with the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics should be less than or equal to sps_max_dec_pic_buffering_minus1, and setOfRefPics should be the same for all slices of a picture. The pictures referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture should be in the same access unit as the current picture.Pictures referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture should be present in the DPB and should have a nuh_layer_id smaller than the nuh_layer_id of the current picture. Each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice should be an active entry.

[0128] An exemplary decoding process for reference picture marking is as follows. This process is invoked once per picture after the decoding of the slice header and the decoding process for the reference picture list construction for the slice, but before the decoding of the slice data. This process may result in one or more reference pictures in the DPB being marked as not used for reference or used for long-term reference. A decoded picture in the DPB may be marked as not used for reference, used for short-term reference, or used for long-term reference. However, a decoded picture may only be marked as one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes another of these markings, when applicable. When a picture is referred to as being marked as being used for reference, this collectively refers to the picture being marked as being used for short-term reference or used for long-term reference, but not both. STRPs and ILRPs may be identified by their nuh_layer_id and PicOrderCntVal values. LTRPs may be identified by their nuh_layer_id value and the Log2( MaxLtPicOrderCntLsb ) least significant bits (LSBs) of their PicOrderCntVal values. If the current picture is a coded layer video sequence start (CLVSS) picture, then all reference pictures in the current DPB, if any, with the same nuh_layer_id as the current picture are marked as not used for reference. Otherwise, the following applies: For each LTRP entry in RefPicList

[0000] or RefPicList

[0001] , if the referenced picture is a STRP with the same nuh_layer_id as the current picture, then the picture is marked as used for long term reference.Each reference picture with the same nuh_layer_id as the current picture in the DPB that is not referenced by any entry in RefPicList

[0000] or RefPicList

[0001] is marked as unused for reference. For each ILRP entry in RefPicList

[0000] or RefPicList

[0001] , the referenced picture is marked as used for long-term reference.

[0129] A second exemplary implementation of the above method is described below. An exemplary definition is as follows: ILRP is a picture in the same access unit as the current picture, has a nuh_layer_id smaller than the nuh_layer_id of the current picture, and is marked as used for long-term reference. LTRP is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture and is marked as used for long-term reference. A reference picture is a picture that is a short-term reference picture or a long-term reference picture or an inter-layer reference picture. A reference picture contains samples that can be used for inter prediction in the decoding process of a subsequent picture in decoding order. STRP is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture, and is marked as used for short-term reference.

[0130] An example video parameter set syntax is as follows:

[0131] [Table 4]

[0132] Exemplary video parameter set RBSP semantics are as follows: vps_video_parameter_set_id provides an identifier for the VPS for reference by other syntax elements. vps_max_layer_id specifies the maximum allowed value of nuh_layer_id in each CVS that references the video parameter set (VPS). Once determined, the value of vps_max_layer_id is expected to remain unchanged when generating the original bitstream. This includes cases where the original bitstream is rewritten or the rewritten bitstream is further rewritten. Otherwise, picture order count values ​​may be disrupted and unexpected behavior may occur. Alternatively, the maximum allowed value of nuh_layer_id may be signaled in the SPS instead and referred to as sps_max_layer_id.

[0133] Exemplary generic slice header semantics are as follows: slice_type specifies the coding type of the slice according to the table below.

[0134] [Table 5]

[0135] When NalUnitType is a value of NalUnitType in the range IDR_W_RADL to CRA_NUT, inclusive, and the current picture is the first picture in the access unit, slice_type shall be equal to 2. slice_pic_order_cnt_lsb specifies the value of PicOrderCntVal modulo MaxPicOrderCntLsb for the current picture. The length of the slice_pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4+4 bits. The value of slice_pic_order_cnt_lsb shall be in the range 0 to MaxPicOrderCntLsb-1, inclusive. slice_poc_lsb_lt[ i ][ j ] specifies the value of PicOrderCntVal modulo MaxPicOrderCntLsb for the jth LTRP entry in the ith reference picture list. The length of the slice_poc_lsb_lt[ i ][ j ] syntax element is log2_max_pic_order_cnt_lsb_minus4+4 bits. To specify that the i-th entry in the ref_pic_list_struct( listIdx, rplsIdx ) syntax structure is a non-LTRP entry (STRP entry or ILRP entry), st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be set equal to 1. To specify that the i-th entry in the ref_pic_list_struct( listIdx, rplsIdx ) syntax structure is a LTRP entry, st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be set equal to 0. When not present, the value of st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be inferred to be equal to 1.

[0136] The variables NumLtrpEntries[ listIdx ][ rplsIdx ] may be derived as follows: for( i = 0, NumLtrpEntries[ listIdx ][ rplsIdx ] = 0; i < num_ref_entries[ listIdx ][ rplsIdx ]; i++ ) if( !st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] ) (7-87) NumLtrpEntries[ listIdx ][ rplsIdx ]++

[0137] abs_delta_poc_st[listIdx][rplsIdx][i] specifies the absolute difference between the CrossLayerPoc value of the current picture and the CrossLayerPoc value of the picture referenced by the i-th entry when the i-th entry is the first non-LTRP entry in the ref_pic_list_struct(listIdx, rplsIdx) syntax structure. abs_delta_poc_st[listIdx][rplsIdx][i] specifies the absolute difference between the CrossLayerPoc value of the picture referenced by the i-th entry and the CrossLayerPoc value of the picture referenced by the previous non-LTRP entry in the ref_pic_list_struct(listIdx,rplsIdx) syntax structure when the i-th entry is a non-LTRP entry but is not the first non-LTRP entry in the ref_pic_list_struct(listIdx,rplsIdx) syntax structure. The value of abs_delta_poc_st[listIdx][rplsIdx][i] should be in the range 0 to (2 inclusive). 15- 1)*(vps_max_layer_id+1). entry_sign_flag[listIdx][rplsIdx][i] may be set equal to 1 to specify that the i-th entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) has a value greater than or equal to 0. entry_sign_flag[listIdx][rplsIdx] may be set equal to 0 to specify that the i-th entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) has a value less than 0. When not present, the value of entry_sign_flag[i][j] is inferred to be equal to 1.

[0138] The list DeltaPoc[ listIdx ][ rplsIdx ] can be derived as follows: for( i = 0; i < num_ref_entries[ listIdx ][ rplsIdx ]; i++ ) { if( st_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] ) { (7-88) DeltaPoc[ listIdx ][ rplsIdx ][ i ] = ( entry_sign_flag[ listIdx ][ rplsIdx ][ i ]) ? abs_delta_poc_st[ listIdx ][ rplsIdx ][ i ] : 0 - abs_delta_poc_st[ listIdx ][ rplsIdx ][ i ] } }

[0139] rpls_poc_lsb_lt[listIdx][rplsIdx][i] specifies the value of PicOrderCntVal modulo MaxPicOrderCntLsb for the picture referenced by the i-th entry in the ref_pic_list_struct(listIdx,rplsIdx) syntax structure. The length of the rpls_poc_lsb_lt[listIdx][rplsIdx][i] syntax element is log2_max_pic_order_cnt_lsb_minus4+4 bits.

[0140] An exemplary decoding process for a coded picture is as follows: The decoding process operates as follows for the current picture CurrPic: The decoding of NAL units is as specified herein. The following decoding process uses syntax elements in the slice header layer or above: Variables and functions related to picture order counts are derived. Such derivation may be invoked only for the first slice of a picture. At the start of the decoding process for each slice of a non-IDR picture, a decoding process for reference picture list construction is invoked for the derivation of reference picture list 0 (RefPicList

[0000] ) and reference picture list 1 (RefPicList

[0001] ). A decoding process for reference picture marking is invoked. Reference pictures may be marked as not used for reference or as used for long-term reference. This mechanism may be invoked only for the first slice of a picture. The decoding process to generate unavailable reference pictures is invoked when the current picture is a CRA picture with NoIncorrectPicOutputFlag equal to 1 or a GRA picture with NoIncorrectPicOutputFlag equal to 1. This process may only be invoked for the first slice of a picture.

[0141] PictureOutputFlag may be set as follows: PictureOutputFlag is set equal to 0 if one of the following conditions is true: When the current picture is a RASL picture and the associated IRAP picture's NoIncorrectPicOutputFlag is equal to 1, PictureOutputFlag is set equal to 0. When gra_enabled_flag is equal to 1 and the current picture is a GRA picture with NoIncorrectPicOutputFlag equal to 1, PictureOutputFlag is set equal to 0. When gra_enabled_flag is equal to 1, the current picture is associated with a GRA picture with NoIncorrectPicOutputFlag equal to 1, and the current picture's PicOrderCntVal is less than the associated GRA picture's RpPicOrderCntVal, PictureOutputFlag is set equal to 0. Otherwise, PictureOutputFlag is set equal to 1. After all slices of the current picture have been decoded, the currently decoded picture may be marked as being used for short-term reference, and each ILRP entry in RefPicList

[0000] or RefPicList

[0001] may be marked as being used for short-term reference.

[0142] An exemplary decoding process for picture order count is as follows: The output of this process is CrossLayerPoc, which is the picture order count of the current picture. Each coded picture is associated with a picture order count variable, denoted as CrossLayerPoc. When the current picture is not a CLVSS picture, the variables prevPicOrderCntLsb and prevPicOrderCntMsb may be derived as follows: Let prevTid0Pic be the previous picture in decoding order that has nuh_layer_id equal to the nuh_layer_id of the current picture and TemporalId equal to 0, and that is not a RASL or RADL picture. The variable prevPicOrderCntLsb may be set equal to the slice_pic_order_cnt_lsb of prevTid0Pic. The variable prevPicOrderCntMsb may be set equal to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current picture may be derived as follows: If the current picture is a CLVSS picture, then PicOrderCntMsb may be set equal to 0. Otherwise, PicOrderCntMsb may be derived as follows: if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( ( prevPicOrderCntLsb - slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if( (slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ( ( slice_pic_order_cnt_lsb - prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb - MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb

[0143] PicOrderCntVal can be derived as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb For CLVSS pictures, from the time PicOrderCntMsb is set equal to 0, all CLVSS pictures may have PicOrderCntVal set equal to slice_pic_order_cnt_lsb. The value of PicOrderCntVal shall be −2 inclusive. 31 ~2 31 The PicOrderCntVal value for any two coded pictures with the same value of nuh_layer_id in a CVS may be in the range -1 to -1. All pictures in any particular access unit may have the same value of PicOrderCntVal.

[0144] The variable CrossLayerPoc may be derived as follows: CrossLayerPoc = PicOrderCntVal * ( vps_max_layer_id + 1 ) + nuh_layer_id The function PicOrderCnt(picX) may be specified as follows: PicOrderCnt( picX ) = PicOrderCntVal of picture picX The function DiffPicOrderCnt(picA, picB) may be specified as follows: DiffPicOrderCnt( picA, picB ) = PicOrderCnt( picA ) -PicOrderCnt( picB ) The bit stream is -2 inclusive. 15 ~2 15 It may not include data that would result in a value of DiffPicOrderCnt(picA, picB) used in the decoding process that is not within the range of -1. Let the current picture be X and two other pictures in the same CVS be Y and Z. When DiffPicOrderCnt(X,Y) and DiffPicOrderCnt(X,Z) are both positive or both negative, Y and Z may be considered to be in the same output order direction from X.

[0145] An exemplary decoding process for reference picture list construction is as follows: This process may be invoked at the start of the decoding process for each slice of a non-IDR picture. A reference picture is addressed through a reference index, which is an index into a reference picture list. When decoding an I slice, no reference picture list is used in decoding slice data. When decoding a P slice, only reference picture list 0 (RefPicList

[0000] ) is used in decoding slice data. When decoding a B slice, both reference picture list 0 and reference picture list 1 (RefPicList

[0001] ) are used in decoding slice data. At the start of the decoding process for each slice of a non-IDR picture, reference picture lists RefPicList

[0000] and RefPicList

[0001] are derived. The reference picture lists may be used in marking reference pictures or in decoding slice data. For an I-slice of a non-IDR picture, where it is not the first slice of the picture, RefPicList

[0000] and RefPicList

[0001] may be derived for bitstream conformance checking purposes, but their derivation may not be mandatory for decoding of the current picture or pictures that follow the current picture in decoding order. For a P-slice, where it is not the first slice of the picture, RefPicList

[0001] may be derived for bitstream conformance checking purposes, but their derivation may not be mandatory for decoding of the current picture or pictures that follow the current picture in decoding order. The reference picture lists RefPicList

[0000] and RefPicList

[0001] may be constructed as follows: for( i = 0 ; i < 2 ; i++ ) { for( j = 0, k = 0, pocBase = CrossLayerPoc; j < num_ref_entries[ i ][ RplsIdx[ i ] ]; j++) { if( st_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ] = pocBase - DeltaPoc[ i ][ RplsIdx[ i ] ][ j ] if(reference picture picA with CrossLayerPoc equal to RefPicPocList[i][j] is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" (8-5) pocBase = RefPicPocList[ i ][ j ] } else { if( !delta_poc_msb_cycle_lt[ i ][ k ] ) { if(reference picA is in DPB, has the same nuh_layer_id as current picture and PicOrderCntVal & (MaxPicOrderCntLsb - 1) is equal to PocLsbLt[i][k]) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } else { if (reference picA with the same nuh_layer_id as the current picture and PicOrderCntVal equal to FullPocLt[i][RplsIdx[i]][k] is in the DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } k++ } } }

[0146] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are referred to as active entries in RefPicList[i], and the other entries in RefPicList[i] are referred to as inactive entries in RefPicList[i]. An entry in a reference picture list may be referred to as an ILRP entry when the picture referenced by the entry has a different value of nuh_layer_id than the current picture. It is possible for a particular picture to be referenced by both an entry in RefPicList

[0000] and an entry in RefPicList

[0001] . It is also possible for a particular picture to be referenced by more than one entry in RefPicList

[0000] or by more than one entry in RefPicList

[0001] . The active entries in RefPicList

[0000] and the active entries in RefPicList

[0001] collectively refer to all reference pictures that may be used for inter-prediction of the current picture and one or more pictures that follow the current picture in decoding order. The inactive entries in RefPicList

[0000] and the inactive entries in RefPicList

[0001] collectively refer to all reference pictures that are not used for inter-prediction of the current picture but may be used in inter-prediction for one or more pictures that follow the current picture in decoding order. There may be one or more entries in RefPicList

[0000] or RefPicList

[0001] that are not equal to any reference picture because the corresponding picture is not present in the DPB. Each inactive entry in RefPicList

[0000] or RefPicList

[0000] that is not equal to any reference picture may be ignored. For each active entry in RefPicList

[0000] or RefPicList

[0001] that is not equal to any reference picture, an accidental picture loss may be inferred.

[0147] Bitstream adaptation may require that the following constraints apply: For each i equal to 0 or 1, num_ref_entries[ i ][RplsIdx[ i ] ] may not be less than NumRefIdxActive[ i ]. The pictures referenced by each active entry in RefPicList

[0000] or RefPicList

[0001] may be present in the DPB and may have a TemporalId less than or equal to the TemporalId of the current picture. The pictures referenced by each entry in RefPicList

[0000] or RefPicList

[0001] may not be the current picture. The STRP entries in RefPicList

[0000] or RefPicList

[0001] of a slice of a picture and the LTRP entries in RefPicList

[0000] or RefPicList

[0001] of the same slice or different slices of the same picture may not reference the same picture. There may be no LTRP entries in RefPicList

[0000] or RefPicList

[0001] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referenced by the entry is 224 or more. Let setOfRefPics be the set of unique pictures referenced by all entries in RefPicList

[0000] with the same nuh_layer_id as the current picture and all entries in RefPicList

[0001] with the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics may be less than or equal to sps_max_dec_pic_buffering_minus1, and setOfRefPics may be the same for all slices of a picture. The pictures referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture may be present in the DPB and may be in the same access unit as the current picture.A picture referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture may have a nuh_layer_id that is smaller than the nuh_layer_id of the current picture. Each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice may be an active entry.

[0148] An exemplary decoding process for reference picture marking is as follows: This process may be invoked once per picture. This may occur after the decoding of the slice header and the decoding process for the reference picture list construction for the slice, but before the decoding of the slice data. This process may result in one or more reference pictures in the DPB being marked as not used for reference or used for long-term reference. A decoded picture in the DPB may be marked as not used for reference, used for short-term reference, or used for long-term reference. A decoded picture in the DPB may be marked as only one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes another of these markings, when applicable. When a picture is referred to as being marked as being used for reference, this collectively refers to the picture being marked as being used for short-term reference or used for long-term reference (but not both). STRPs are identified by their nuh_layer_id and PicOrderCntVal values. LTRPs are identified by their nuh_layer_id value and the Log2( MaxLtPicOrderCntLsb ) LSBs of their PicOrderCntVal values. If the current picture is a CLVSS picture, all reference pictures in the current DPB (if any) that have the same nuh_layer_id as the current picture may be marked as unused for reference. Otherwise the following applies: For each LTRP entry in RefPicList

[0000] or RefPicList

[0001] , when the referenced picture is a STRP with the same nuh_layer_id as the current picture, the referenced picture is marked as used for long term reference.Each reference picture in the DPB that is not referenced by any entry in RefPicList

[0000] or RefPicList

[0001] and has the same nuh_layer_id as the current picture may be marked as unused for reference. For each ILRP entry in RefPicList

[0000] or RefPicList

[0001] , the referenced picture may be marked as used for long-term reference.

[0149] A third exemplary implementation of the above method is described below. An example of the definition is as follows: ILRP is a picture in the same access unit as the current picture, has a nuh_layer_id smaller than the nuh_layer_id of the current picture, and is marked as being used for long-term reference. LTRP is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture and is marked as being used for long-term reference. Reference picture is a picture that is a short-term reference picture or a long-term reference picture or an inter-layer reference picture. Reference picture includes samples that can be used for inter prediction in the decoding process of a subsequent picture in decoding order. STRP is a picture that has a nuh_layer_id equal to the nuh_layer_id of the current picture, and is marked as being used for short-term reference.

[0150] Exemplary generic slice header semantics are as follows: slice_type may specify the coding type of the slice according to the table below.

[0151] [Table 6]

[0152] When NalUnitType is a value of NalUnitType in the range of IDR_W_RADL to CRA_NUT, inclusive, and the current picture is the first picture in an access unit, Slice_type may be set equal to 2.

[0153] An exemplary decoding process for a coded picture is as follows: The decoding process operates as follows for the current picture CurrPic: The decoding of NAL units is as specified herein. The following decoding process uses syntax elements in the slice header layer or above: Variables and functions related to picture order counts are derived. Such derivation may be invoked only for the first slice of a picture. At the start of the decoding process for each slice of a non-IDR picture, a decoding process for reference picture list construction is invoked for the derivation of reference picture list 0 (RefPicList

[0000] ) and reference picture list 1 (RefPicList

[0001] ). A decoding process for reference picture marking is invoked. Reference pictures may be marked as not used for reference or as used for long-term reference. This mechanism may be invoked only for the first slice of a picture. The decoding process to generate unavailable reference pictures is invoked when the current picture is a CRA picture with NoIncorrectPicOutputFlag equal to 1 or a GRA picture with NoIncorrectPicOutputFlag equal to 1. This process may only be invoked for the first slice of a picture.

[0154] PictureOutputFlag may be set as follows: PictureOutputFlag is set equal to 0 if one of the following conditions is true: When the current picture is a RASL picture and the associated IRAP picture's NoIncorrectPicOutputFlag is equal to 1, PictureOutputFlag is set equal to 0. When gra_enabled_flag is equal to 1 and the current picture is a GRA picture with NoIncorrectPicOutputFlag equal to 1, PictureOutputFlag is set equal to 0. When gra_enabled_flag is equal to 1, the current picture is associated with a GRA picture with NoIncorrectPicOutputFlag equal to 1, and the current picture's PicOrderCntVal is less than the associated GRA picture's RpPicOrderCntVal, PictureOutputFlag is set equal to 0. Otherwise, PictureOutputFlag is set equal to 1. After all slices of the current picture have been decoded, the currently decoded picture may be marked as being used for short-term reference, and each ILRP entry in RefPicList

[0000] or RefPicList

[0001] may be marked as being used for short-term reference.

[0155] An exemplary decoding process for picture order count is as follows: The output of this process is PicOrderCntVal, which is the picture order count of the current picture. Each coded picture may be associated with a picture order count variable, denoted as PicOrderCntVal. When the current picture is not a CLVSS picture that is the first picture in the access unit in decoding order, the variables prevPicOrderCntLsb and prevPicOrderCntMsb may be derived as follows: Let prevTid0Pic be the previous picture in decoding order that has a nuh_layer_id less than or equal to the nuh_layer_id of the current picture and a TemporalId equal to 0, and is not a RASL or RADL picture. The variable prevPicOrderCntLsb may be set equal to the slice_pic_order_cnt_lsb of prevTid0Pic. The variable prevPicOrderCntMsb may be set equal to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current picture may be derived as follows: If the current picture is a CLVSS picture that is the first picture in the access unit in decoding order, then PicOrderCntMsb may be set equal to 0. Otherwise, PicOrderCntMsb is derived as follows: if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( ( prevPicOrderCntLsb - slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if( (slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ( ( slice_pic_order_cnt_lsb - prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb - MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb

[0156] PicOrderCntVal may be derived as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb Since PicOrderCntMsb is set equal to 0, each CLVSS picture that is the first picture in an access unit in decoding order may have PicOrderCntVal set equal to slice_pic_order_cnt_lsb. The value of PicOrderCntVal may range from -2, inclusive. 31 ~2 31 The PicOrderCntVal values ​​for any two coded pictures in a CVS may be in the range -1 to -1. The PicOrderCntVal values ​​for any two coded pictures in a CVS may not be the same. The function PicOrderCnt(picX) may be specified as follows: PicOrderCnt( picX ) = PicOrderCntVal of picture picX The function DiffPicOrderCnt(picA, picB) may be specified as follows: DiffPicOrderCnt( picA, picB ) = PicOrderCnt( picA ) -PicOrderCnt( picB ) The bit stream is -2 inclusive. 15 ~2 15It may not include data that would result in a value of DiffPicOrderCnt(picA, picB) used in the decoding process that is not within the range of -1. Let the current picture be X and two other pictures in the same CVS be Y and Z. When DiffPicOrderCnt(X,Y) and DiffPicOrderCnt(X,Z) are both positive or both negative, Y and Z may be considered to be in the same output order direction from X.

[0157] An exemplary decoding process for reference picture list construction is as follows: This process may be invoked at the start of the decoding process for each slice of a non-IDR picture. A reference picture is addressed through a reference index, which is an index into a reference picture list. When decoding an I slice, no reference picture list is used in decoding slice data. When decoding a P slice, only reference picture list 0 (RefPicList

[0000] ) is used in decoding slice data. When decoding a B slice, both reference picture list 0 and reference picture list 1 (RefPicList

[0001] ) are used in decoding slice data. At the start of the decoding process for each slice of a non-IDR picture, reference picture lists RefPicList

[0000] and RefPicList

[0001] are derived. The reference picture lists are used in marking reference pictures or in decoding slice data. For an I slice of a non-IDR picture, if it is not the first slice of the picture, RefPicList

[0000] and RefPicList

[0001] may be derived for bitstream conformance checking purposes. Their derivation is not required for decoding of the current picture or pictures that follow the current picture in decoding order. For a P slice, if it is not the first slice of the picture, RefPicList

[0001] may be derived for bitstream conformance checking purposes. However, such derivation is not required for decoding of the current picture or pictures that follow the current picture in decoding order. The reference picture lists RefPicList

[0000] and RefPicList

[0001] may be constructed as follows: for( i = 0 ; i < 2 ; i++ ) { for( j = 0, k = 0, pocBase = PicOrderCntVal; j < num_ref_entries[ i ][ RplsIdx[ i ] ]; j++) { if( st_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ] = pocBase - DeltaPocSt[ i ][ RplsIdx[ i ] ][ j ] if(reference picture picA with PicOrderCntVal equal to RefPicPocList[i][j] is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" (8-5) pocBase = RefPicPocList[ i ][ j ] } else { if( !delta_poc_msb_cycle_lt[ i ][ k ] ) { if(PicOrderCntVal & ( MaxPicOrderCntLsb - 1 ) equals PocLsbLt[ i ][ k ]. Reference picA is in the DPB. RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } else { if(Reference picA with PicOrderCntVal equal to FullPocLt[i][k] is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } k++ } } }

[0158] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are referred to as active entries in RefPicList[i], and the other entries in RefPicList[i] are referred to as inactive entries in RefPicList[i]. An entry in a reference picture list may be referred to as an ILRP entry when the picture referenced by the entry has a different value of nuh_layer_id than the current picture. It is possible for a particular picture to be referenced by both an entry in RefPicList

[0000] and an entry in RefPicList

[0001] . It is also possible for a particular picture to be referenced by more than one entry in RefPicList

[0000] or by more than one entry in RefPicList

[0001] . The active entries in RefPicList

[0000] and the active entries in RefPicList

[0001] may collectively refer to all reference pictures that may be used for inter-prediction of the current picture and one or more pictures that follow the current picture in decoding order. The inactive entries in RefPicList

[0000] and the inactive entries in RefPicList

[0001] may collectively refer to all reference pictures that are not used for inter-prediction of the current picture but may be used in inter-prediction for one or more pictures that follow the current picture in decoding order. There may be one or more entries in RefPicList

[0000] or RefPicList

[0001] that are not equal to any reference picture because the corresponding picture is not present in the DPB. Each inactive entry in RefPicList

[0000] or RefPicList

[0000] that is not equal to any reference picture may be ignored. For each active entry in RefPicList

[0000] or RefPicList

[0001] that is not equal to any reference picture, an accidental picture loss may be inferred.

[0159] The following constraints may apply to bitstream adaptation: For each i equal to 0 or 1, num_ref_entries[ i ][RplsIdx[ i ] ] may not be smaller than NumRefIdxActive[ i ]. The pictures referenced by each active entry in RefPicList

[0000] or RefPicList

[0001] may be present in the DPB and may have a TemporalId less than or equal to the TemporalId of the current picture. The pictures referenced by each entry in RefPicList

[0000] or RefPicList

[0001] may not be the current picture. The STRP entries in RefPicList

[0000] or RefPicList

[0001] of a slice of a picture and the LTRP entries in RefPicList

[0000] or RefPicList

[0001] of the same slice or different slices of the same picture may not reference the same picture. There may be no LTRP entries in RefPicList

[0000] or RefPicList

[0001] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referenced by the entry is greater than or equal to 224. Let setOfRefPics be the set of unique pictures referenced by all entries in RefPicList

[0000] that have the same nuh_layer_id as the current picture and all entries in RefPicList

[0001] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics may be less than or equal to sps_max_dec_pic_buffering_minus1, and setOfRefPics may be the same for all slices of a picture. The pictures referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture may be in the same access unit as the current picture.A picture referenced by each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice of the current picture may be present in the DPB and may have a nuh_layer_id that is smaller than the nuh_layer_id of the current picture. Each ILRP entry in RefPicList

[0000] or RefPicList

[0001] of a slice may be an active entry.

[0160] An exemplary decoding process for reference picture marking is as follows. This process may be invoked once per picture after the decoding of the slice header and the decoding process for the reference picture list construction for the slice. The process may also be invoked before the decoding of the slice data. This process may result in one or more reference pictures in the DPB being marked as not used for reference or used for long-term reference. A decoded picture in the DPB may be marked as not used for reference, used for short-term reference, or used for long-term reference. A decoded picture in the DPB may be marked as only one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes another of these markings, when applicable. When a picture is referred to as being marked as being used for reference, this collectively refers to the picture being marked as being used for short-term reference or used for long-term reference (but not both). STRPs and ILRPs may be identified by their nuh_layer_id and PicOrderCntVal values. LTRPs may be identified by their nuh_layer_id value and the Log2( MaxLtPicOrderCntLsb ) LSBs of their PicOrderCntVal values. If the current picture is a CLVSS picture, all reference pictures in the current DPB (if any) that have the same nuh_layer_id as the current picture may be marked as not used for reference. Otherwise, the following applies: For each LTRP entry in RefPicList

[0000] or RefPicList

[0001] , when the referenced picture is a STRP with the same nuh_layer_id as the current picture, the picture is marked as used for long-term reference.Each reference picture with the same nuh_layer_id as the current picture in the DPB that is not referenced by any entry in RefPicList

[0000] or RefPicList

[0001] may be marked as unused for reference. For each ILRP entry in RefPicList

[0000] or RefPicList

[0001] , the referenced picture may be marked as used for long-term reference.

[0161] A fourth exemplary implementation of the above method is described below. An exemplary decoding process for picture order count is as follows. The output of this process is PicOrderCntVal, which is the picture order count of the current picture. Each coded picture is associated with a picture order count variable, denoted as PicOrderCntVal. When the current picture is not a CLVSS picture, the variables prevPicOrderCntLsb and prevPicOrderCntMsb may be derived as follows: Let prevTid0Pic be the previous picture in decoding order that has a nuh_layer_id less than or equal to the nuh_layer_id of the current picture and a TemporalId equal to 0, and is not a RASL or RADL picture. The variable prevPicOrderCntLsb may be set equal to the slice_pic_order_cnt_lsb of prevTid0Pic. The variable prevPicOrderCntMsb may be set equal to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current picture may be derived as follows: If the current picture is a CLVSS picture, then PicOrderCntMsb may be set equal to 0. Otherwise, PicOrderCntMsb may be derived as follows: if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( ( prevPicOrderCntLsb - slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if( (slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ( ( slice_pic_order_cnt_lsb - prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb - MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb

[0162] PicOrderCntVal may be derived as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb For CLVSS pictures, PicOrderCntMsb is set equal to 0, and then all CLVSS pictures may have PicOrderCntVal equal to slice_pic_order_cnt_lsb. The value of PicOrderCntVal shall be -2 inclusive. 31 ~2 31 The PicOrderCntVal values ​​for any two coded pictures in a CVS may be in the range -1 to -1. The PicOrderCntVal values ​​for any two coded pictures in a CVS may not be the same. The function PicOrderCnt(picX) may be specified as follows: PicOrderCnt( picX ) = PicOrderCntVal of picture picX The function DiffPicOrderCnt(picA, picB) may be specified as follows: DiffPicOrderCnt( picA, picB ) = PicOrderCnt( picA ) -PicOrderCnt( picB ) The bit stream is -2 inclusive.15 ~2 15 It may not include data that would result in a value of DiffPicOrderCnt(picA, picB) used in the decoding process that is not within the range -1. Let the current picture be X and two other pictures in the same CVS be Y and Z. When DiffPicOrderCnt(X, Y) and DiffPicOrderCnt(X, Z) are both positive or both negative, Y and Z are considered to be in the same output order direction from X.

[0163] An exemplary decoding process for reference picture list construction is as follows: This process may be invoked at the start of the decoding process for each slice of a non-IDR picture. A reference picture is addressed through a reference index, which is an index into a reference picture list. When decoding an I slice, no reference picture list is used in decoding slice data. When decoding a P slice, only reference picture list 0 (RefPicList

[0000] ) is used in decoding slice data. When decoding a B slice, both reference picture list 0 and reference picture list 1 (RefPicList

[0001] ) are used in decoding slice data. At the start of the decoding process for each slice of a non-IDR picture, reference picture lists RefPicList

[0000] and RefPicList

[0001] may be derived. The reference picture lists may be used in marking reference pictures or in decoding slice data. For an I slice of a non-IDR picture that is not the first slice of the picture, RefPicList

[0000] and RefPicList

[0001] may be derived for bitstream conformance checking purposes. Such derivation may not be required for decoding of the current picture or pictures that follow the current picture in decoding order. For a P slice that is not the first slice of a picture, RefPicList

[0001] may be derived for bitstream conformance checking purposes. However, such derivation may not be required for decoding of the current picture or pictures that follow the current picture in decoding order.

[0164] The reference picture lists RefPicList

[0000] and RefPicList

[0001] may be constructed as follows: for( i = 0 ; i < 2 ; i++ ) { for( j = 0, k = 0, pocBase = PicOrderCntVal; j < num_ref_entries[ i ][ RplsIdx[ i ] ]; j++) { if( st_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ] = pocBase - DeltaPocSt[ i ][ RplsIdx[ i ] ][ j ] if(reference picture picA with PicOrderCntVal equal to RefPicPocList[i][j] is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" (8-5) pocBase = RefPicPocList[ i ][ j ] } else { if( !delta_poc_msb_cycle_lt[ i ][ k ] ) { if(PicOrderCntVal & ( MaxPicOrderCntLsb - 1 ) equals PocLsbLt[ i ][ k ]. Reference picA is in the DPB. RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } else { if(Reference picA with PicOrderCntVal equal to FullPocLt[i][k] is in DPB) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = "No reference pictures" } k++ } } }

[0165] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are referred to as active entries in RefPicList[i], and the other entries in RefPicList[i] are referred to as inactive entries in RefPicList[i]. It is possible for a particular picture to be referenced by both an entry in RefPicList

[0000] and an entry in RefPicList

[0001] . It is also possible for a particular picture to be referenced by more than one entry in RefPicList

[0000] or by more than one entry in RefPicList

[0001] . The active entries in RefPicList

[0000] and the active entries in RefPicList

[0001] may collectively refer to all reference pictures that may be used for inter prediction of the current picture and one or more pictures that follow the current picture in decoding order. Inactive entries in RefPicList

[0000] and inactive entries in RefPicList

[0001] may collectively refer to all reference pictures that are not used for inter-prediction of the current picture but may be used in inter-prediction for one or more pictures that follow the current picture in decoding order. There may be one or more entries in RefPicList

[0000] or RefPicList

[0001] that are not equal to any reference picture because no corresponding picture exists in the DPB. Each inactive entry in RefPicList

[0000] or RefPicList

[0000] that is not equal to any reference picture may be ignored. For each active entry in RefPicList

[0000] or RefPicList

[0001] that is not equal to any reference picture, an accidental picture loss may be inferred.

[0166] Bitstream adaptation may require that the following constraints apply: For each i equal to 0 or 1, num_ref_entries[ i ][RplsIdx[ i ] ] may not be less than NumRefIdxActive[ i ]. The pictures referenced by each active entry in RefPicList

[0000] or RefPicList

[0001] may be present in the DPB and may have a TemporalId less than or equal to the TemporalId of the current picture. The pictures referenced by each entry in RefPicList

[0000] or RefPicList

[0001] may not be the current picture. The STRP entries in RefPicList

[0000] or RefPicList

[0001] of a slice of a picture and the LTRP entries in RefPicList

[0000] or RefPicList

[0001] of the same slice or different slices of the same picture may not reference the same picture. There may be no LTRP entries in RefPicList

[0000] or RefPicList

[0001] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referenced by the entry is greater than or equal to 224. Let setOfRefPics be the set of unique pictures referenced by all entries in RefPicList

[0000] and all entries in RefPicList

[0001] . The number of pictures in setOfRefPics may be less than or equal to sps_max_dec_pic_buffering_minus1, and setOfRefPics may be the same for all slices of a picture. Pictures referenced by each active entry in RefPicList

[0000] or RefPicList

[0001] may be present in the DPB and may have a nuh_layer_id less than or equal to the nuh_layer_id of the current picture.

[0167] An exemplary decoding process for reference picture marking is as follows. This process may be invoked once per picture after the decoding of the slice header and the decoding process for the reference picture list construction for the slice. The process may be invoked before the decoding of the slice data. This process may result in one or more reference pictures in the DPB being marked as not used for reference or used for long-term reference. A decoded picture in the DPB may be marked as not used for reference, used for short-term reference, or used for long-term reference. A decoded picture in the DPB may be marked as only one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes another of these markings, when applicable. When a picture is referred to as being marked as being used for reference, this collectively refers to the picture being marked as being used for short-term reference or used for long-term reference (but not both). STRPs may be identified by their PicOrderCntVal values. LTRPs may be identified by the Log2(MaxLtPicOrderCntLsb) LSBs of their PicOrderCntVal values. If the current picture is a CLVSS picture, then all reference pictures in the current DPB (if any) may be marked as unused for reference. Otherwise, the following applies: For each LTRP entry in RefPicList

[0000] or RefPicList

[0001] , when the referenced picture is a STRP, the referenced picture is marked as used for long-term reference. Each reference picture in the DPB that is not referenced by any entry in RefPicList

[0000] or RefPicList

[0001] is marked as unused for reference.

[0168] FIG. 10 is a schematic diagram of an exemplary video coding device 1000. The video coding device 1000 is suitable for implementing the disclosed examples / embodiments as described herein. The video coding device 1000 comprises a downstream port 1020, an upstream port 1050, and / or a transceiver unit (Tx / Rx) 1010 including a transmitter and / or a receiver for communicating data upstream and / or downstream over a network. The video coding device 1000 also includes a processor 1030 including a logic unit and / or a central processing unit (CPU) for processing data, and a memory 1032 for storing data. The video coding device 1000 may also comprise electrical components, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 1050 and / or the downstream port 1020 for communication of data over an electrical communication network, an optical communication network, or a wireless communication network. Video coding device 1000 may also include input and / or output (I / O) devices 1060 for communicating data to and from a user. I / O devices 1060 may include output devices, such as a display for displaying video data, speakers for outputting audio data, etc. I / O devices 1060 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0169] The processor 1030 is implemented by hardware and software. The processor 1030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1030 is in communication with downstream ports 1020, Tx / Rx 1010, upstream ports 1050, and memory 1032. The processor 1030 comprises a coding module 1014. The coding module 1014 implements the disclosed embodiments described herein, such as methods 100, 1100, and / or 1200, which may employ a bitstream 900 including pictures that may be coded according to the RPL structure 800 and unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700. The coding module 1014 may also implement any other methods / mechanisms described herein. Furthermore, the coding module 1014 may implement the codec system 200, the encoder 300, and / or the decoder 400. For example, the coding module 1014 may be employed to code an ILRP flag and / or an ILRP layer indicator in a reference picture structure to manage reference pictures to support inter-layer prediction as described above. Thus, the coding module 1014 allows the video coding device 1000 to provide additional functionality and / or coding efficiency when coding video data. Thus, the coding module 1014 improves the functionality of the video coding device 1000 and addresses problems specific to video coding techniques. Furthermore, the coding module 1014 affects the transformation of the video coding device 1000 into different states. Alternatively, the coding module 1014 may be implemented as instructions stored in the memory 1032 (e.g., as a computer program product stored on a non-transitory medium) and executed by the processor 1030.

[0170] The memory 1032 comprises one or more memory types such as a disk, a tape drive, a solid state drive, a read only memory (ROM), a random access memory (RAM), a flash memory, a ternary content addressable memory (TCAM), a static random access memory (SRAM), etc. The memory 1032 may be used as an overflow data storage device for storing such programs when such programs are selected for execution and for storing instructions and data read during program execution.

[0171] 11 is a flowchart of an example method 1100 for encoding a video sequence according to inter-layer prediction into a bitstream, such as bitstream 900. Method 1100 may be employed by an encoder, such as codec system 200, encoder 300, and / or video coding device 1000, when performing method 100 for encoding pictures according to unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700 by employing RPL structure 800.

[0172] The method 1100 may start when an encoder receives a video sequence including a plurality of pictures and determines, for example, based on a user input, that the video sequence should be encoded into a bitstream. In step 1101, the encoder encodes a current picture into the bitstream. For example, the current picture may be encoded according to inter-layer prediction based on an inter-layer reference picture. For example, such inter-layer prediction may be performed according to inter-layer prediction 721. As described above, the inter-layer reference picture may be in the same AU as the current picture. Thus, the inter-layer reference picture may include the same POC as the current picture. Furthermore, the inter-layer reference picture is located in a different layer than the current picture. For example, the inter-layer reference picture may be associated with a layer lower than the layer of the current picture. Thus, the inter-layer reference picture may be associated with a layer ID lower than the layer ID of the current picture.

[0173] In step 1103, the encoder may encode a reference picture list structure, such as the RPL structure 800, into the bitstream. The reference picture list structure comprises multiple entries for multiple reference pictures. Such entries include an entry associated with the current picture. The entry indicates an inter-layer reference picture. For example, the reference picture list structure may be denoted as ref_pic_list_struct( listIdx, rplsIdx ), where listIdx identifies a reference picture list, rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure that returns an entry based on listIdx and rplsIdx.

[0174] In step 1105, the encoder may code an inter-layer reference picture flag into the bitstream. The inter-layer reference picture flag indicates that an entry related to the current picture is an ILRP entry. For example, the inter-layer reference picture flag may be indicated as inter_layer_ref_pic_flag[listIdx][rplsIdx][i]. Specifically, when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 1. Furthermore, when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 0. For example, ref_pic_list_struct(listIdx,rplsIdx) and inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be coded into the bitstream in the SPS. In another example, ref_pic_list_struct(listIdx,rplsIdx) and inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be coded into the bitstream in a header associated with the current picture, such as a slice header and / or a picture header.

[0175] In step 1107, the encoder may encode an ILRP layer indicator into the bitstream. The ILRP layer indicator indicates a layer of an inter-layer reference picture. For example, the ILRP layer indicator may indicate one or more layers for each entry of a reference picture list structure associated with an inter-layer reference picture as indicated by an inter-layer reference picture flag.

[0176] In step 1109, the encoder may store the bitstream for communication towards a decoder, eg, for communication upon request.

[0177] 12 is a flowchart of an example method for decoding a video sequence from a bitstream such as bitstream 900 when employing inter-layer prediction. Method 1200 may be employed by a decoder, such as codec system 200, decoder 400, and / or video coding device 1000, when performing method 100 for decoding pictures according to unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700 by employing RPL structure 800.

[0178] Method 1200 may begin when a decoder begins receiving a bitstream of coded data representing a video sequence, for example as a result of method 1100. At step 1201, the decoder receives the bitstream. The bitstream may comprise a current picture, a reference picture list structure, an inter-layer reference picture flag, and / or an ILRP layer indicator. For example, the reference picture list structure may comprise an inter-layer reference picture flag and / or an ILRP layer indicator.

[0179] In step 1203, the decoder may determine that an entry in the reference picture list structure associated with the current picture is an ILRP entry based on the inter-layer reference picture flag. For example, the reference picture list structure comprises multiple entries for multiple reference pictures. Such entries include an entry associated with the current picture. The entry indicates the reference picture. For example, the reference picture list structure may be denoted as ref_pic_list_struct( listIdx, rplsIdx ), where listIdx identifies a reference picture list, rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure that returns an entry based on listIdx and rplsIdx. Furthermore, the inter-layer reference picture flag indicates whether the entry associated with the current picture is an ILRP entry. For example, the inter-layer reference picture flag may be denoted as inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ]. Specifically, when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 1. Furthermore, when the i-th entry in ref_pic_list_struct(listIdx,rplsIdx) is not an ILRP entry, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be set equal to 0. For example, ref_pic_list_struct(listIdx,rplsIdx) and inter_layer_ref_pic_flag[listIdx][rplsIdx][i] may be coded into the bitstream in the SPS.In another example, ref_pic_list_struct( listIdx, rplsIdx ) and inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ] may be coded in the bitstream in a header associated with the current picture, such as a slice header and / or a picture header.

[0180] In step 1205, the decoder may determine a layer of the inter-layer reference picture based on the ILRP layer indicator when the entry is an ILRP entry. For example, the ILRP layer indicator may indicate one or more layers for each entry of the reference picture list structure associated with the inter-layer reference picture as indicated by the inter-layer reference picture flag. For example, the inter-layer reference picture may be included in the same AU as the current picture. Thus, the inter-layer reference picture may include the same POC as the current picture. Furthermore, the inter-layer reference picture is located in a different layer than the current picture. For example, the inter-layer reference picture may be associated with a layer lower than the layer of the current picture. Thus, the inter-layer reference picture may be associated with a layer ID lower than the layer ID of the current picture.

[0181] In step 1207, when the entry is an ILRP entry, the decoder may decode the current picture according to inter-layer prediction based on the inter-layer reference picture indicated by the entry in the reference picture list structure at the level indicated by the ILRP layer indicator.

[0182] In step 1209, the decoder may decode the current picture according to intra-layer prediction based on the intra-layer reference picture indicated by the entry in the reference picture list structure when the entry is not an ILRP entry. The intra-layer reference picture is simply referred to as a reference picture in a single layer context. Furthermore, the intra-layer prediction may include intra-prediction and / or single layer inter-prediction depending on the context. When a reference picture is employed, the intra-layer prediction indicates single layer inter-prediction.

[0183] In step 1211, the decoder forwards the decoded / reconstructed current picture for display as part of the decoded video sequence.

[0184] 13 is a schematic diagram of an example system 1300 for coding a video sequence of images in a bitstream such as bitstream 900 when employing inter-layer prediction. System 1300 may be implemented by an encoder and a decoder, such as codec system 200, encoder 300, decoder 400, and / or video coding device 1000. Furthermore, system 1300 may be employed when implementing methods 100, 1100, and / or 1200 for decoding pictures according to unidirectional inter prediction 500, bidirectional inter prediction 600, and / or layer-based prediction 700 by employing RPL structure 800.

[0185] The system 1300 includes a video encoder 1302. The video encoder 1302 comprises an encoding module 1301 for encoding a current picture into a bitstream, the current picture being encoded according to inter-layer prediction based on an inter-layer reference picture. The encoding module 1301 is further for encoding a reference picture list structure into the bitstream, the reference picture list structure comprising a plurality of entries for a plurality of reference pictures including an entry associated with the current picture and indicating the inter-layer reference picture. The encoding module 1301 is further for encoding an inter-layer reference picture flag into the bitstream, the inter-layer reference picture flag indicating that the entry associated with the current picture is an ILRP entry. The video encoder 1302 further comprises a storage module 1303 for storing the bitstream for communication towards a decoder. The video encoder 1302 further comprises a transmission module 1305 for transmitting the bitstream towards the video decoder 1310. The video encoder 1302 may be further configured to perform any of the steps of the method 1100.

[0186] The system 1300 also includes a video decoder 1310. The video decoder 1310 comprises a receiving module 1311 for receiving a bitstream comprising a current picture, a reference picture list structure, and an inter-layer reference picture flag. The video decoder 1310 further comprises a determining module 1313 for determining whether an entry in the reference picture list structure associated with the current picture is an ILRP entry based on the inter-layer reference picture flag. The video decoder 1310 further comprises a decoding module 1315 for decoding the current picture according to inter-layer prediction based on the inter-layer reference picture indicated by the entry in the reference picture list structure when the entry is an ILRP entry. The video decoder 1310 further comprises a forwarding module 1317 for forwarding the current picture for display as part of a decoded video sequence. The video decoder 1310 may be further configured to perform any of the steps of the method 1200.

[0187] A first component is directly coupled to a second component when there is no intervening component between the first component and the second component, except for a line, trace, or another medium. A first component is indirectly coupled to a second component when there is an intervening component between the first component and the second component, other than a line, trace, or another medium. The term "coupled" and variations thereof include both directly coupled and indirectly coupled. Use of the term "about" means a range that includes ±10% of the number that follows, unless otherwise specified.

[0188] It should also be understood that the steps of the exemplary methods described herein need not necessarily be performed in the order described, and the order of steps of such methods should be understood to be exemplary only. Similarly, additional steps may be included in such methods, and some steps may be omitted or combined in a manner consistent with various embodiments of the present disclosure.

[0189] Although several embodiments are provided in this disclosure, it can be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples are to be considered as illustrative rather than restrictive, and the intention is not to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or some features may be omitted or not implemented.

[0190] Additionally, the techniques, systems, subsystems, and methods described and illustrated in various embodiments as separate or distinct may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations will be ascertainable by one of ordinary skill in the art and may be made without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0191] 200 Codec System 201 Segmented Video Signal 211 General purpose coder control components 213 Transform Scaling and Quantization Components 215 In-picture estimation components 217 Intra-picture prediction components 219 Motion Compensation Components 221 Motion Estimation Components 223 Decoded Picture Buffer Components 225 In-loop filter components 227 Filter Control Analysis Components 229 Scaling and Inverse Transformation Components 231 Header Formatting and Context-Adaptive Binary Arithmetic Coding Components 300 Video Encoder 301 Segmented Video Signal 313 Transformation and Quantization Components 317 Intra-picture prediction components 321 Motion Compensation Components 323 Decoded Picture Buffer Components 325 In-loop filter components 329 Inverse Transformation and Quantization Components 331 Entropy Coding Components 400 Video Decoder 417 Intra-picture prediction components 421 Motion Compensation Components 423 Decoded Picture Buffer Components 425 In-loop filter components 429 Inverse Transformation and Quantization Components 433 Entropy Decoding Components 500 Unidirectional Inter Prediction 510 Current Picture 511 Current Block 513 Motion Trajectory 530 Reference Pictures 531 Reference Block 533 Temporal distance 535 Motion Vector 600 Two-way Inter Prediction 610 Current Picture 611 Current Block 613 Motion Trajectory 620 Previous Reference Picture 621 Previous Reference Block 623 Preceding Temporal Distance 625 Leading Motion Vector 630 Subsequent Reference Pictures 631 Subsequent Reference Blocks 633 Subsequent Temporal Distance 635 Subsequent Motion Vectors 700 Layer-based Prediction 711~718 Pictures 721 Inter-layer Prediction 723 Inter Prediction 731 Layer N 732 Layer N+1 800 Reference Picture List Structure (RPL Structure) 811 RPL0 812 RPL1 815 Reference Picture List Structure Entry 821 listIdx 825 rplsIdx 833 ILRP Flag 835 ILRP Layer Indicator 900 Bitstream 910 Sequence Parameter Set (SPS) 911 Picture Parameter Set (PPS) 915 Slice Header 920 Image data 921 Access Unit 923 Pictures 925 slices 931 Reference Picture List Structure 933 ILRP Flag 935 ILRP Layer Indicator 1000 Video Coding Devices 1010 Transceiver unit (Tx / Rx) 1014 Coding Module 1020 downstream port 1030 Processor 1032 memory 1050 Upstream Port 1060 Input and / or Output (I / O) Devices 1300 System 1301 Encoding Module 1302 Video Encoder 1303 Memory Module 1305 Transmitting Module 1310 Video Decoder 1311 Receiver Module 1313 Decision Module 1315 Decryption Module 1317 Transfer Module

Claims

1. A device for storing a bitstream, comprising at least one storage device and at least one receiver, the at least one receiver configured to receive one or more bitstreams; the at least one storage device is configured to store the one or more bitstreams; The bitstream comprises a reference picture list structure comprising an inter-layer reference picture flag, the reference picture list structure being denoted as ref_pic_list_struct( listIdx , rplsIdx ), the inter-layer reference picture flag being denoted as inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx ] [ i ], and the inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx [ i ] is used to indicate whether the i-th entry in the ref_pic_list_struct( listIdx , rplsIdx ) associated with the current picture is an inter-layer reference picture (ILRP) entry, and the inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx When [i] is equal to 1, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is the ILRP entry, and when the inter_layer_ref_pic_flag [listIdx] [rplsIdx] [i] is equal to 0, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is not the ILRP entry. device.

2. 10. The device of claim 1, wherein the reference picture list structure further comprises an ILRP layer indicator that indicates a layer of an inter-layer reference picture.

3. The device of claim 1 or 2, wherein the listIdx identifies a reference picture list, the rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure.

4. The bitstream comprises a sequence parameter set (SPS), and the ref_pic_list_struct( listIdx, rplsIdx ) and the inter_layer_ref_pic_flag[ listIdx ][ rplsIdx 4. The device of claim 1 , wherein [ i ] is included in the SPS.

5. 5. The device of claim 1, wherein an inter-layer reference picture is in the same access unit (AU) as the current picture, and the inter-layer reference picture is associated with a lower layer identifier than the current picture.

6. 1. A method for storing a bitstream, comprising: receiving one or more bitstreams through at least one receiver; and storing the bitstream in one or more storage devices, wherein the bitstream comprises a reference picture list structure comprising an inter-layer reference picture flag, the reference picture list structure being denoted as ref_pic_list_struct( listIdx , rplsIdx ), the inter-layer reference picture flag being denoted as inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx ] [ i ], and the inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx [ i ] is used to indicate whether the i-th entry in the ref_pic_list_struct( listIdx , rplsIdx ) associated with the current picture is an inter-layer reference picture (ILRP) entry, and the inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx When [i] is equal to 1, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is the ILRP entry, and when the inter_layer_ref_pic_flag [listIdx] [rplsIdx] [i] is equal to 0, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is not the ILRP entry. method.

7. The method of claim 6 , wherein the bitstream further comprises an ILRP layer indicator indicating a layer of an inter-layer reference picture.

8. The method of claim 6 or 7, wherein the listIdx identifies a reference picture list, the rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure that returns the entry based on the listIdx and the rplsIdx.

9. The bitstream comprises a sequence parameter set (SPS), and the ref_pic_list_struct( listIdx, rplsIdx ) and the inter_layer_ref_pic_flag[ listIdx ][ rplsIdx 9. The method of claim 6, wherein [i] is included in the SPS.

10. 10. The method of claim 6, wherein an inter-layer reference picture is in the same access unit (AU) as the current picture, and the inter-layer reference picture is associated with a lower layer identifier than the current picture.

11. 1. A system for processing a bitstream, comprising: an encoding device, one or more storage devices, and a decoding device; The encoding device is configured to obtain a video signal and to encode the video signal to obtain one or more bitstreams, the bitstream comprising a reference picture list structure comprising an inter-layer reference picture flag, the reference picture list structure being denoted as ref_pic_list_struct( listIdx, rplsIdx ), the inter-layer reference picture flag being denoted as inter_layer_ref_pic_flag[ listIdx ][ rplsIdx ][ i ], and the inter_layer_ref_pic_flag[ listIdx ][ rplsIdx [ i ] is used to indicate whether the i-th entry in the ref_pic_list_struct( listIdx , rplsIdx ) associated with the current picture is an inter-layer reference picture (ILRP) entry, and the inter_layer_ref_pic_flag[ listIdx ] [ rplsIdx When [i] is equal to 1, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is the ILRP entry, and when the inter_layer_ref_pic_flag [listIdx] [rplsIdx] [i] is equal to 0, it indicates that the i-th entry in the ref_pic_list_struct (listIdx, rplsIdx) is not the ILRP entry; the one or more storage devices are used to store the one or more bitstreams; the decoding device is used to decode the one or more bitstreams; system.

12. The system of claim 11 , wherein the bitstream further comprises an ILRP layer indicator indicating a layer of an inter-layer reference picture.

13. The system of claim 11 or 12, wherein the listIdx identifies a reference picture list, the rplsIdx identifies an entry in the reference picture list, and ref_pic_list_struct is a syntax structure that returns the entry based on the listIdx and the rplsIdx.

14. 14. The system of claim 11, wherein an inter-layer reference picture is in the same access unit (AU) as the current picture, and the inter-layer reference picture is associated with a lower layer identifier than the current picture.

Images