HRD parameters for layer-based conformity testing

Abstract

To provide a video coding mechanism.SOLUTION: A video coding system includes encoding multiple sub-layer representations into a bitstream. Hypothetical reference decoder (HRD) parameters and a sub-layer coding picture buffer parameter present flag (sublayer_cpb_params_present_flag) are also encoded into the bitstream. When the sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sub-layer representations with a temporal identifier (TemporalId) less than the maximum TemporalId are inferred to be equal to the HRD parameters of the largest sub-layer representation with the maximum TemporalId, and a set of bitstream conformance tests is performed on the bitstream based on the HRD parameters. The bitstream is stored for communication toward a decoder.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Cross-Reference to Related Applications This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 905,244, filed Sep. 24, 2019, entitled “Hypothetical Reference Decoder (HRD) for Multi-Layer Video Bitstreams” by Ye-Kui Wang, which is incorporated herein by reference.

[0002] This disclosure generally relates to video coding, and more particularly to virtual reference decoder (HRD) parameter changes to support efficient encoding and / or compliance testing of multi-layer bitstreams.

Background Art

[0003] Even for relatively short videos, the amount of video data required for their depiction can be enormous, and thus there may be difficulties when streaming or otherwise communicating data over a communication network with limited bandwidth capacity. Therefore, video data is generally compressed before being communicated over modern telecommunications networks. Also, when storing video on a storage device, the size of the video can be a problem because memory resources may be limited. Video compression devices often use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Improved compression and decompression techniques that improve the compression ratio without sacrificing much or any of the image quality are desired, given limited network resources and the increasing demand for higher quality video.

Summary of the Invention

[0004] In one embodiment, the disclosure includes a method implemented by a decoder, the method comprising: receiving a bitstream by the decoder's receiver, which includes a plurality of sublayer representations, virtual reference decoder (HRD) parameters, and a sublayer coding picture buffer (CPB) parameter presence flag (sublayer_cpb_params_present_flag); the decoder's processor inferring that the HRD parameters of all sublayer representations whose TemporalId is less than the MaximumTemporalId are equal to the HRD parameters of the largest sublayer representation having the MaximumTemporalId when sublayer_cpb_params_present_flag is set to 0; and the processor decoding an image from the sublayer representations.

[0005] Video coding systems employ various conformance tests to ensure that the bitstream is decodeable by the decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for potential decodeable outputs for conformance. Corresponding parameters are included in the bitstream to perform conformance checks. The HRD can read the parameters and perform the tests. Video can contain many layers and many different output layer sets (OLS). On request, the encoder transmits one or more layers of selected OLS. For example, the encoder can transmit the best layer from the OLS that can be supported by the current network bandwidth. Problems may arise when the video is separated into multiple layers and / or sublayers. The encoder can encode these layers into a bitstream. In addition, the encoder can use an HRD to perform conformance tests to check the bitstream for conformance to the standard. To support such conformance tests, the encoder may be configured to include layer-specific HRD parameters in the bitstream. Layer-specific HRD parameters may be encoded per layer in some video coding systems. In some cases, layer-specific HRD parameters are the same for each layer, resulting in redundant information that unnecessarily increases the size of the video encoding. This embodiment includes a mechanism for reducing HRD parameter redundancy in videos employing multiple layers. The encoder can encode the HRD parameters of the top layer. The encoder can also encode the sublayer_cpb_params_present_flag. Setting sublayer_cpb_params_present_flag to 0 indicates that all lower layers should use the same HRD parameters as the top layer.In this context, the top layer has the largest layer identifier (ID), and lower layers are any layers with a layer ID smaller than the top layer's layer ID. In this way, the HRD parameters of lower layers can be omitted from the bitstream. This reduces the bitstream size and therefore the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0006] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, sublayer_cpb_params_present_flag is included in the video parameter set (VPS) in the bitstream.

[0007] Optionally, another embodiment of the embodiment is provided in which the maximum TemporalId of the maximum sublayer representation is represented as HRD maximum TemporalId(hrd_max_tid[i]) in any of the embodiments described above, where i represents the i-th HRD parameter syntax structure.

[0008] Optionally, another embodiment of the embodiment is provided in any of the embodiments described above, wherein a TemporalId smaller than the maximum TemporalId is in the range of 0 to hrd_max_tid[i]-1.

[0009] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameter includes a fixed picture rate general flag (fixed_pic_rate_general_flag[i]) indicating whether the temporal distance between the HRD output times of consecutive images in output order is constrained.

[0010] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameters include a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers.

[0011] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameters include a general video coding layer (VCL) HRD parameter presence flag (general_vcl_hrd_params_present_flag) indicating whether VCL HRD parameters related to the fitting point are present in the general HRD parameter syntax structure.

[0012] In one embodiment, the disclosure includes a method implemented by an encoder, the method comprising: the processor of the encoder encoding a plurality of sublayer representations into a bitstream; the processor encoding HRD parameters and sublayer_cpb_params_present_flag into a bitstream; the processor inferring that the HRD parameters of all sublayer representations whose TemporalId is less than the MaxTemporalId are equal to the HRD parameters of the largest sublayer representation having the MaxTemporalId when sublayer_cpb_params_present_flag is set to 0; and the processor performing a set of bitstream conformance tests on the bitstream based on the HRD parameters.

[0013] Video coding systems employ various conformance tests to ensure that the bitstream is decodeable by the decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for a potential decodeable output for conformance. Corresponding parameters are included in the bitstream to perform conformance checks. The HRD can read the parameters and perform the tests. Video can contain many layers and many OLS. Upon request, the encoder transmits one or more layers of selected OLS. For example, the encoder can transmit the best layer from the OLS that can be supported by the current network bandwidth. Problems may arise when the video is separated into multiple layers and / or sublayers. The encoder can encode these layers into a bitstream. Furthermore, the encoder may use an HRD to perform conformance tests to check the bitstream for conformance to the standard. To support such conformance tests, the encoder may be configured to include layer-specific HRD parameters in the bitstream. Layer-specific HRD parameters may be encoded per layer in some video coding systems. In some cases, layer-specific HRD parameters are the same for each layer, resulting in redundant information that unnecessarily increases the size of the video encoding. This embodiment includes a mechanism for reducing HRD parameter redundancy in videos employing multiple layers. The encoder can encode the HRD parameters of the top layer. The encoder can also encode the sublayer_cpb_params_present_flag. Setting sublayer_cpb_params_present_flag to 0 indicates that all lower layers should use the same HRD parameters as the top layer.In this context, the top layer has the largest layer ID, and lower layers are any layers with a layer ID smaller than the top layer's layer ID. In this way, the HRD parameters of lower layers can be omitted from the bitstream. This reduces the bitstream size and therefore the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0014] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, sublayer_cpb_params_present_flag is encoded into VPS in the bitstream.

[0015] Optionally, another embodiment of the embodiment is provided in any of the embodiments described above, where the maximum TemporalId of the maximum sublayer representation is represented as hrd_max_tid[i], where i represents the i-th HRD parameter syntax structure.

[0016] Optionally, another embodiment of the embodiment is provided in any of the embodiments described above, wherein a TemporalId smaller than the maximum TemporalId is in the range of 0 to hrd_max_tid[i]-1.

[0017] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameter includes fixed_pic_rate_general_flag[i] indicating whether the temporal distance between the HRD output times of consecutive images in output order is constrained.

[0018] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameters include a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers.

[0019] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the HRD parameter includes a general_vcl_hrd_params_present_flag indicating whether a VCL HRD parameter related to a fitting point is present in the general HRD parameter syntax structure.

[0020] In one embodiment, the disclosure includes a video coding apparatus comprising a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform any of the methods described above.

[0021] In one embodiment, the disclosure includes a non-transient computer-readable medium comprising a computer program product for use by a video coding apparatus, the computer program product comprising computer-executable instructions stored on the non-transient computer-readable medium, which, when executed by a processor, cause the video coding apparatus to perform any of the methods described above.

[0022] In one embodiment, the disclosure includes a decoder comprising: receiving means for receiving a bitstream including a plurality of sublayer representations, HRD parameters, and sublayer_cpb_params_present_flag; inference means for inferring that the HRD parameters of all sublayer representations whose TemporalId is less than the MaxTemporalId are equal to the HRD parameters of the Max sublayer representation having the MaxTemporalId when sublayer_cpb_params_present_flag is set to 0; decoding means for decoding images from the sublayer representations; and transferring means for transferring images for display as part of a decoded video sequence.

[0023] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the decoder is further configured to perform any of the methods described above.

[0024] In one embodiment, the present disclosure includes an encoder comprising: encoding means for encoding a plurality of sublayer representations into a bitstream; encoding means for encoding HRD parameters and a sublayer_cpb_params_present_flag into the bitstream; inferring means for inferring that the HRD parameters of all sublayer representations whose TemporalId is less than the maximum TemporalId are equal to the HRD parameters of the maximum sublayer representation having the maximum TemporalId when the sublayer_cpb_params_present_flag is set to 0; HRD means for performing a set of bitstream compliance tests on the bitstream based on the HRD parameters; and storage means for storing the bitstream for communication to a decoder.

[0025] Optionally, in any of the foregoing aspects, another embodiment of the aspect is provided in which the encoder is further configured to execute the method of any of the foregoing aspects.

[0026] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0027] These and other features will be more clearly understood from the following detailed description in conjunction with the accompanying drawings and the claims.

Brief Description of the Drawings

[0028] To understand the present disclosure more fully, reference is now made to the following brief description obtained in connection with the accompanying drawings and detailed description in which like reference numerals represent like parts.

[0029] [Figure 1] It is a flowchart of an exemplary method for coding a video signal.

[0030] [Figure 2] This is a schematic diagram illustrating an exemplary coding / decoding (codec) system for video encoding.

[0031] [Figure 3] This is a schematic diagram illustrating an example video encoder.

[0032] [Figure 4] This is a schematic diagram illustrating an exemplary video decoder.

[0033] [Figure 5] This is a schematic diagram illustrating an exemplary virtual reference decoder (HRD).

[0034] [Figure 6] This is a schematic diagram showing an exemplary multilayer video sequence configured for interlayer prediction.

[0035] [Figure 7] This is a schematic diagram showing an exemplary multi-layer video sequence configured for temporal scalability.

[0036] [Figure 8] This is a schematic diagram showing an example bitstream.

[0037] [Figure 9] This is a schematic diagram of an exemplary video coding device.

[0038] [Figure 10] This is a flowchart illustrating an exemplary method for encoding a video sequence into a bitstream by including inferred HRD parameters to support bitstream conformance testing by HRD.

[0039] [Figure 11]This is a flowchart illustrating an exemplary method for decoding a video sequence from a bitstream containing inferred HRD parameters.

[0040] [Figure 12] This is a schematic diagram of an exemplary system for coding a video sequence into a bitstream by including inferred HRD parameters. [Modes for carrying out the invention]

[0041] While exemplary embodiments of one or more embodiments are provided below, it should be understood from the outset that the disclosed systems and / or methods may be carried out using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary embodiments, drawings, and techniques shown below, including the exemplary designs and embodiments illustrated and described herein, and may be modified in the entirety of their equivalents within the scope of the appended claims.

[0042] The following terms are defined as set forth below, unless used in the opposite context herein. Specifically, the following definitions are intended to provide further clarity to this disclosure. However, terms may be described differently in different contexts. Therefore, the following definitions should be considered supplementary and should not be considered to limit any other definitions of such terms provided herein.

[0043] A bitstream is a sequence of bits containing compressed video data for transmission between an encoder and a decoder. An encoder is a device configured to employ an encoding process to compress video data into a bitstream. A decoder is a device configured to employ a decoding process to reconstruct video data from the bitstream for display. An image is an array of luma samples and / or chroma samples that make up a frame or its fields. An image being encoded or decoded is sometimes called the current picture for clarity. A Network Abstraction Layer (NAL) unit is a syntax structure containing data in the form of a raw byte sequence payload (RBSP), a representation of the data type, and interspersed anti-emulation bytes as needed. A Video Coding Layer (VCL) NAL unit is an NAL unit encoded to contain video data, such as an encoded slice of an image. A non-VCL NAL unit is an NAL unit containing non-video data, such as syntax and / or parameters that support decoding video data, performing conformance checks, or other operations. An Access Unit (AU) is a set of NAL units related to one particular output time, associated with each other according to a given classification rule. A decoding unit (DU) is an AU or a subset of an AU and its associated non-VCL NAL units. For example, an AU contains VCL NAL units and any non-VCL NAL units associated with the VCL NAL units within the AU. Furthermore, a DU contains a set or subset of VCL NAL units from an AU, as well as any non-VCL NAL units associated with the VCL NAL units within the DU. A layer is a set of VCL NAL units and their associated non-VCL NAL units that share a specified characteristic (e.g., common resolution, frame rate, image size, etc.). The decoding order is the order in which the syntax elements are processed by the decoding process.A video parameter set (VPS) is a data unit containing parameters related to the entire video.

[0044] A time-scalable bitstream is a bitstream encoded in multiple layers that provide various time resolutions / frame rates (for example, each layer is encoded to support a different frame rate). A sublayer is a time-scalable layer of a time-scalable bitstream that contains VCL NAL units with a specific time identifier value and associated non-VCL NAL units. For example, a time sublayer is a layer that contains video data associated with a specified frame rate. A sublayer representation is a subset of the bitstream containing the NAL units of a particular sublayer and its lower sublayers. Thus, one or more time sublayers can be combined to achieve a sublayer representation that can be decoded to become a video sequence with a specified frame rate. An output layer set (OLS) is a set of layers in which one or more layers are designated as output layers. An output layer is a layer designated for output (for example, to a display). An OLS index is an index that uniquely identifies a corresponding OLS. The zeroth (0th) OLS is an OLS that contains only the lowest layer (the layer with the lowest layer identifier) ​​and therefore only the output layer. A time identifier (ID) is a data element that indicates the time position of the data within the video sequence. The sub-bitstream extraction process removes NAL units from the bitstream that do not belong to the target set, as determined by the target OLS index and target highest time ID. The sub-bitstream extraction process outputs a sub-bitstream from the bitstream that contains the NAL units that are part of the target set.

[0045] HRD is a decoder model that operates on an encoder and checks the variability of the bitstream generated by the encoding process to verify its compliance with specified constraints. Bitstream compliance testing is a test to determine whether the encoded bitstream conforms to standards such as VVC (Versatile Video Coding). HRD parameters are syntactic elements that initialize and / or define the operating conditions of HRD. Sequence-level HRD parameters are HRD parameters that apply to the entire coded video sequence. The maximum HRD time ID (hrd_max_tid[i]) specifies the time ID of the highest sublayer representation in which the HRD parameter is included in the i-th OLS HRD parameter set. The general_hrd_parameters syntax structure is a syntax structure that includes sequence-level HRD parameters. An operating point (OP) is a time subset of an OLS, identified by its OLS index and highest time ID. The target OP is the OP selected in HRD for compliance testing. The target OLS is the OLS selected to be extracted from the bitstream. The decoding unit HRD parameter presence flag (decoding_unit_hrd_params_present_flag) indicates whether the corresponding HRD parameter operates at the DU level or the AU level. The coded image buffer (CPB) is a first-in, first-out buffer within the HRD that contains coded images in decoding order for use during bitstream conformance verification. The decoded image buffer (DPB) is a buffer for holding decoded images for reference, output sorting, and / or output delay.

[0046] Supplemental Expansion Information (SEI) messages are semantically specified syntax structures that convey information not required by the decoding process to determine the sample values ​​of a decoded image. Scalable nesting SEI messages are messages containing multiple SEI messages corresponding to one or more OLS or one or more layers. Non-scalable nesting SEI messages are messages that are not nested and therefore contain a single SEI message. Buffering Period (BP) SEI messages are SEI messages containing HRD parameters for initializing HRD to manage CPB. Image Timing (PT) SEI messages are SEI messages containing HRD parameters for managing AU delivery information in CPB and / or DPB. Decoded Unit Information (DUI) SEI messages are SEI messages containing HRD parameters for managing delivery information for DU in CPB and / or DPB.

[0047] The CPB removal delay is the period during which the corresponding current AU can remain in the CPB before being removed and output to the DPB. The initial CPB removal delay is the default CPB removal delay for each image, AU, and / or DU in the bitstream, OLS, and / or layer. The CPB removal offset is the location within the CPB used to determine the boundary of the corresponding AU in the CPB. The initial CPB removal offset is the default CPB removal offset associated with each image, AU, and / or DU in the bitstream, OLS, and / or layer. The Decoded Image Buffer (DPB) output delay information is the period during which the corresponding AU can remain in the DPB before output. The CPB removal delay information is information related to the removal of the corresponding DU from the CPB. The delivery schedule specifies the timing for the delivery of video data to and from memory locations such as the CPB and / or DPB. The VPS layer ID (vps_layer_id) is a syntax element that indicates the layer ID of the i-th layer shown in the VPS. The number of output layer sets minus 1 (num_output_layer_sets_minus1) is a syntax element that specifies the total number of OLS specified by the VPS. The HRD-encoded image buffer count (hrd_cpb_cnt_minus1) is a syntax element that specifies the number of alternative CPB delivery schedules. The sublayer CPB parameter presence flag (sublayer_cpb_params_present_flag) is a syntax element that specifies whether the set of OLS HRD parameters includes the HRD parameters for the specified sublayer representation. The schedule index (ScIdx) is an index that identifies the delivery schedule. The BP CPB count minus 1 (bp_cpb_cnt_minus1) is a syntax element that specifies the number of pairs of initial CPB removal delay and offset, and therefore the number of delivery schedules available for the time sublayer. The NAL unit header layer identifier (nuh_layer_id) is a syntax element that specifies the identifier of the layer containing the NAL unit.The fixed_pic_rate_general_flag syntax element specifies whether to constrain the temporal distance between HRD output times of consecutive images in output order. The sublayer_hrd_parameters syntax structure is a syntax structure that includes the HRD parameters of the corresponding sublayer. The general_vcl_hrd_params_present_flag flag specifies whether VCL HRD parameters are present in the syntax structure of general HRD parameters. The bp_max_sublayers_minus1 syntax element specifies the maximum number of time sublayers in which CPB removal delay and CPB removal offset are indicated in BP SEI messages. The vps_max_sublayers_minus1 syntax element specifies the maximum number of time sublayers that may exist in a layer specified by VPS. The Scalable Nesting OLS flag specifies whether a scalable nesting SEI message applies to a specific OLS or to a specific layer. Scalable Nesting OLS Number - 1 (num_olss_minus1) is a syntax element that specifies the number of OLS to which a scalable nesting SEI message applies. Nesting OLS Index (NestingOlsIdx) is a syntax element that specifies the OLS index of the OLS to which a scalable nesting SEI message applies. Target OLS Index (targetOlsIdx) is a variable that identifies the OLS index of the OLS to be decrypted. Total OLS Number - 1 (OLS-1) is a syntax element that specifies the total number of OLS specified in the VPS.

[0048] The following acronyms are used herein: Access Unit (AU), Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Codified Layer Video Sequence (CLVS), Codified Layer Video Sequence Start (CLVSS), Codified Video Sequence (CVS), Codified Video Sequence Start (CVSS), Joint Video Expert Team (JVET), Virtual Reference Decoder (HRD), Motion Constrained Tilesets (MCTS), Maximum Transmission Unit (MTU), Network Abstraction Layer (NAL), Output Layer Set (OLS), Image Order Count (POC), Random Access Point (RAP), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), Video Parameter Set (VPS), and General-Purpose Video Coding (VVC).

[0049] Many video compression techniques can be employed to minimize data loss and reduce the size of video files. For example, video compression techniques may include performing spatial (e.g., within an image) prediction and / or temporal (e.g., between images) prediction to reduce or eliminate data redundancy in a video sequence. In block-based video coding, a video slice (e.g., a video image or a portion of a video image) may be divided into video blocks, which are sometimes called tree blocks, coded tree blocks (CTBs), coded tree units (CTUs), coded units (CUs), and / or coded nodes. Video blocks in an intra-coding (I) slice of an image are coded using spatial prediction with respect to reference samples of adjacent blocks in the same image. Video blocks in an intercoding (P) or bidirectional prediction (B) slice of an image may be coded by employing spatial prediction with respect to reference samples of adjacent blocks in the same image, or temporal prediction with respect to reference samples of other reference images. A picture may be called a frame and / or image, and a reference picture may be called a reference frame and / or reference image. Spatial or temporal prediction yields predicted blocks representing image blocks. Residual data represents the difference in pixels between the original image blocks and the predicted blocks. Thus, intercoded blocks are encoded according to motion vectors pointing to the blocks of reference samples forming the predicted blocks, and residual data showing the difference between the encoded blocks and the predicted blocks. Intracoded blocks are encoded according to the intracoding mode and residual data. For further compression, the residual data may be transformed from the pixel region to the transformation region. This results in residual transformation coefficients that can be quantized. The quantized transformation coefficients may first be arranged in a two-dimensional array. The quantized transformation coefficients may be scanned to generate a one-dimensional vector of transformation coefficients. Entropy coding may be applied to achieve even greater compression. Such video compression techniques are described in more detail below.

[0050] To ensure that encoded video can be accurately decoded, video is encoded and decoded according to the corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding plus Depth (MVC+D), as well as 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC joint video experts team (JVET) began developing a video coding standard called Versatile Video Coding (VVC). VVC is included in Working Drafts (WDs), including JVET-O2001-v14.

[0051] Video coding systems employ various conformance tests to ensure that a bitstream is decodeable by a decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for potential decodeable outputs for conformance. Corresponding parameters are included in the bitstream to perform the conformance checks. A virtual reference decoder (HRD) can read the parameters and perform the tests. Video can contain many layers and many different output layer sets (OLS). Upon request, the encoder transmits one or more layers of a selected OLS. For example, the encoder can transmit the best layer from the OLS that can be supported by the current network bandwidth. The first problem with this technique is that a considerable number of layers are tested but are not actually transmitted to the decoder. However, parameters to support such tests may still be included in the bitstream, unnecessarily increasing the size of the bitstream.

[0052] In the first embodiment, a mechanism for applying bitstream conformance testing to only each OLS is disclosed herein. In this way, when testing the corresponding OLS, the entire bitstream, each layer, and the decodeable output are tested together. Thus, the number of conformance tests is reduced, thereby reducing the use of processor and memory resources in the encoder. Furthermore, by reducing the number of conformance tests, the number of associated parameters included in the bitstream can be reduced. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0053] The second problem is that the signaling process for HRD parameters used in HRD compliance testing in some video coding systems can become complex in a multi-layer context. For example, a set of HRD parameters may be signaled for each layer of each OLS. Such HRD parameters may be signaled at different locations within the bitstream, depending on the intended range of the parameters. As a result, the scheme becomes more complex as more layers and / or OLS are added. Furthermore, HRD parameters for different layers and / or OLS may contain redundant information.

[0054] In a second embodiment, a mechanism for signaling a global set of HRD parameters for OLS and corresponding layers is disclosed herein. For example, all sequence-level HRD parameters applicable to all OLS and all layers contained within the OLS are signaled by a video parameter set (VPS). The VPS is signaled once with the bitstream, and therefore the sequence-level HRD parameters are signaled once. Furthermore, the sequence-level HRD parameters may be constrained to be identical across all OLS. In this way, redundant signaling is reduced and coding efficiency is improved. This technique also simplifies the HRD process. As a result, the usage of processor, memory, and / or network signaling resources is reduced in both the encoder and decoder.

[0055] A third problem can arise when the video coding system performs bitstream conformance checks. Video may be coded into multiple layers and / or sublayers, which can then be organized into an OLS. Each layer and / or sublayer in each OLS is checked for conformance according to a delivery schedule. Each delivery schedule is associated with different coded image buffer (CPB) sizes and CPB delays to account for different transmission bandwidths and system capabilities. Some video coding systems allow each sublayer to define any number of delivery schedules. This can result in a large amount of signaling to support conformance checks, which in turn reduces the efficiency of bitstream coding.

[0056] In a third embodiment, a mechanism for improving the coding efficiency of video containing multiple layers is disclosed herein. Specifically, all layers and / or sublayers are constrained to contain the same number of CPB delivery schedules. For example, the encoder can determine the maximum number of CPB delivery schedules to be used for any one layer and set the number of CPB delivery schedules for all layers to this maximum. The number of delivery schedules may then be signaled once, for example, as part of the HRD parameter in a VPS. This eliminates the need to signal several schedules per layer / sublayer. In some examples, all layers / sublayers of the OLS may also share the same delivery schedule index. These changes reduce the amount of data used to signal data related to compliance checks. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0057] A fourth issue can arise when video is coded into multiple layers and / or sublayers, which are then organized into an OLS. The OLS may include a zeroth (0th) OLS containing only the output layer. Supplemental Extension Information (SEI) messages may be included in the bitstream to inform the HRD of layer / OLS-specific parameters used to test multiple layers of the bitstream for compliance with the standard. Specifically, if an OLS is included in the bitstream, a scalable nesting SEI message is employed. A scalable nesting SEI message consists of a group of nested SEI messages that apply to one or more OLSs and / or one or more layers of an OLS. Each nested SEI message may include an indicator to show its association with the corresponding OLS and / or layer. Nested SEI messages are configured for use with multiple layers and may contain irrelevant information when applied to a 0th OLS containing a single layer.

[0058] In a fourth embodiment, a mechanism for improving the coding efficiency of a video including a zeroth OLS is disclosed herein. A non-scalable nested SEI message is employed in the zeroth OLS. The non-scalable nested SEI message is constrained to apply only to the zeroth OLS and therefore only to the output layers included in the zeroth OLS. In this way, irrelevant information such as nesting relationships and layer indications can be omitted from the SEI message. The non-scalable nested SEI message may be used as a buffering period (BP) SEI message, an image timing (PT) SEI message, a decoding unit (DU) SEI message, or a combination thereof. These modifications reduce the amount of data used to signal conformance check-related information to the zeroth OLS. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0059] A fifth problem can also arise when video is separated into multiple layers and / or sublayers. Encoders can encode these layers into bitstreams. Furthermore, encoders can use HRDs to perform conformance tests to check the bitstream for compliance with standards. Encoders may be configured to include layer-specific HRD parameters in the bitstream to support such conformance tests. Layer-specific HRD parameters may be encoded per layer in some video coding systems. In some cases, the layer-specific HRD parameters are the same per layer, resulting in redundant information that unnecessarily increases the size of the video encoding.

[0060] In a fifth embodiment, a mechanism for reducing HRD parameter redundancy for video employing multiple layers is disclosed herein. The encoder can encode the HRD parameters of the top layer. The encoder can also encode a sublayer CPB parameter presence flag (sublayer_cpb_params_present_flag). Setting sublayer_cpb_params_present_flag to 0 indicates that all lower layers should use the same HRD parameters as the top layer. In this context, the top layer is the layer with the largest layer identifier (ID), and lower layers are any layers with a layer ID smaller than the layer ID of the top layer. In this way, the HRD parameters of lower layers can be omitted from the bitstream. This reduces the bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0061] The sixth issue concerns the use of Sequence Parameter Sets (SPS), which contain syntax elements associated with each video sequence in a video. A video coding system can encode video in layers and / or sublayers. Video sequences may behave differently in different layers and / or sublayers. Therefore, different layers may refer to different SPS. BP SEI messages can indicate which layers / sublayers are being checked for compliance with the standard. Some video coding systems may indicate that a BP SEI message applies to the layer / sublayer indicated by the SPS. This can lead to unexpected errors, as different layers may refer to different SPS, and such SPS may contain conflicting information.

[0062] In a sixth embodiment, a mechanism for addressing errors related to conformance checks when multiple layers are employed in a video sequence is disclosed herein. Specifically, the BP SEI message is modified to indicate that the conformance of any number of layers / sublayers described in the VPS can be checked. For example, the BP SEI message may include a bp_max_sublayers_minus1 syntax element, which indicates the number of layers / sublayers associated with the data in the BP SEI message. On the other hand, the vps_max_sublayers_minus1 syntax element in the VPS indicates the number of sublayers in the entire video. The bp_max_sublayers_minus1 syntax element can be set to any value from 0 to the value of the vps_max_sublayers_minus1 syntax element. In this way, the conformance of any number of layers / sublayers in the video can be checked while avoiding layer-based sequencing problems related to SPS inconstancy. Therefore, this disclosure avoids layer-based coding errors and thus improves the functionality of encoders and / or decoders. Furthermore, this embodiment supports layer-based coding, which can improve coding efficiency. Thus, this embodiment supports a reduction in processor, memory, and / or network resource usage in encoders and / or decoders.

[0063] The seventh issue concerns the layers included in the OLS. Each OLS includes at least one output layer configured to be displayed by a decoder. The encoder's HRD can check each OLS for compliance with the standard. A compliant OLS can always be decoded and displayed by a compliant decoder. The HRD process may be partially managed by SEI messages. For example, a scalable nesting SEI message may contain scalable nested SEI messages. Each scalable nested SEI message may contain data related to the corresponding layer. When performing a compliance check, the HRD can perform a bitstream extraction process on the target OLS. Data not related to the layers within the OLS is generally removed before compliance testing (e.g., before transmission) so that each OLS can be checked separately. Some video coding systems do not remove such messages during the sub-bitstream extraction process because scalable nesting SEI messages relate to multiple layers. Therefore, even when a scalable nesting SEI message is not relevant to any layer of the target OLS (the extracted OLS), it may remain in the bitstream after sub-bitstream extraction. This can increase the size of the final bitstream without providing any additional functionality.

[0064] In a seventh embodiment, a mechanism for reducing the size of a multilayer bitstream is disclosed herein. During sub-bitstream extraction, scalable nesting SEI messages may be considered for removal from the bitstream. If a scalable nesting SEI message is associated with one or more OLSs, the scalable nested SEI messages within the scalable nesting SEI message are checked. If a scalable nested SEI message is not associated with any layer of the target OLS, the entire scalable nesting SEI message can be removed from the bitstream. As a result, the size of the bitstream sent to the decoder is reduced. Thus, this embodiment improves coding efficiency and reduces the use of processor, memory, and / or network resources in both the encoder and decoder.

[0065] Figure 1 is a flowchart of an exemplary operation method 100 for coding a video signal. Specifically, the video signal is encoded by an encoder. The encoding process compresses the video signal by employing various mechanisms to reduce the video file size. Reducing the file size allows the compressed video file to be sent to the user while reducing the associated bandwidth overhead. The decoder decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process so that the decoder can consistently reconstruct the video signal.

[0066] In step 101, a video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. In another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. A video file can contain both audio and video components. The video component contains a series of image frames, which, when viewed sequentially, give the impression of visual movement. Each frame contains pixels, represented in this specification with respect to light, called the lumens component (or lumens sample), and color, called the chromens component (or color sample). In some examples, the frames may also include depth values ​​to support a three-dimensional view.

[0067] In step 103, the video is divided into blocks. The division involves subdividing the pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels × 64 pixels). A CTU contains both luminous and chroma samples. A coding tree can be employed to divide the CTUs into blocks, and then the blocks can be recursively subdivided until a configuration supporting further coding is achieved. For example, the luminous component of a frame may be subdivided until the individual blocks contain relatively uniform lighting values. Furthermore, the chroma component of a frame may be subdivided until the individual blocks contain relatively uniform color values. Thus, the division mechanism varies depending on the content of the video frame.

[0068] In step 105, various compression mechanisms are employed to compress the image blocks divided in step 103. For example, interpretation and / or intrapretation can be employed. Interpretation is designed to take advantage of the fact that objects tend to appear in consecutive frames within a common scene. Therefore, it is not necessary to repeatedly describe the block representing the object in the reference frame to adjacent frames. Specifically, objects such as tables may remain in the same position across multiple frames. Therefore, a table is described once, and adjacent frames can refer to the reference frame. A pattern matching mechanism can be used to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to the movement of the object or the movement of the camera. As a particular example, a video may show a car moving across the screen across multiple frames. Such motion can be described using motion vectors. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in a frame to the coordinates of an object in a reference frame. Therefore, interpretation can encode the image block in the current frame as a set of motion vectors that indicate the offset from the corresponding block in the reference frame.

[0069] Intra-prediction encodes blocks within a common frame. It leverages the fact that luma and chroma components tend to cluster within a frame. For example, some green patches in a tree tend to be adjacent to similar green patches. Intra-prediction employs multiple directional prediction modes (e.g., 33 in HEVC), a planar mode, and a DC mode. Directional modes indicate that the current block is similar / identical to samples of adjacent blocks in the corresponding direction. Planar modes indicate that a sequence of blocks along a row / column (e.g., a plane) can be interpolated based on adjacent blocks at the edges of the row. Planar modes effectively show smooth light / color transitions across rows / columns by using a relatively constant slope when changing values. DC modes are employed for boundary smoothing and indicate that a block is similar / identical to the mean associated with samples of all adjacent blocks associated with the angular direction of the directional prediction mode. Thus, intra-predicted blocks can represent image blocks not as actual values, but as various relational prediction mode values. Furthermore, the interpretation block can represent the image block as a motion vector value rather than its actual value. In either case, the prediction block may not accurately represent the image block in some cases. The difference is stored in the residual block. Transformations can be applied to the residual block to further compress the file.

[0070] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. Block-based prediction, as described above, can generate blocky images in the decoder. Furthermore, block-based prediction may reconstruct the encoded blocks after encoding them for later use as reference blocks. In-loop filtering repeatedly applies noise suppression filters, deblocking filters, adaptive loop filters, and sample-adaptive offset (SAO) filters to blocks / frames. These filters reduce such blocking artifacts, allowing the encoded file to be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference blocks, making it less likely that artifacts will generate additional artifacts in subsequent blocks encoded based on the reconstructed reference blocks.

[0071] Once the video signal has been split, compressed, and filtered, in step 109 the resulting data is encoded into a bitstream. The bitstream contains the data described above, as well as any signaling data desired to support appropriate video signal reconstruction in the decoder. For example, such data may include split data, prediction data, residual blocks, and various flags that give coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. Alternatively, the bitstream may be broadcast and / or multicast to multiple decoders. The creation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 may be performed sequentially and / or simultaneously across many frames and blocks. The order shown in Figure 1 is presented for clarity and ease of discussion and is not intended to restrict the video coding process to a specific order.

[0072] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder employs an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. In step 111, the decoder uses the syntax data from the bitstream to determine the division of frames. This division should match the result of the block division in step 103. Next, the entropy coding / decoding employed in step 111 is described. The encoder makes many choices during the compression process, such as selecting a block division scheme from several possible options based on the spatial arrangement of values ​​in the input image. Signaling the precise selection may involve using a number of bins. As used herein, a bin is a binary value (e.g., a bit value that may change depending on the context) treated as a variable. Entropy coding allows the encoder to discard any option that is obviously not feasible for a particular case, leaving a set of acceptable options. Each acceptable option is assigned a codeword. The length of the codeword is based on the number of allowed options (e.g., one bin for two options, two bins for three or four options, etc.). The encoder then encodes the codeword for the selected option. This method reduces the size of the codeword because the desired size is sufficient to uniquely indicate a selection from a small subset of allowed options, rather than uniquely indicating a selection from a potentially large set of all possible options. The decoder then decodes this selection by determining the set of allowed options, similar to the encoder. By determining the set of allowed options, the decoder can read the codeword and determine the selection made by the encoder.

[0073] In step 113, the decoder performs block decoding. Specifically, the decoder generates residual blocks using the inverse transform. The decoder then reconstructs the image blocks according to the partitioning using the residual blocks and the corresponding prediction blocks. The prediction blocks may include both intra-prediction blocks and inter-prediction blocks generated by the encoder in step 105. The reconstructed image blocks are then placed into the frames of the reconstructed video signal according to the partitioning data determined in step 111. The syntax of step 113 may also be signaled in the bitstream via entropy coding, as described above.

[0074] In step 115, the frames of the reconstructed video signal are filtered in a manner similar to that in step 107 of the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be applied to the frames to remove blocking artifacts. Once the frames are filtered, the video signal may be output to a display for viewing by the end user in step 117.

[0075] Figure 2 is a schematic diagram of an exemplary coding / decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functions to support the implementation of operation method 100. The codec system 200 is generalized to show components used in both the encoder and the decoder. The codec system 200 receives and divides the video signal, as described with respect to steps 101 and 103 of operation method 100, resulting in the divided video signal 201. The codec system 200 then compresses the divided video signal 201 into an encoded bitstream, as described with respect to steps 105, 107, and 109 of method 100. When operating as a decoder, the codec system 200 generates an output video signal from the bitstream, as described with respect to steps 111, 113, 115, and 117 of operation method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-image estimation component 215, an intra-image prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter-controlled analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. These components are coupled together as shown in the figure. In Figure 2, black lines indicate the movement of data being encoded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All components of the codec system 200 may reside within the encoder. The decoder may contain a subset of the components of the codec system 200.For example, the decoder may include an intra-image prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.

[0076] The partitioned video signal 201 is a captured video sequence that has been divided into blocks of pixels by a coding tree. The coding tree subdivides blocks of pixels into smaller blocks of pixels using various partitioning modes. These blocks can then be subdivided into even smaller blocks. Blocks are sometimes called nodes on the coding tree. Larger parent nodes are subdivided into smaller child nodes. The number of times a node is subdivided is called the node / coding tree depth. The partitioned blocks may sometimes be contained within a coding unit (CU). For example, a CU can be a sub-part of a CTU, including a luma block, a red difference chroma (Cr) block, and a blue difference chroma (Cb) block, along with the corresponding syntax instructions for the CU. Partitioning modes can include binary trees (BT), ternary trees (TT), and quadary trees (QT), which are employed to subdivide nodes into two, three, or four child nodes of various shapes, depending on the partitioning mode used. The divided video signal 201 is transferred for compression to a general coder control component 211, a transformation scaling and quantization component 213, an intra-image estimation component 215, a filter control analysis component 227, and a motion estimation component 221.

[0077] The general coder control component 211 is configured to make decisions related to coding the images of a video sequence into a bitstream according to application constraints. For example, the general coder control component 211 manages the optimization of bitrate / bitstream size for reconstruction quality. Such decisions can be made based on memory space / bandwidth availability and image resolution requests. The general coder control component 211 also manages buffer utilization considering transmission speed and mitigates buffer underrun and overrun issues. To manage these issues, the general coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the general coder control component 211 can dynamically increase the complexity of compression to improve resolution and increase bandwidth usage, or decrease the complexity of compression to reduce resolution and bandwidth usage. Thus, the general coder control component 211 controls other components of the codec system 200 to balance the relationship between video signal reconstruction quality and bitrate. The general coder control component 211 creates control data that controls the operation of other components. The control data is also transferred to the header formatting and CABAC component 231, encoded into a bitstream, and signaling parameters for decoding by the decoder.

[0078] The divided video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for interprediction. A frame or slice of the divided video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform relative interprediction coding of the received video blocks for one or more blocks within one or more reference frames, providing time predictions. The codec system 200 can perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0079] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors can, for example, represent the relative displacement of the coded object with respect to the predicted block. A predicted block is a block that is found to closely match the coded block in terms of pixel difference. Predicted blocks are sometimes called reference blocks. Such pixel differences may be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. HEVC employs several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be split into CTBs, and then the CTBs can be split into CBs to be included in the CUs. A CU can be coded as a prediction unit (PU) containing the prediction data and / or a transformation unit (TU) containing the residual data transformed for the CUs. The motion estimation component 221 generates motion vectors, PUs, and TUs by using rate-distortion analysis as part of the rate-distortion optimization process. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame and select the reference blocks, motion vectors, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of video reconstruction (e.g., the amount of data loss due to compression) and coding efficiency (e.g., the size of the final encoding).

[0080] In some examples, the codec system 200 can calculate values ​​for sub-integer pixel positions of the reference image stored in the decoded image buffer component 223. For example, the video codec system 200 can interpolate values ​​for quarter-pixel, eighth-pixel, or other fractional-pixel positions of the reference image. Thus, the motion estimation component 221 can perform a relative motion search between full-pixel positions and fractional-pixel positions and output a motion vector with fractional-pixel precision. The motion estimation component 221 calculates the motion vector of the PU of the reference image in the video block during the intercoded slice by comparing the PU position with the predicted block position. The motion estimation component 221 outputs the calculated motion vector as motion data to the header formatting and CABAC component 231 for encoding and outputs the motion to the motion compensation component 219.

[0081] Motion compensation performed by the motion compensation component 219 may include fetching or generating a predicted block based on a motion vector determined by the motion estimation component 221. In this case as well, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated in some examples. Upon receiving a motion vector for the PU of the current video block, the motion compensation component 219 can determine the location of the predicted block pointed to by the motion vector. The residual video block is then formed by subtracting the pixel values ​​of the predicted block from the coded pixel values ​​of the current video block to form a pixel difference value. Generally, the motion estimation component 221 performs motion estimation for the lumens component, and the motion compensation component 219 uses a motion vector calculated based on the lumens component for both the chromens and lumens components. The predicted and residual blocks are then transferred to the transformation scaling and quantization component 213.

[0082] The divided video signal 201 is also sent to the intra-image estimation component 215 and the intra-image prediction component 217. Similar to the motion estimation component 221 and the motion compensation component 219, the intra-image estimation component 215 and the intra-image prediction component 217 may be highly integrated, but are shown separately for conceptual purposes. The intra-image estimation component 215 and the intra-image prediction component 217 intra-predict the current block for the block in the current frame, as an alternative to the inter-prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames, as described above. In particular, the intra-image estimation component 215 determines the intra-prediction mode to use for encoding the current block. In some examples, the intra-image estimation component 215 selects an appropriate intra-prediction mode from several tested intra-prediction modes to encode the current block. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0083] For example, the intra-image estimation component 215 calculates rate-distortion values ​​for various tested intra-prediction modes using rate-distortion analysis and selects the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original unencoded block encoded to produce the encoded block, as well as the bit rate (e.g., number of bits) used to produce the encoded block. The intra-image estimation component 215 calculates a ratio from the distortion and rate of various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block. In addition, the intra-image estimation component 215 may be configured to encode depth blocks of the depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

[0084] The intra-image prediction component 217, when implemented on an encoder, can generate residual blocks from prediction blocks based on a selected intra-prediction mode determined by the intra-image estimation component 215, or, when implemented on a decoder, can read residual blocks from a bitstream. The residual blocks contain the difference in values ​​between the prediction blocks and the original blocks and are represented as a matrix. The residual blocks are then transferred to the transformation scaling and quantization component 213. The intra-image estimation component 215 and the intra-image prediction component 217 can operate on both luma and chroma components.

[0085] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transform, to the residual block to generate a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms can also be used. The transform can convert the residual information from the pixel value domain to a transformation domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bitrate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the transformation scaling and quantization component 213 can then perform a scan of a matrix containing the quantized transformation coefficients. The quantized transformation coefficients are then transferred to the header formatting and CABAC component 231 to be encoded into a bitstream.

[0086] The scaling and inverse transform component 229 supports motion estimation by applying the inverse operations of the transform scaling and quantization component 213. The scaling and inverse transform component 229 reconstructs the residual block in the pixel region by applying inverse scaling, transform, and / or quantization for later use as a reference block that may become a prediction block for another current block, for example. The motion estimation component 221 and / or motion compensation component 219 can compute the reference block by adding the residual block to the corresponding prediction block for use in motion estimation of a later block / frame. A filter is applied to the reconstructed reference block to mitigate artifacts generated during scaling, quantization, and transform. Such artifacts can normally lead to inaccurate predictions (and generate additional artifacts) when predicting subsequent blocks.

[0087] The filter-controlled analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, a transformed residual block from the scaling and inverse transform component 229 can be combined with the corresponding predicted block from the intra-image prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. A filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block instead. Like the other components in Figure 2, the filter-controlled analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are shown separately for conceptual purposes. A filter applied to a reconstructed reference block is applied to a specific spatial region and includes several parameters to adjust how such a filter is applied. The filter-controlled analysis component 227 analyzes the reconstructed reference block to determine where such a filter is applied and sets the corresponding parameters. Such data is transferred to the header formatting and CABAC component 231 as filter-controlled data for encoding. The in-loop filter component 225 applies such a filter based on the filter-controlled data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Depending on the embodiment, such filters may be applied in the spatial / pixel domain (e.g., on reconstructed pixel blocks) or in the frequency domain.

[0088] When operating as an encoder, the filtered and reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for later use in the motion estimation described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and transfers them to the display as part of the output video signal. The decoded image buffer component 223 may be any memory device capable of storing the prediction blocks, residual blocks, and / or reconstructed image blocks.

[0089] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to the decoder. Specifically, the header formatting and CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. Furthermore, prediction data, including intra-prediction and motion data, as well as residual data in the form of quantized transformation coefficient data, are all encoded into a bitstream. The final bitstream contains all the information desired by the decoder to reconstruct the original segmented video signal 201. Such information may include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, indications of the most probable intra-prediction mode, and indications of segmentation information. Such data can be encoded using entropy coding. For example, information can be encoded by employing context-adaptive variable-length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), stochastic interval-partitioned entropy (PIPE) coding, or other entropy coding techniques. Following entropy coding, the encoded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0090] Figure 3 is a block diagram showing an exemplary video encoder 300. The video encoder 300 can be employed to perform the encoding function of the codec system 200 and / or to perform steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 splits the input video signal, resulting in a split video signal 301 that is substantially similar to the split video signal 201. The split video signal 301 is then compressed by the components of the encoder 300 and encoded into a bitstream.

[0091] Specifically, the divided video signal 301 is transferred to the intra-image prediction component 317 for intra-prediction. The intra-image prediction component 317 may be substantially the same as the intra-image estimation component 215 and the intra-image prediction component 217. The divided video signal 301 is also transferred to the motion compensation component 321 for intra-prediction based on a reference block in the decoding image buffer component 323. The motion compensation component 321 may be substantially the same as the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-image prediction component 317 and the motion compensation component 321 are transferred to the transformation and quantization component 313 for transformation and quantization of the residual blocks. The transformation and quantization component 313 may be substantially the same as the transformation scaling and quantization component 213. The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are transferred to the entropy coding component 331 for encoding into a bitstream. The entropy coding component 331 may be substantially the same as the header formatting and CABAC component 231.

[0092] The transformed and quantized residual blocks and / or corresponding predicted blocks are also transferred from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into reference blocks for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. Depending on the embodiment, the in-loop filter of the in-loop filter component 325 is also applied to the residual blocks and / or the reconstructed reference blocks. The in-loop filter component 325 may be substantially similar to the filter-controlled analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters as described with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded image buffer component 323 for use as reference blocks by the motion compensation component 321. The decoded image buffer component 323 may be substantially similar to the decoded image buffer component 223.

[0093] Figure 4 is a block diagram showing an exemplary video decoder 400. The video decoder 400 can be employed to perform the decoding function of the codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of the operation method 100. The decoder 400 receives a bitstream from, for example, the encoder 300 and generates an output video signal reconstructed based on the bitstream for display to the end user.

[0094] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may employ header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, segmentation information, motion data, prediction data, and quantized transformation coefficients from residual blocks. The quantized transformation coefficients are transferred to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0095] The reconstructed residual blocks and / or predicted blocks are transferred to the intra-image prediction component 417 for reconstruction into image blocks based on intra-predictive operation. The intra-image prediction component 417 may be similar to the intra-image estimation component 215 and the intra-image prediction component 217. Specifically, the intra-image prediction component 417 uses a prediction mode to locate the position of a reference block in the frame and applies the residual block to the result to reconstruct the intra-predicted image block. The reconstructed intra-predicted image block and / or residual block, along with the corresponding inter-predictive data, are transferred to the decoded image buffer component 423 via the in-loop filter component 425, which may be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image blocks, residual blocks, and / or predicted blocks, and such information is stored in the decoded image buffer component 423. The reconstructed image blocks from the decoded image buffer component 423 are transferred to the motion compensation component 421 for inter-prediction. The motion compensation component 421 may be substantially the same as the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 generates a predicted block using a motion vector from a reference block and reconstructs the image block by applying a residual block to the result. The resulting reconstructed block may also be transferred to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 continues to store additional reconstructed image blocks that can be reconstructed into frames via the division information. Such frames can also be placed in a sequence. The sequence is output to a display as a reconstructed output video signal.

[0096] Figure 5 is a schematic diagram showing an exemplary HRD500. The HRD500 can be employed in an encoder such as the codec system 200 and / or encoder 300. The HRD500 can check the bitstream created in step 109 of method 100 before the bitstream is transferred to a decoder such as the decoder 400. In some examples, the bitstream may be transferred sequentially through the HRD500 as the bitstream is encoded. If a portion of the bitstream does not conform to the associated constraints, the HRD500 can indicate such failure to the encoder, causing the encoder to re-encode the corresponding section of the bitstream using a different mechanism.

[0097] The HRD500 includes a Virtual Stream Scheduler (HSS) 541. The HSS 541 is a component configured to run a virtual distribution mechanism. The virtual distribution mechanism is used to check the conformity of a bitstream or decoder with respect to the timing and data flow of the bitstream 551 input to the HRD500. For example, the HSS 541 can receive the bitstream 551 output from an encoder and manage the conformity testing process for the bitstream 551. In a specific example, the HSS 541 can control the rate at which an encoded image moves through the HRD500 and verify that the bitstream 551 does not contain any non-conforming data.

[0098] HSS541 can transfer bitstream 551 to CPB543 at a predetermined rate. HRD500 can manage data in decoding unit (DU) 553. DU553 is an AU or a subset of AUs, and associated non-video coding layer (VCL) network abstraction layer (NAL) units. Specifically, an AU contains one or more images associated with output time. For example, an AU can contain a single image in a single-layer bitstream, or an image for each layer in a multi-layer bitstream. Each image in an AU can be divided into slices, each contained in a corresponding VCL NAL unit. Thus, DU553 can contain one or more images, one or more slices of images, or a combination thereof. Parameters used to decode the AU, images, and / or slices can also be contained in non-VCL NAL units. Therefore, DU553 contains non-VCL NAL units containing the data necessary to support the decoding of VCL NAL units in DU553. CPB543 is a first-in, first-out buffer in HRD500. The CPB543 contains DU553s with video data in the decoded order. The CPB543 stores the video data for use during bitstream conformance verification.

[0099] CPB543 transfers DU553 to the decoding process component 545. The decoding process component 545 is a component compliant with the VVC standard. For example, the decoding process component 545 can emulate a decoder 400 used by an end user. The decoding process component 545 decodes DU553 at a rate achievable by an exemplary end-user decoder. If the decoding process component 545 cannot decode DU553 fast enough to prevent an overflow in CPB543, the bitstream 551 is non-compliant with the standard and needs to be re-encoded.

[0100] The decoding process component 545 decodes DU553 and produces the decoded DU555. The decoded DU555 contains the decoded image. The decoded DU555 is transferred to DPB547. DPB547 may be substantially similar to the decoded image buffer components 223, 323, and / or 423. To support interpretation, an image marked to be used as a reference image 556 obtained from the decoded DU555 is returned to the decoding process component 545 to support further decoding. DPB547 outputs the decoded video sequence as a series of images 557. Images 557 are reconstructed images that are a mirror image of the images encoded by the encoder into the bitstream 551.

[0101] Image 557 is transferred to the output cropping component 549. The output cropping component 549 is configured to apply a suitable cropping window to image 557. This yields the cropped output image 559. The cropped output image 559 is a fully reconstructed image. Thus, the cropped output image 559 mimics what the end user would expect to see when decoding the bitstream 551. In this way, the encoder can review the cropped output image 559 to ensure that the encoding is satisfactory.

[0102] The HRD500 is initialized based on HRD parameters in the bitstream 551. For example, the HRD500 can read HRD parameters from VPS, SPS, and / or SEI messages. The HRD500 can then perform conformity testing operations on the bitstream 551 based on the information in such HRD parameters. Specifically, the HRD500 can determine one or more CPB delivery schedules 561 from the HRD parameters. The delivery schedules specify the timing for the delivery of video data to and from memory locations such as CPB and / or DPB. Thus, the CPB delivery schedule 561 specifies the timing for the delivery of AU, DU553, and / or images to and from the CPB543. For example, the CPB delivery schedule 561 can describe the bitrate and buffer size of the CPB543, such bitrate and buffer size corresponding to a particular class of decoder and / or network conditions. Thus, the CPB delivery schedule 561 can indicate how long data can remain in the CPB543 before evacuation. The inability of the HRD500 to maintain the CPB distribution schedule 561 during conformance testing indicates that the decoder corresponding to the CPB distribution schedule 561 is unable to decode the corresponding bitstream. Note that the HRD500 can use a DPB distribution schedule similar to the CPB distribution schedule 561 for the DPB547.

[0103] Video may be coded into different layers and / or OLS for use by decoders with varying levels of hardware capabilities and for various network conditions. The CPB distribution schedule 561 is selected to reflect these challenges. Thus, the sub-bitstreams of the upper layers are specified for optimal hardware and network conditions, and the upper layers can receive one or more CPB distribution schedules 561 using a large amount of memory in the CPB 543 and a short delay for the transfer of the DU 553 toward the DPB 547. Similarly, the sub-bitstreams of the lower layers are specified for limited decoder hardware capabilities and / or poor network conditions. Thus, the lower layers can receive one or more CPB distribution schedules 561 using a small amount of memory in the CPB 543 and a longer delay for the transfer of the DU 553 toward the DPB 547. The OLS, layers, sublayers, or combinations thereof can then be tested according to the corresponding distribution schedule 561 to ensure that the resulting sub-bitstreams can be correctly decoded under the conditions expected for the sub-bitstreams. Each CPB delivery schedule 561 is associated with a schedule index (ScIdx) 563. ScIdx 563 is an index that identifies the delivery schedule. Therefore, the HRD parameters in bitstream 551 can not only indicate the CPB delivery schedule 561 by ScIdx 563, but can also contain enough data to enable HRD 500 to determine the CPB delivery schedule 561 and correlate it with the corresponding OLS, layer, and / or sublayer.

[0104] Figure 6 is a schematic diagram showing an exemplary multilayer video sequence 600 configured for interlayer prediction 621. The multilayer video sequence 600 may be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Furthermore, the multilayer video sequence 600 can be checked for compliance with standards by an HRD such as HRD 500. The multilayer video sequence 600 is included to illustrate an exemplary application of layers in an encoded video sequence. The multilayer video sequence 600 is an arbitrary video sequence using multiple layers such as layer N631 and layer N+1632.

[0105] In one example, a multilayer video sequence 600 may employ interlayer prediction 621. Interlayer prediction 621 is applied between images 611, 612, 613, and 614 of different layers and images 615, 616, 617, and 618. In the illustrated example, images 611, 612, 613, and 614 are part of layer N+1 632, and images 615, 616, 617, and 618 are part of layer N631. Layers such as layer N631 and / or layer N+1 632 are groups of images all associated with similar characteristics such as similar size, quality, resolution, signal-to-noise ratio, and capability. A layer can be formally defined as a set of VCL NAL units and associated non-VCL NAL units. A VCL NAL unit is a NAL unit coded to contain video data, such as an encoded slice of an image. A non-VCL NAL unit is a NAL unit that contains non-video data such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations.

[0106] In the illustrated example, layer N+1 632 is associated with a larger image size than layer N631. Therefore, in this example, images 611, 612, 613, and 614 of layer N+1 632 have a larger image size (e.g., greater height and width, and therefore more samples) than images 615, 616, 617, and 618 of layer N631. However, such images can be separated between layer N+1 632 and layer N631 by other characteristics. Although only two layers, layer N+1 632 and layer N631, are shown, a set of images can be separated into any number of layers based on associated characteristics. Layers N+1 632 and layer N631 may also be represented by layer IDs. A layer ID is a data item associated with an image that indicates that the image is part of a specified layer. Therefore, each image 611-618 can be associated with a corresponding layer ID to indicate which layer N+1 632 or layer N631 contains the corresponding image. For example, a layer ID can include a NAL unit header layer identifier (nuh_layer_id), which is a syntax element that specifies the identifier of a layer containing NAL units (e.g., slices and / or parameters of the image within the layer). Layers associated with lower quality / bitstream sizes, such as layer N631, are generally assigned lower layer IDs and are called lower layers. Furthermore, layers associated with higher quality / bitstream sizes, such as layer N+1 632, are generally assigned higher layer IDs and are called upper layers.

[0107] Images 611-618 from different layers 631-632 are configured to be displayed alternately. Therefore, images from different layers 631-632 can share a time ID 622, as long as these images are included in the same AU. A time ID 622 is a data element that indicates that the data corresponds to a time position in the video sequence. An AU is a set of NAL units associated with one particular output time, related to each other according to a given classification rule. For example, an AU can contain one or more images in different layers, such as images 611 and 615, if such images are associated with the same time ID 622. Specifically, the decoder can decode and display image 615 at the current display time if a smaller image is desired, or the decoder can decode and display image 611 at the current display time if a larger image is desired. Thus, images 611-614 from the upper layer N+1 632 contain substantially the same image data as the corresponding images 615-618 from the lower layer N631 (despite the difference in image size). Specifically, image 611 contains substantially the same image data as image 615, image 612 contains substantially the same image data as image 616, and so on.

[0108] Images 611-618 can be coded by referencing other images 611-618 from the same layer N631 or N+1 632. Coding an image by referencing another image from the same layer results in an interpretation 623. Interpretations 623 are depicted by solid arrows. For example, image 613 may be coded by employing an interpretation 623 that references one or two of images 611, 612, and / or 614 from layer N+1 632, one image referenced for unidirectional interpretation and / or two images referenced for bidirectional interpretation. Furthermore, image 617 may be coded by employing an interpretation 623 that references one or two of images 615, 616, and / or 618 from layer N531, one image referenced for unidirectional interpretation and / or two images referenced for bidirectional interpretation. When performing interprediction 623, if an image is used as a reference to another image on the same layer, that image is sometimes called a reference image. For example, image 612 may be a reference image used to encode image 613 according to interprediction 623. Interprediction 623 is sometimes called intralayer prediction in a multilayer context. Therefore, interprediction 623 is a mechanism for encoding a sample of the current image by referencing a specified sample in a reference image that is different from the current image, and the reference image and the current image are on the same layer.

[0109] Images 611-618 can also be coded by referencing other images 611-618 in different layers. This process is known as inter-layer prediction 621 and is illustrated by a dashed arrow. Inter-layer prediction 621 is a mechanism for coding a sample of the current image by referencing an indicated sample in a reference image, where the current image and the reference image are in different layers and therefore have different layer IDs. For example, an image in lower layer N631 can be used as a reference image to code the corresponding image in upper layer N+1632. In concrete terms, image 611 can be coded by referencing image 615 according to inter-layer prediction 621. In such a case, image 615 is used as the inter-layer reference image. The inter-layer reference image is the reference image used in inter-layer prediction 621. In most cases, inter-layer prediction 621 is constrained so that the current image, such as image 611, can only use inter-layer reference images in lower layers, such as image 615, that are contained within the same AU. If multiple layers (e.g., three or more) are available, the inter-layer prediction 621 can encode / decode the current image based on multiple inter-layer reference images at lower levels than the current image.

[0110] The video encoder can encode images 611-618 using a multilayer video sequence 600 via many different combinations and / or permutations of inter-prediction 623 and inter-layer prediction 621. For example, image 615 may be coded according to intra-prediction. Then images 616-618 can be coded according to inter-prediction 623 by using image 615 as a reference image. Furthermore, image 611 may be coded according to inter-layer prediction 621 by using image 615 as an inter-layer reference image. Then images 612-614 can be coded according to inter-prediction 623 by using image 611 as a reference image. Thus, the reference image can play the role of both a single-layer reference image and an inter-layer reference image for different coding mechanisms. By coding the images of upper layers N+1 632 based on the images of lower layers N 631, upper layers N+1 632 can avoid employing intra-prediction, which is far less coding efficient than inter-prediction 623 and inter-layer prediction 621. Therefore, the low coding efficiency of intra prediction can be limited to minimum / lowest quality images, and thus limited to encoding the minimum amount of video data. Images used as reference images and / or inter-layer reference images can be indicated by entries in the reference image list contained in the reference image list structure.

[0111] To perform this operation, layers such as layer N631 and layer N+1 632 may be included in one or more OLS625 and 626. Specifically, images 611-618 are encoded as layers 631-632 of bitstream 600, and then each layer 631-632 of the images is assigned to one or more of OLS625 and 626. Then, OLS625 and / or 626 can be selected, and depending on the capabilities of the decoder and / or network conditions, the corresponding layers 631 and / or 632 can be sent to the decoder. OLS625 is a set of layers in which one or more layers are designated as output layers. Output layers are layers designated to be output (e.g., to a display). For example, layer N631 may be included only to support inter-layer prediction 621 and may never be output. In such a case, layer N+1 632 is decoded based on layer N631 and output. In such cases, OLS625 includes layer N+1 632 as output layers. When an OLS contains only output layers, it is called the 0th OLS626. The 0th OLS626 is an OLS that contains only the lowest layer (the layer having the lowest layer identifier), and therefore an OLS that contains only output layers. In other cases, OLS625 may contain many layers in different combinations. For example, the output layers of OLS625 can be coded according to inter-layer prediction 621 based on one, two, or many lower layers. Furthermore, OLS625 can contain two or more output layers. Thus, OLS625 can contain one or more output layers and any support layers necessary to reconstruct the output layers. Although only two OLS625 and 626 are shown, a multi-layer video sequence 600 may be coded by employing many different OLS625 and / or 626, each employing a different combination of layers. OLS625 and 626 are each associated with OLS index 629, which is an index that uniquely identifies the corresponding OLS625 and 626.

[0112] Checking the compliance of a multilayer video sequence 600 with the HRD500 can be complex depending on the number of layers 631-632 and OLS 625 and 626. The HRD500 can separate the multilayer video sequence 600 into a series of operating points 627 for testing. OLS 625 and / or 626 are identified by OLS index 629. An operating point 627 is a temporal subset of OLS 625 / 626. An operating point 627 can be identified by both the OLS index 629 and the highest time ID 622 of the corresponding OLS 625 / 626. For example, a first operating point 627 may contain all images in the first OLS 625 from time ID 0 to time ID 200, a second operating point 627 may contain all images in the first OLS 625 from time ID 201 to time ID 400, and so on. In this case, the first operating point 627 is described by the OLS index 629 of the first OLS 625 and a time ID of 200. Furthermore, the second operating point 627 is described by the OLS index 629 of the first OLS 625 and a time ID of 400. The operating point 627 selected to be tested at a specified moment is called the target Op. Thus, the target Op is the operating point 627 selected for conformity testing in HRD500.

[0113] Figure 7 is a schematic diagram showing an exemplary multilayer video sequence 700 configured for time scalability. The multilayer video sequence 700 can be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Furthermore, the multilayer video sequence 700 can be checked for compliance with standards by an HRD such as HRD 500. The multilayer video sequence 700 is included to demonstrate another exemplary application for layers in coded video sequences. For example, the multilayer video sequence 700 may be employed as a separate embodiment or in combination with the techniques described with respect to the multilayer video sequence 600.

[0114] The multilayer video sequence 700 includes sublayers 710, 720, and 730. A sublayer is a time-scalable layer of a time-scalable bitstream containing VCL NAL units (e.g., images) with specific time identifier values, as well as associated non-VCL NAL units (e.g., support parameters). For example, layers such as layer N631 and / or layer N+1632 may be further divided into sublayers 710, 720, and 730 to support time scalability. Sublayer 710 may be called the base layer, and sublayers 720 and 730 may be called enhancement layers. As illustrated, sublayer 710 contains images 711 at a first frame rate, such as 30 frames per second. Since sublayer 710 contains the base / minimum frame rate, sublayer 710 is the base layer. Sublayer 720 contains images 721 that are time-offset from images 711 of sublayer 710. As a result, sublayers 710 and 720 can be combined, resulting in a higher overall frame rate than that of sublayer 710 alone. For example, sublayers 710 and 720 together can have a frame rate of 60 frames per second. Thus, sublayer 720 improves the frame rate of sublayer 710. Furthermore, sublayer 730 includes image 731 which is temporally offset from images 721 and 711 of sublayers 720 and 710. Thus, sublayer 730 can be combined with sublayers 720 and 710 to further enhance sublayer 710. For example, sublayers 710, 720, and 730 together can have a frame rate of 90 frames per second.

[0115] A sublayer representation 740 can be dynamically created by combining sublayers 710, 720, and / or 730. A sublayer representation 740 is a subset of a bitstream containing NAL units of a particular sublayer and its lower sublayers. In the illustrated example, the sublayer representation 740 includes image 741, which is a composite image 711, 721, and 731 of sublayers 710, 720, and 730. Thus, a multilayer video sequence 700 can be temporally scaled to a desired frame rate by selecting a sublayer representation 740 containing a desired set of sublayers 710, 720, and / or 730. The sublayer representation 740 may also be created by employing an OLS that includes sublayers 710, 720, and / or 730 as layers. In such a case, the sublayer representation 740 is selected as the output layer. Therefore, temporal scalability is one of several mechanisms that can be achieved using the multilayer mechanism.

[0116] Figure 8 is a schematic diagram showing an exemplary bitstream 800. For example, bitstream 800 can be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, bitstream 800 may include multilayer video sequences 600 and / or 700. In addition, bitstream 800 may include various parameters for controlling the operation of an HRD, such as HRD500. Based on such parameters, the HRD can check bitstream 800 for compliance with standards before sending it to the decoder for decoding.

[0117] Bitstream 800 includes a VPS811, one or more SPS813s, multiple Image Parameter Sets (PPS)815s, multiple slice headers817s, image data820, and SEI messages819. The VPS811 contains data relevant to the entire bitstream 800. For example, the VPS811 may include data related to the OLS, layers, and / or sublayers used in the bitstream 800. The SPS813 contains sequence data common to all images within the coded video sequences included in the bitstream 800. For example, each layer may contain one or more coded video sequences, and each coded video sequence may reference the SPS813 for its corresponding parameters. Parameters in the SPS813 may include image sizing, bit depth, coding tool parameters, bitrate limits, etc. While each sequence references the SPS813, note that in some examples, a single SPS813 may contain data for multiple sequences. The PPS815 contains parameters applicable to the entire image. Therefore, each image in a video sequence can reference a PPS815. While each image references a PPS815, it should be noted that in some examples, a single PPS815 can contain data for multiple images. For example, multiple similar images may be coded according to similar parameters. In such cases, a single PPS815 can contain data for such similar images. The PPS815 can indicate the coding tools, quantization parameters, offsets, etc., available for the corresponding image slices.

[0118] The slice header 817 contains parameters specific to each slice in the image. Therefore, there may be one slice header 817 for each slice in a video sequence. The slice header 817 can include slice type information, POC, reference image list, prediction weights, tile entry points, deblocking parameters, etc. It should be noted that in some examples, the bitstream 800 may also include an image header, which is a syntactic structure containing parameters applicable to all slices in a single image. For this reason, the image header and slice header 817 may be used interchangeably in some contexts. For example, certain parameters may be moved between the slice header 817 and the image header depending on whether such parameters are common to all slices in the image.

[0119] Image data 820 includes video data encoded according to inter-prediction and / or intra-prediction, as well as corresponding transformed and quantized residual data. For example, image data 820 may include AU821, DU822, and / or image 823. AU821 is a set of NAL units related to one particular output time, associated with one another according to a given classification rule. DU822 is an AU or a subset of an AU, and associated non-VCL NAL units. Image 823 is an array of lumens and / or chromens that make up a frame or its fields. Simply put, AU821 includes various video data that may appear at a given moment in a video sequence, as well as supporting syntax data. Thus, AU821 may include a single image 823 in a single-layer bitstream, or multiple images from multiple layers, all associated at the same moment in a multi-layer bitstream. Image 823, on the other hand, is a coded image that may be output for display or used to support the encoding of other images 823 for output. DU822 can contain one or more images 823 and any supporting syntax data necessary for decoding. For example, DU822 and AU821 may be used interchangeably in a simple bitstream (for example, if the AU contains a single image). However, in more complex multi-layer bitstreams, DU822 may contain only a portion of the video data from AU821. For example, AU821 may contain images 823 in several layers and / or sublayers, some of which are associated with different OLSs. In such cases, DU822 may contain only images 823 from the specified OLS and / or specified layers / sublayers.

[0120] Image 823 contains one or more slices 825. A slice 825 may be defined as an integer number of complete tiles (e.g., within a tile) or an integer number of consecutive complete coding tree unit (CTU) rows in Image 823, where a tile or CTU row is exclusively contained within a single NAL unit 829. Thus, a slice 825 is also contained within a single NAL unit 829. A slice 825 is further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a group of samples of a predefined size that can be divided by a coding tree. A CTB is a subset of a CTU and contains the luminous or chroma components of the CTU. A CTU / CTB is further divided into coding blocks based on the coding tree. The coding blocks can then be coded / decoded according to a prediction mechanism.

[0121] Bitstream 800 is a sequence of NAL units 829. A NAL unit 829 is a container for video data and / or supporting syntax. A NAL unit 829 can be a VCL NAL unit or a non-VCL NAL unit. A VCL NAL unit is a NAL unit 829 coded to contain video data, such as an encoded slice 825 and an associated slice header 817. A non-VCL NAL unit is a NAL unit 829 containing non-video data, such as syntax and / or parameters that support decoding the video data, performing conformance checks, or other operations. For example, a non-VCL NAL unit may contain VPS811, SPS813, PPS815, SEI messages 819, or other supporting syntax.

[0122] SEI message 819 is a semantically specified syntactic structure that conveys information not required by the decoding process to determine the sample values ​​of the decoded image. For example, an SEI message may include data to support the HRD process or other supporting data not directly related to decoding bitstream 800 in the decoder. SEI message 819 can include scalable nesting SEI messages and / or non-scalable nested SEI messages. A scalable nesting SEI message is a message containing multiple SEI messages corresponding to one or more OLS or one or more layers. A non-scalable nested SEI message is a message that is not nested and therefore contains a single SEI message. SEI message 819 may include a BP SEI message containing HRD parameters for initializing HRD to manage CPB. SEI message 819 may also include a PT SEI message containing HRD parameters for managing delivery information about AU821 in CPB and / or DPB. SEI message 819 may also include a DUI SEI message containing HRD parameters for managing distribution information about DU822 in the CPB and / or DPB.

[0123] Bitstream 800 contains a set of integer(i) HRD parameters 833, which are syntax elements that initialize and / or define the operating conditions of an HRD, such as HRD500. In some examples, the general_hrd_parameters syntax structure can contain HRD parameters 833 that apply to all OLS specified by VPS811. In one example, an encoder can encode a video sequence into layers. The encoder can then encode the HRD parameters 833 into a bitstream to properly configure the HRD to perform conformance checks. The HRD parameters 833 can also indicate to the decoder that the bitstream can be decoded according to the delivery schedule. The HRD parameters 833 may be included in VPS811 and / or SPS813. Additional parameters used to configure the HRD may also be included in SEI message 819.

[0124] As mentioned above, a video stream can contain many OLSs and many layers, such as OLS625, layer N631, layer N+1 632, sublayer 710, sublayer 720, and / or sublayer 730. Furthermore, some layers may be contained within multiple OLSs. Therefore, multilayer video sequences, such as multilayer video sequences 600 and / or 700, can become quite complex. As a result, the bitstream conformance checking process in HRD can become complicated. Some video coding systems use layer-specific HRD parameters 833 for each layer / sublayer. HRD reads the layer-specific HRD parameters 833 from bitstream 800 and then performs bitstream conformance testing layer by layer based on the HRD parameters 833. In some cases, several of the various layers / sublayers use the same HRD parameters 833. As a result, redundant HRD parameters 833 are encoded into bitstream 800, reducing coding efficiency. Furthermore, this method involves repeatedly obtaining redundant information from bitstream 800 for the HRD, wasting the encoder's memory and / or processor resources. Therefore, redundant HRD parameters 833 can waste the encoder and / or decoder's processor, memory, and / or network resources.

[0125] This disclosure includes a mechanism for reducing the redundancy of HRD parameters 833 for video using multiple layers. If the HRD parameters 833 are the same for all sublayers in a sublayer representation and / or OLS, the encoder can encode the HRD parameters 833 for the top layer. The encoder can also encode a sublayer_cpb_params_present_flag831, which is a syntax element that specifies whether the set of HRD parameters 833 (for example, for an OLS) contains the HRD parameters for a given sublayer / sublayer representation. Setting sublayer_cpb_params_present_flag831 to 0 can indicate that all lower layers should use the same HRD parameters as the top layer. Setting sublayer_cpb_params_present_flag831 to 1 can also indicate that each layer contains separate (e.g., different) HRD parameters 833. Therefore, by setting sublayer_cpb_params_present_flag831 to 0, it can be inferred that the HRD parameters 833 of lower layers are equal to the HRD parameters 833 of the top layer. Thus, the HRD parameters 833 of lower sublayers can be omitted from bitstream 800 if they are the same as the HRD parameters 833 of the top sublayer, in order to avoid redundant signaling. This mechanism reduces the size of bitstream 800. Therefore, this mechanism reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder. Furthermore, by reducing the number of HRD parameters 833, the set of signaled HRD parameters 833 can be read and adopted for the complete set of sublayers, thereby reducing resource usage during the HRD process in the encoder.

[0126] The top layer / sublayer is the layer with the highest value for its corresponding layer ID in the OLS and / or sublayer representation. For example, VPS811 may contain hrd_max_tid[i]832. hrd_max_tid[i]832 specifies the time ID of the highest sublayer representation in which HRD parameter 833 is included in the i-th set of OLS HRD parameters 833. Thus, HRD can read sublayer_cpb_params_present_flag831. HRD can then determine that HRD parameter 833 applies to the top layer / sublayer, as indicated by hrd_max_tid[i]832 in VPS811. HRD can also infer that the same HRD parameter 833 applies to all lower layers / sublayers whose ID is smaller than hrd_max_tid[i]832.

[0127] The method described above allows various redundant HRD parameters 833 to be omitted from the bitstream 800 of lower sublayers. Redundancy omission can be applied to several HRD parameters 833. Specifically, HRD parameters 833 may include fixed_pic_rate_general_flag835, sublayer_hrd_parameters837, and general_vcl_hrd_params_present_flag839, each of which can be inferred to apply to the top-level sublayer and similarly to lower sublayers. fixed_pic_rate_general_flag835 is a syntax element that specifies whether the temporal distance between HRD output times of consecutive images in output order is constrained by other HRD parameters 833. For example, fixed_pic_rate_general_flag835 can be set to 1 to indicate that such constraints apply, or to 0 to indicate that such constraints do not apply. sublayer_hrd_parameters837 is a syntax structure containing the HRD parameters of the corresponding sublayer, indicated by the sublayer ID. general_vcl_hrd_params_present_flag839 is a flag that specifies whether VCL HRD parameters exist in the general HRD parameter syntax structure. For example, setting general_vcl_hrd_params_present_flag839 to 1 can indicate that VCL HRD parameters related to a first type of fit point exist in the general HRD parameter syntax structure, or setting it to 0 can indicate that such VCL HRD parameters do not exist (e.g., a second type of fit point is adopted).

[0128] Hereinafter, the information described above will be explained in more detail below in this specification. Layered video coding is also called scalable video coding or video coding with scalability. Scalability in video coding can be supported by using multilayer coding techniques. A multilayer bitstream includes a base layer (BL) and one or more enhancement layers (EL). Examples of scalability include spatial scalability, signal-to-noise ratio (SNR) scalability, multiview scalability, and frame rate scalability. When using multilayer coding techniques, an image or part thereof may be coded without using a reference image (intra-prediction), coded by referencing a reference image in the same layer (inter-prediction), and / or coded by referencing a reference image in another layer (interlayer-prediction). The reference image used for interlayer prediction of the current image is called the interlayer reference image (ILRP). Figure 6 shows an example of multilayer coding for spatial scalability where images in different layers have different resolutions.

[0129] Several video coding families offer support for scalability of profiles separated from profiles for single-layer coding. Scalable Video Coding (SVC) is a scalable extension of Advanced Video Coding (AVC) that supports spatial, temporal, and quality scalability. In SVC, each macroblock (MB) of an EL image is signaled with a flag indicating whether the EL MB is predicted using a collocation block from a lower layer. Predictions from a collocation block can include texture, motion vector, and / or coding mode. An implementation of SVC cannot directly reuse an unmodified implementation of AVC in design. The syntax and decoding processes for SVC EL macroblocks differ from those for AVC.

[0130] Scalable HEVC (SHVC) is an extension of HEVC that supports spatial and quality scalability. Multiview HEVC (MV-HEVC) is an extension of HEVC that supports multiview scalability. 3D HEVC (3D-HEVC) is an extension of HEVC that supports more advanced and efficient 3D video coding than MV-HEVC. Temporal scalability may be included as an integral part of the single-layer HEVC codec. In the multi-layer extensions of HEVC, the decoded images used for inter-layer prediction are only those from the same AU and are treated as long-term reference images (LTRPs). Such images, along with other temporal reference images of the current layer, are assigned a reference index in the reference image list. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the reference index value to refer to the inter-layer reference image in the reference image list. Spatial scalability is achieved by resampling the reference image or a portion thereof if the ILRP has a different spatial resolution than the current image being encoded or decoded. Resampling of a reference image can be achieved at either the image level or the coding block level.

[0131] VVC can also support layered video coding. A VVC bitstream can contain multiple layers. All layers may be independent of each other. For example, each layer can be coded without using inter-layer prediction. In this case, the layer is also called a simulcast layer. In some cases, some of the layers are coded using ILP. A flag in the VPS can indicate whether a layer is a simulcast layer or whether some layers use ILP. If some layers use ILP, the layer dependencies between layers are also signaled in the VPS. Unlike SHVC and MV-HEVC, VVC may not specify an OLS. The OLS contains a specified set of layers, one or more of which are specified as output layers. The output layers are the layers of the output OLS. In some embodiments of VVC, if a layer is a simulcast layer, only one layer may be selected for decoding and output. In some embodiments of VVC, the entire bitstream containing all layers is specified to be decoded if any of the layers use ILP. In addition, a specific layer among the layers is specified as the output layer. The output layers may be instructed to be only the top layer, all layers, or the top layer plus a set of specified lower layers.

[0132] Video coding standards can specify HRDs for verifying bitstream conformance through designated HRD conformance tests. SHVC and MV-HEVC employ three sets of bitstream conformance tests to check bitstream conformance. The bitstream is referred to as the entire bitstream and is denoted as entireBitstream. The first set of bitstream conformance tests is for testing the conformance of the entire bitstream and its corresponding time subset. Such tests are employed regardless of whether there is a layer set specified by the active VPS containing all nuh_layer_id values ​​for VCL NAL units present in the entire bitstream. Therefore, the entire bitstream is always checked for conformance, even if one or more layers are not included in the output set. The second set of bitstream conformance tests is employed to test the conformance of the layer set and associated time subset specified by the active VPS. For all of these tests, only the base layer image (e.g., an image where nuh_layer_id is equal to 0) is decoded and output. Other images are ignored by the decoder when the decoding process is invoked. A third set of bitstream compliance tests is employed to test the compliance of the OLS and associated time subsets specified by the VPS extension portion of the active VPS, based on the OLS and bitstream partitioning. Bitstream partitioning includes one or more layers of the OLS of a multilayer bitstream.

[0133] The aforementioned embodiments involve certain problems. For example, the first two sets of conformance testing may apply to layers that are not decoded and are not output. For instance, layers other than the lowest layer are not decoded and are not output. In a real application, the decoder can only receive data that is decoded. Therefore, using the first two sets of conformance testing both complicates the codec design and can waste bits for carrying both sequence-level and image-level parameters used to support conformance testing. A third set of conformance testing involves bitstream splitting. Such splitting may relate to one or more layers of the OLS of a multilayer bitstream. Alternatively, if conformance testing always operates separately for each layer, the HRD can be greatly simplified.

[0134] Signaling of sequence-level HRD parameters can be complex. For example, sequence-level HRD parameters may be signaled in multiple locations, such as both SPS and VPS. Furthermore, sequence-level HRD parameter signaling can involve redundancy. For instance, information that might be generally the same across the entire bitstream may be repeated at each layer of each OLS. Additionally, exemplary HRD schemes allow for the selection of different delivery schedules per layer. Such delivery schedules may be selected from a list of signaled schedules for each layer at each operating point, where the operating point is an OLS or a time subset of an OLS. Such systems are complex. Furthermore, exemplary HRD schemes allow for the association of incomplete AUs with buffered-period SEI messages. An incomplete AU is an AU that lacks an image for all layers present in the CVS. However, HRD initialization in such AUs can be problematic. For example, HRD may not be properly initialized for layers that have layer access units that are not present in the incomplete AU. In addition, the demultiplexing process for deriving layer bitstreams may not be sufficiently efficient at removing nested SEI messages that do not apply to the target layer. Layer bitstreams occur when the bitstream splitting contains only one layer. Furthermore, the applicable OLS for non-scalable nested buffering periods, image timings, and decoding unit information SEI messages may be specified for the entire bitstream. However, the non-scalable nested buffering period should instead be applicable to the 0th OLS.

[0135] Furthermore, depending on the VVC implementation, it may fail to infer HDR parameters when sub_layer_cpb_params_present_flag is equal to 0. Such inference may enable proper HRD operation. In addition, the values ​​of bp_max_sub_layers_minus1 and pt_max_sub_layers_minus1 may need to be equal to the value of sps_max_sub_layers_minus1. However, buffering period and image timing SEI messages may be nested and applicable to multiple OLSs and multiple layers of each of multiple OLSs. In such contexts, the layers involved may refer to multiple SPSs. Therefore, it may be difficult for the system to track which SPS corresponds to each layer. Thus, the values ​​of these two syntax elements should instead be constrained based on the value of vps_max_sub_layers_minus1. Furthermore, since different layers can have different numbers of sublayers, the values ​​of these two syntax elements are not always equal to a specific value in all buffering period and image timing SEI messages.

[0136] Furthermore, the following issues are associated with HRD design in both SHVC / MV-HEVC and VVC: The subbitstream extraction process may not remove SEI NAL units containing nested SEI messages that are not required for the target OLS.

[0137] Generally, this disclosure describes a technique for scalable nesting of SEI messages for output layer sets in a multilayer video bitstream. The description of this technique is based on VVC. However, this technique is also applicable to layered video coding based on other video codec specifications.

[0138] One or more of the above-mentioned problems can be solved as follows. Specifically, this disclosure includes a method and related embodiments for HRD design that enables efficient signaling of HRD parameters using HRD operation that is much simpler compared to SHVC and MV-HEVC. Each of the solutions described below addresses the above-mentioned problems. For example, instead of requiring three sets of conformance tests, this disclosure requires only one set of conformance tests to test the conformance of an OLS specified by the VPS. Furthermore, instead of a bitstream partitioning-based design, the disclosed HRD mechanism can always operate separately for each layer of the OLS. In addition, sequence-level HRD parameters that are global for all layers and sublayers of all OLSs only need to be signaled once in the VPS, for example. In addition, a single number of delivery schedules need to be signaled for all layers and sublayers of all OLSs. The same delivery schedule index can also be applied to all layers of the OLS. Furthermore, incomplete AUs do not need to be associated with buffering period SEI messages. An incomplete AU is an AU that does not contain images for all layers present in the CVS. This ensures that the HRD is always properly initialized for all layers of the OLS. Furthermore, a mechanism is disclosed for efficiently removing nested SEI messages that are not applicable to the target layer in the OLS. This supports a demultiplexing process for deriving the layer bitstream. In addition, the OLS to which non-scalable nested buffering periods, image timings, and decoded unit information SEI messages are applicable may be specified as the 0th OLS. Furthermore, if sub_layer_cpb_params_present_flag is equal to 0, HDR parameters can be inferred, potentially enabling proper HRD operation. The values ​​of bp_max_sub_layers_minus1 and pt_max_sub_layers_minus1 may need to be in the range of 0 to vps_max_sub_layers_minus1.Thus, these parameters do not need to be specific values ​​for all buffering periods and image timing SEI messages. Furthermore, the sub-bitstream extraction process can remove SEI NAL units containing nested SEI messages that are not applicable to the target OLS.

[0139] An exemplary embodiment of the aforementioned mechanism is as follows: The output layer is the output layer from the output layer set. The OLS is a set of layers containing a specified set of layers, one or more layers in the set of layers are designated as output layers. The OLS layer index is the index of a layer in the OLS to the list of layers in the OLS. The sub-bitstream extraction process is a specified process in which NAL units in a bitstream that do not belong to the target set, determined by the target OLS index and the target highest TemporalId, are removed from the bitstream, and the output sub-bitstream contains NAL units in the bitstream that belong to the target set.

[0140] The syntax for an example video parameter set is as follows: [Table 1]

[0141] The syntax for an exemplary sequence parameter set RBSP is as follows: [Table 2]

[0142] The syntax for an example DPB parameter is as follows: [Table 3]

[0143] The syntax for an example of a general HRD parameter is as follows: [Table 4]

[0144] The semantics of the exemplary video parameter set RBSP are as follows: Setting each_layer_is_an_ols_flag to equal 1 specifies that each output layer set contains only one layer, and each layer in the bitstream is itself an output layer set, with the single layer it contains being the only output layer. Setting each_layer_is_an_ols_flag to equal 0 specifies that an output layer set can contain two or more layers. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 1. Otherwise, if vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 0.

[0145] Setting ols_mode_idc to 0 specifies that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including both endpoints), and for each OLS, only the top layer of the OLS is output. Setting ols_mode_idc to 1 specifies that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including both endpoints), and for each OLS, all layers of the OLS are output. Setting ols_mode_idc to 2 specifies that the total number of OLS specified by the VPS is explicitly signaled, and for each OLS, the top layer of the OLS and a set of explicitly signaled lower layers are output. The value of ols_mode_idc should be in the range of 0 to 2 (including both endpoints). The value of ols_mode_idc 3 is reserved. If vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, then the value of ols_mode_idc is presumed to be equal to 2. num_output_layer_sets_minus1+1 specifies the total number of OLS specified by the VPS, assuming ols_mode_idc is equal to 2.

[0146] The variable TotalNumOlss, which specifies the total number of OLS designated by the VPS, is derived as follows:

number

[0147] The `layer_included_flag[i][j]` specifies whether to include the j-th layer (the layer whose `nuh_layer_id` is equal to `vps_layer_id[j]`) in the i-th OLS when `ols_mode_idc` is equal to 2. Setting `layer_included_flag[i][j]` to equal 1 specifies that the j-th layer should be included in the i-th OLS. Setting `layer_included_flag[i][j]` to equal 0 specifies that the j-th layer should not be included in the i-th OLS.

[0148] The variables NumLayersInOls[i], which specify the number of layers in the i-th OLS, and LayerIdInOls[i][j], which specify the nuh_layer_id value of the j-th layer of the i-th OLS, are derived as follows:

number

[0149] The variable LayerIdInOls[i][j], which specifies the OLS layer index of the layer where nuh_layer_id is equal to OlsLayeIdx[i][j], is derived as follows:

number

[0150] The lowest layer in each OLS is assumed to be an independent layer. In other words, for each i in the range from 0 to TotalNumOlss-1, including both endpoints, the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] is equal to 1. Each layer is assumed to be contained in at least one OLS specified by the VPS. In other words, for each k in the range from 0 to vps_max_layers_minus1, including both endpoints, for each layer where a particular value of nuh_layer_id nuhLayerId is equal to one of vps_layer_id[k], there may be at least one pair of i and j values, where i is in the range from 0 to TotalNumOlss-1, including both endpoints, j is in the range up to NumLayersInOls[i]-1, including both endpoints, and the value of LayerIdInOls[i][j] is equal to nuhLayerId. Any layer in OLS shall be an output layer of OLS, or a (direct or indirect) reference layer of an output layer of OLS.

[0151] The `vps_output_layer_flag[i][j]` specifies whether the j-th layer of the i-th OLS is output when `ols_mode_idc` is equal to 2. `vps_output_layer_flag[i]` equal to 1 specifies that the j-th layer of the i-th OLS is output. Setting `vps_output_layer_flag[i]` equal to 0 specifies that the j-th layer of the i-th OLS is not output. When `vps_all_independent_layers_flag` is equal to 1 and `each_layer_is_an_ols_flag` is equal to 0, the value of `vps_output_layer_flag[i]` is presumed to be equal to 1. The variable `OutputLayerFlag[i][j]`, which specifies that the j-th layer of the i-th OLS is output when valued at 1 and not output when valued at 0, is derived as follows:

number

[0152] Setting vps_extension_flag to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS RBSP syntax structure. Setting vps_extension_flag to 1 specifies that the vps_extension_data_flag syntax element exists in the VPS RBSP syntax structure. vps_extension_data_flag can have any value. The presence and value of vps_extension_data_flag do not affect the decoder's suitability for the specified profile. The decoder shall ignore all vps_extension_data_flag syntax elements.

[0153] The semantics of an example DPB parameter are as follows: The dpb_parameters() syntax structure provides DPB size information and, optionally, maximum image sort count and maximum latency (MRML) information. Each SPS contains one or more dpb_parameters() syntax structures. The first dpb_parameters() syntax structure of an SPS contains both DPB size information and MRML information. If present, the second dpb_parameters() syntax structure of an SPS contains only DPB size information. The MRML information of the first dpb_parameters() syntax structure of an SPS applies to the layer referencing the SPS, regardless of whether that layer is an OLS output layer. The DPB size information of the first dpb_parameters() syntax structure of an SPS applies to the layer referencing the SPS if that layer is an OLS output layer. The DPB size information contained in the second dpb_parameters() syntax structure of the SPS, if present, applies to the layer referencing the SPS if that layer is a non-output layer of the OLS. If the SPS contains only one dpb_parameters() syntax structure, it is inferred that the DPB size information for the layer as a non-output layer is the same as that for the layer as an output layer.

[0154] The semantics of exemplary general HRD parameters are as follows: The general_hrd_parameters() syntax structure provides the HRD parameters used in HRD operation. Setting sub_layer_cpb_params_present_flag equal to 1 specifies that the i-th layer_level_hrd_parameters() syntax structure contains HRD parameters for sublayer representations where TemporalId ranges from 0 to hrd_max_temporal_id[i], including both endpoints. Setting sub_layer_cpb_params_present_flag equal to 0 specifies that the i-th layer_level_hrd_parameters() syntax structure contains HRD parameters for sublayer representations where TemporalId is equal only to hrd_max_temporal_id[i]. If vps_max_sub_layers_minus1 is equal to 0, the value of sub_layer_cpb_params_present_flag is inferred to be equal to 0. If sub_layer_cpb_params_present_flag is equal to 0, the HRD parameters for sublayer representations where TemporalId is in the range of 0 to hrd_max_temporal_id[i]-1, including both endpoints, are presumed to be the same as those for sublayer representations where TemporalId is equal to hrd_max_temporal_id[i]. These include the HRD parameters in the layer_level_hrd_parameters syntax structure, starting from the fixed_pic_rate_general_flag[i] syntax element up to the sub_layer_hrd_parameters(i) syntax structure immediately below the conditional statement if(general_vcl_hrd_params_present_flag). num_layer_hrd_params_minus1+1 specifies the number of layer_level_hrd_parameters() syntax structures present in the general_hrd_parameters() syntax structure.The value of num_layer_hrd_params_minus1 shall be in the range of 0 to 63, including the endpoints. hrd_cpb_cnt_minus1+1 specifies the number of alternative CPB specifications in the CVS bitstream. The value of hrd_cpb_cnt_minus1 shall be in the range of 0 to 31, including the endpoints. hrd_max_temporal_id[i] specifies the TemporalId of the top-level sublayer representation in which the HRD parameters are included in the i-th layer_level_hrd_parameters() syntax structure. The value of hrd_max_temporal_id[i] shall be in the range of 0 to vps_max_sub_layers_minus1, including the endpoints. If vps_max_sub_layers_minus1 is equal to 0, the value of hrd_max_temporal_id[i] is presumed to be equal to 0. `layer_level_hrd_idx[i][j]` specifies the index of the `layer_level_hrd_parameters()` syntax structure applied to the j-th layer of the i-th OLS. The value of `layer_level_hrd_idx[[i][j]]`, including both endpoints, must be in the range of 0 to `num_layer_hrd_params_minus1`. If it does not exist, the value of `layer_level_hrd_idx[[0][0]` is presumed to be equal to 0.

[0155] An exemplary sub-bitstream extraction process is as follows: The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx, and the target highest TemporalId value tIdTarget. The output of this process is the sub-bitstream outBitstream. The bitstream conformance requirement for the input bitstream is that any output sub-bitstream is a conforming bitstream, which is the output of the process specified in this section with respect to the bitstream, with input targetOlsIdx equal to the index in the list of OLS specified by the VPS, and tIdTarget equal to any value in the range of 0 to 6, including the end values, and satisfies the following conditions: The output sub-bitstream should contain at least one VCL NAL unit where nuh_layer_id is equal to each of the nuh_layer_id values ​​in LayerIdInOls[targetOlsIdx]. The output sub-bitstream should contain at least one VCL NAL unit where TemporalId is equal to tIdTarget. A compatible bitstream contains one or more coded slice NAL units with a TemporalId equal to 0, but does not need to contain any coded slice NAL units with a nuh_layer_id equal to 0.

[0156] The output sub-bitstream OutBitstream is derived as follows: The bitstream outBitstream is set to be identical to the bitstream inBitstream. Remove all NAL units from outBitstream whose TemporalId is greater than tIdTarget. Remove all NAL units from outBitstream whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx]. Remove all SEI NAL units from outBitstream that contain scalable nesting SEI messages whose value of i is not in the range of 0 to nesting_num_olss_minus1, including both endpoints, such that nesting_ols_flag is equal to 1 and NestingOlsIdx[i] is equal to targetOlsIdx. If targetOlsIdx is greater than 0, remove all SEI NAL units from the outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information).

[0157] A typical example of an HRD is as follows. This section specifies HRDs and their use for checking bitstream and decoder conformance. A set of bitstream conformance tests is used to check the conformance of a bitstream called the entire bitstream, denoted as entireBitstream. A set of bitstream conformance tests is for testing the conformance of each OLS and time subsets of each OLS specified by the VPS. For each test, the following ordered steps are applied in the order listed.

[0158] The operating point under test, denoted as targetOp, is selected by selecting a target OLS having an OLS index opOlsIdx and the highest TemporalId value opTid. The value of opOlsIdx, including both endpoints, is in the range of 0 to TotalNumOlss-1. The value of opTid, including both endpoints, is in the range of 0 to vps_max_sub_layers_minus1. The values ​​of opOlsIdx and opTid are such that the sub-bitstream BitstreamToDecode, which is the output of calling the sub-bitstream extraction process with entireBitstream, opOlsIdx, and opTid as inputs, satisfies the following conditions: There is at least one VCL NAL unit where nuh_layer_id is equal to each of the nuh_layer_id values ​​of LayerIdInOls[opOlsIdx] in BitstreamToDecode. There is at least one VCL NAL unit where TemporalId is equal to opTid in BitstreamToDecode.

[0159] The values ​​of TargetOlsIdx and Htid are set to be equal to opOlsIdx and opTid of targetOp, respectively. The value of ScIdx is selected. The selected ScIdx shall be in the range of 0 to hrd_cpb_cnt_minus1, including both endpoints. The access unit in BitstreamToDecode associated with the buffering period SEI message applicable to TargetOlsIdx (present in TargetLayerBitstream or available through an external mechanism not specified herein) is selected as the HRD initialization point and is referred to as access unit 0 for each layer of the target OLS.

[0160] The subsequent steps apply to each layer in the target OLS that has an OLS layer index, TargetOlsLayerIdx. If the target OLS has only one layer, the layer bitstream under test, TargetLayerBitstream, is set to be identical to BitstreamToDecode. Otherwise, TargetLayerBitstream is derived by calling a demultiplex process to derive the layer bitstream, taking BitstreamToDecode, TargetOlsIdx, and TargetOlsLayerIdx as inputs, and the output is assigned to TargetLayerBitstream.

[0161] The `layer_level_hrd_parameters()` and `sub_layer_hrd_parameters()` syntax structures applicable to `TargetLayerBitstream` are selected as follows: The `layer_level_hrd_idx[TargetOlsIdx][TargetOlsLayerIdx]`-th `layer_level_hrd_parameters()` syntax structure within the VPS (or provided through an external mechanism such as user input) is selected. In the selected `layer_level_hrd_parameters()` syntax structure, if `BitstreamToDecode` is a type I bitstream, the `sub_layer_hrd_parameters(Htid)` syntax structure immediately following the condition `if(general_vcl_hrd_params_present_flag)` is selected, and the variable `NalHrdModeFlag` is set to equal 0. Otherwise (if BitstreamToDecode is a type II bitstream), the sub_layer_hrd_parameters(Htid) syntax structure immediately following either the condition if(general_vcl_hrd_params_present_flag) (in which case the variable NalHrdModeFlag is set to equal 0) or the condition if(general_nal_hrd_params_present_flag) (in which case the variable NalHrdModeFlag is set to equal 1) is selected. If BitstreamToDecode is a type II bitstream and NalHrdModeFlag is equal to 0, all non-VCL NAL units except filler data NAL units, as well as all leading_0_8bits, 0_byte, start_code_prefix_one_3bytes, and trailing_0_8 bit syntax elements that form a byte stream from the NAL unit stream, are discarded from TargetLayerBitstream if they exist, and the remaining bitstream is allocated to TargetLayerBitstream.

[0162] If decoding_unit_hrd_params_present_flag is equal to 1, the CPB is scheduled to operate either at the access unit level (in which case the variable DecodingUnitHrdFlag is set to equal to 0) or at the decoding unit level (in which case the variable DecodingUnitHrdFlag is set to equal to 1). Otherwise, DecodingUnitHrdFlag is set to equal to 0, and the CPB is scheduled to operate at the access unit level. For each access unit in TargetLayerBitstream starting from access unit 0, a buffering period SEI message associated with the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected; an image timing SEI message associated with the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected; and if DecodingUnitHrdFlag is equal to 1 and decoding_unit_cpb_params_in_pic_timing_sei_flag is equal to 0, a decoding unit information SEI message associated with the decoding unit in the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected.

[0163] Each conformance test includes a combination of one option in each of the steps described above. If there are two or more options for a given step, only one option is selected for any particular conformance test. All possible combinations of all steps form the entire set of conformance tests. For each operating point under test, the number of bitstream conformance tests performed is equal to n0*n1*n2*n3, where the values ​​of n0, n1, n2, and n3 are specified as follows: n1 is equal to hrd_cpb_cnt_minus1+1, where n1 is the number of access units in BitstreamToDecode associated with the buffering period SEI message, and n2 is derived as follows: n0 is equal to 1 if BitstreamToDecode is a type I bitstream; otherwise (if BitstreamToDecode is a type II bitstream), n0 is equal to 2. n3 is derived as follows: n3 is equal to 1 if decoding_unit_hrd_params_present_flag is equal to 0. Otherwise, n3 is equal to 2.

[0164] HRD includes a bitstream demultiplexer (optionally present), coded image buffers (CPBs) for each layer, instantaneous decoding processes for each layer, decoded image buffers (DPBs) containing sub-DPBs for each layer, and output cropping.

[0165] In one example, the HRD operates as follows: The HRD is initialized to zero in the decoding unit, and each CPB and each sub-DPB of the DPB is set to be empty. The sub-DPB fullness of each sub-DPB is set to equal to zero. After initialization, the HRD is not initialized again by subsequent buffering period SEI messages. Data associated with decoding units that have flowed into each CPB according to the specified arrival schedule is delivered by the HSS. The data associated with each decoding unit is removed by the instantaneous decoding process during the decoding unit's CPB removal time and decoded instantaneously. Each decoded image is placed in the DPB. Decoded images are removed from the DPB when they are no longer needed for interpredictive reference and no longer needed for output.

[0166] In one example, the demultiplex process for deriving the layer bitstream is as follows: The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the target OLS layer index targetOlsLayerIdx. The output of this process is the layer bitstream outBitstream. The output layer bitstream outBitstream is derived as follows: The bitstream outBitstream is set to be identical to the bitstream inBitstream. All NAL units whose nuh_layer_id is not equal to LayerIdInOls[targetOlsIdx][targetOlsLayerIdx] are removed from outBitstream. Remove all SEI NAL units containing scalable nesting SEI messages from the outBitstream, such that nesting_ols_flag is equal to 1, and the values ​​of i and j are not in the ranges from 0 to nesting_num_ols_minus1 (including both endpoints) and from 0 to nesting_num_olss_layers_minus1[i] (including both endpoints), such that NestingOlsLayerIdx[i][j] is equal to targetOlsLayerIdx. Remove all SEI NAL units containing scalable nesting SEI messages from the outBitstream, such that nesting_ols_flag is equal to 1, and the values ​​of i and j are in the ranges from 0 to nesting_num_olss_minus1 (including both endpoints) and from 0 to nesting_num_ols_layers_minus1[i] (including both endpoints), such that NestingOlsLayerIdx[i][j] is less than targetOlsLayerIdx.Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 0 and i has no value in the range from 0 to LayerIdInOls-1, including both endpoints, so that NestingLayerId[i] is equal to NestingNumLayers[targetOlsIdx][targetOlsLayerIdx]. Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 0 and i has at least one value in the range from 0 to LayerIdInOls-1, including both endpoints, so that NestingLayerId[i] is less than NestingNumLayers[targetOlsIdx][targetOlsLayerIdx].

[0167] An example of buffered-period SEI message syntax is as follows: [Table 5]

[0168] An example of scalable nesting SEI message syntax is as follows: [Table 6]

[0169] The following are exemplary general SEI payload semantics. The following applies to the applicable layers (in the context of OLS, or generally) of a non-scalable nested SEI message. For a non-scalable nested SEI message, if payloadType is equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information), the non-scalable nested SEI message applies only to the lowest layer in the context of the 0th OLS. For a non-scalable nested SEI message, if payloadType is equal to any value in VclAssociatedSeiList, the non-scalable nested SEI message applies only to layers where the nuh_layer_id of the VCL NAL unit is equal to the nuh_layer_id of the SEI NAL unit containing the SEI message.

[0170] The semantics of an exemplary buffering period SEI message are as follows: The buffering period SEI message provides initial CPB removal delay and initial CPB removal delay offset information for initializing the HRD at the position of the associated access unit in decoding order. If a buffering period SEI message is present, an image whose TemporalId is equal to 0 and is not a RASL or RADL (random access decodable leading) image is said to be a notDiscardablePic image. If the current image is not the first image in the bitstream in decoding order, prevNonDiscardablePic is said to be a preceding image in decoding order whose TemporalId is equal to 0 and is not a RASL or RADL image.

[0171] The presence of a buffering period SEI message is specified as follows: If NalHrdBpPresentFlag is equal to 1 or VclHrdBpPresentFlag is equal to 1, the following applies to each access unit in CVS: If the access unit is an IRAP or Gradual Decoder Refresh (GDR) access unit, the buffering period SEI message applicable to the operating point shall be associated with the access unit. Otherwise, if the access unit contains notDiscardablePic, the buffering period SEI message applicable to the operating point may or may not be associated with the access unit. Otherwise, the access unit shall not be associated with the buffering period SEI message applicable to the operating point. Otherwise (if both NalHrdBpPresentFlag and VclHrdBpPresentFlag are equal to 0), no access unit in CVS shall be associated with the buffering period SEI message. In some applications, it may be desirable to have buffered-period SEI messages frequently (for example, in the case of random access to IRAP or non-IRAP images, or in the case of bitstream splicing). If the image of an access unit is associated with a buffered-period SEI message, the access unit shall have an image for each layer present in the CVS, and each image of the access unit shall be associated with a buffered-period SEI message.

[0172] bp_max_sub_layers_minus1+1 specifies the maximum number of time sublayers for which the CPB removal delay and CPB removal offset are indicated in the buffering period SEI message. The value of bp_max_sub_layers_minus1, including both endpoints, should be in the range of 0 to vps_max_sub_layers_minus1. bp_cpb_cnt_minus1+1 specifies the number of syntax element pairs of nal_initial_cpb_removal_delay[i][j] and nal_initial_cpb_removal_offset[i][j] in the i-th time sublayer when bp_nal_hrd_params_present_flag is equal to 1, and the number of syntax element pairs of vcl_initial_cpb_removal_delay[i][j] and vcl_initial_cpb_removal_offset[i][j] in the i-th time sublayer when bp_vcl_hrd_params_present_flag is equal to 1. The value of bp_cpb_cnt_minus1 shall be in the range of 0 to 31, including the endpoints. The value of bp_cpb_cnt_minus1 shall be equal to the value of hrd_cpb_cnt_minus1.

[0173] The semantics of an exemplary image timing SEI message are as follows: An image timing SEI message provides CPB removal delay and DPB output delay information for the access unit associated with the SEI message. If the bp_nal_hrd_params_present_flag or bp_vcl_hrd_params_present_flag of a buffering period SEI message applicable to the current access unit is equal to 1, the variable CpbDpbDelaysPresentFlag is set to equal to 1. Otherwise, CpbDpbDelaysPresentFlag is set to equal to 0. The existence of an image timing SEI message is specified as follows: If CpbDpbDelaysPresentFlag is equal to 1, the image timing SEI message is associated with the current access unit. Otherwise (CpbDpbDelaysPresentFlag is equal to 0), there is no image timing SEI message associated with the current access unit. The TemporalId in the image timing SEI message syntax is the TemporalId of the SEI NAL unit containing the image timing SEI message. pt_max_sub_layers_minus1+1 specifies the TemporalId of the top-level sublayer representation in which CPB removal delay information is included in the image timing SEI message. The value of pt_max_sub_layers_minus1, including both endpoints, should be in the range of 0 to vps_max_sub_layers_minus1.

[0174] The semantics of an exemplary scalable nesting SEI message are as follows: A scalable nesting SEI message provides a mechanism for associating an SEI message with a specific layer within the context of a particular OLS, or a specific layer that is not within the context of an OLS. A scalable nesting SEI message contains one or more SEI messages. SEI messages contained within a scalable nesting SEI message are also called scalable nested SEI messages. The bitstream conformance requirement is that the following restrictions apply to the inclusion of SEI messages within a scalable nesting SEI message: SEI messages with payloadType equal to 132 (decrypted picture hash) or 133 (scalable nesting) shall not be included in a scalable nesting SEI message. If a scalable nesting SEI message includes a buffering period, image timing, or decoding unit information SEI message, the scalable nesting SEI message shall not contain any other SEI messages whose payloadType is not equal to 0 (buffering period), 1 (image timing), or 130 (decoding unit information).

[0175] The bitstream conformance requirement is that the following restrictions apply to the value of nal_unit_type for SEI NAL units containing scalable nesting SEI messages: If a scalable nesting SEI message contains an SEI message whose payloadType is equal to 0 (buffering period), 1 (image timing), 130 (decoded unit information), 145 (dependent RAP instruction), or 168 (frame field information), then the SEI NAL unit containing the scalable nesting SEI message shall have nal_unit_type equal to PREFIX_SEI_NUT. If a scalable nesting SEI message contains an SEI message whose payloadType is equal to 132 (decoded image hash), then the SEI NAL unit containing the scalable nesting SEI message shall have nal_unit_type equal to SUFFIX_SEI_NUT.

[0176] Setting nesting_ols_flag to 1 specifies that scalable nesting SEI messages apply to a specific layer in the context of a particular OLS. Setting nesting_ols_flag to 0 specifies that scalable nesting SEI messages apply to a specific layer in general (not in the context of an OLS). Bitstream compatibility requirements are that the following restrictions apply to the value of nesting_ols_flag: If a scalable nesting SEI message contains an SEI message with payloadType equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information), the value of nesting_ols_flag shall be equal to 1. If a scalable nesting SEI message contains an SEI message with payloadType equal to a value of VclAssociatedSeiList, the value of nesting_ols_flag shall be equal to 0. nesting_num_olss_minus1+1 specifies the number of OLS to which scalable nesting SEI messages apply. The value of nesting_num_olss_minus1 is assumed to be in the range of 0 to TotalNumOlss-1, including both endpoints. nesting_ols_idx_delta_minus1[i] is used to derive the variable NestingOlsIdx[i], which specifies the OLS index of the i-th OLS to which the scalable nested SEI message is applied when nesting_ols_flag is equal to 1. The value of nesting_ols_idx_delta_minus1[i] is assumed to be in the range of 0 to TotalNumOlss-2, including both endpoints. The variable NestingOlsIdx[i] is derived as follows:

number

[0177] nesting_num_ols_layers_minus1[i]+1 specifies the number of layers to which the scalable nested SEI message is applied in the context of the NestingOlsIdx[i]-th OLS. The value of nesting_num_ols_layers_minus1[i], including both endpoints, is in the range of 0 to NumLayersInOls[NestingOlsIdx[i]]-1. nesting_ols_layer_idx_delta_minus1[i][j] is used to derive the variable NestingOlsLayerIdx[i][j], which specifies the OLS layer index of the j-th layer to which the scalable nested SEI message is applied in the context of the NestingOlsIdx[i]-th OLS, when nesting_ols_flag is equal to 1. The value of nesting_ols_layer_idx_delta_minus1[i], including both endpoints, is assumed to be in the range of 0 to NumLayersInOls[nestingOlsIdx[i]]-2. The variable NestingOlsLayerIdx[i][j] is derived as follows:

number

[0178] The smallest value among all the values ​​of LayerIdInOls[NestingOlsIdx[i]][NestingOlsLayerIdx[i][0]] for i in the range from 0 to nesting_num_olss_minus1, including both endpoints, shall be equal to the nuh_layer_id of the current SEI NAL unit (the SEI NAL unit containing the scalable nesting SEI message). nesting_all_layers_flag is set to 1 to specify that the scalable nesting SEI message applies generally to all layers whose nuh_layer_id is greater than or equal to the nuh_layer_id of the current SEI NAL unit. nesting_all_layers_flag is set to 0 to specify that the scalable nesting SEI message may or may not apply generally to all layers whose nuh_layer_id is greater than or equal to the nuh_layer_id of the current SEI NAL unit. nesting_num_layers_minus1+1 specifies the number of layers to which scalable nested SEI messages are generally applied. The value of nesting_num_layers_minus1, including both ends, is in the range of 0 to vps_max_layers_minus1-GeneralLayerIdx[nuh_layer_id], where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit. nesting_layer_id[i] specifies the nuh_layer_id value of the i-th layer to which scalable nested SEI messages are generally applied when nesting_all_layers_flag is equal to 0. The value of nesting_layer_id[i] is greater than nuh_layer_id, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.If nesting_ols_flag is equal to 0, the variable NestingNumLayers specifies the number of layers to which scalable nested SEI messages are generally applied, and NestingLayerId[i] is a list of nuh_layer_id values ​​for i in the range from 0 to NestingNumLayers-1, including both ends, which specifies a list of nuh_layer_id values ​​for the layers to which scalable nested SEI messages are generally applied, and is derived as follows, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

number

[0179] nesting_num_seis_minus1+1 specifies the number of scalable nested SEI messages. The value of nesting_num_seis_minus1, including both ends, should be in the range of 0 to 63. nesting_0_bit should be equal to 0.

[0180] Figure 9 is a schematic diagram of an exemplary video coding apparatus 900. The video coding apparatus 900 is suitable for carrying out the disclosed embodiments / models described herein. The video coding apparatus 900 comprises a transceiver unit (Tx / Rx) 910 including a downstream port 920, an upstream port 950, and / or a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding apparatus 900 also includes a processor 930 including a logic unit and / or a central processing unit (CPU) for processing data, and memory 932 for storing data. The video coding apparatus 900 may also include electrical, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 950 and / or downstream port 920 for communicating data over an electrical, optical, or wireless communication network. The video coding apparatus 900 may also include input and / or output (I / O) devices 960 for communicating data with a user. The I / O device 960 may include output devices such as a display for showing video data and a speaker for outputting audio data. The I / O device 960 may also include input devices such as a keyboard, mouse, or trackball, and / or corresponding interfaces for interacting with such output devices.

[0181] The processor 930 is implemented by hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 930 communicates with downstream ports 920, Tx / Rx 910, upstream ports 950, and memory 932. The processor 930 includes a coding module 914. The coding module 914 implements embodiments disclosed herein, such as methods 100, 1000, and 1100, which can employ a multi-layer video sequence 600, a multi-layer video sequence 700, and / or bitstream 800. The coding module 914 can also implement any other methods / mechanisms described herein. Furthermore, the coding module 914 can implement a codec system 200, an encoder 300, a decoder 400, and / or HRD 500. For example, the coding module 914 may be used to implement HRD. Furthermore, the coding module 914 may be used to encode parameters into a bitstream and support the HRD conformance check process. Thus, the coding module 914 may be configured to perform a mechanism to address one or more of the aforementioned problems. Thus, the coding module 914 provides additional functionality and / or coding efficiency to the video coding device 900 when encoding video data. Thus, the coding module 914 improves the functionality of the video coding device 900 and also addresses problems specific to video coding technology. Furthermore, the coding module 914 results in a transformation of the video coding device 900 into different states. Alternatively, the coding module 914 can be implemented as instructions stored in memory 932 and executed by the processor 930 (for example, as a computer program product stored on a non-transient medium).

[0182] Memory 932 comprises one or more memory types, such as disks, tape drives, solid-state drives, read-only memory (ROM), random-access memory (RAM), flash memory, terminally associative memory (TCAM), and static random-access memory (SRAM). Memory 932 is used as an overflow data storage device and can store programs when they are selected for execution, and can store instructions and data read during program execution.

[0183] Figure 10 is a flowchart of an exemplary method 1000 for encoding a video sequence into a bitstream by including inferred HRD parameters to support bitstream conformance testing by HRD. Method 1000 may be employed by an encoder such as a codec system 200, encoder 300, and / or video coding device 900 when performing Method 100. Furthermore, Method 1000 can operate on HRD 500 and thus perform conformance testing on a multilayer video sequence 600, a multilayer video sequence 700, and / or bitstream 800.

[0184] Method 1000 can be initiated when an encoder receives a video sequence and decides to encode that video sequence into a multilayer bitstream, for example, based on user input. In step 1001, the encoder encodes multiple sublayers / sublayer representations into a bitstream. The encoder determines the HRD parameters for the sublayers. In this example, the HRD parameters are the same for all multiple sublayers / sublayer representations. The encoder encodes a set of HRD parameters into a bitstream for the largest sublayer / sublayer representation. Furthermore, the encoder encodes sublayer_cpb_params_present_flag into the bitstream. Setting sublayer_cpb_params_present_flag to 0 indicates that the HRD parameters for the topmost sublayer / sublayer representation apply to all sublayers / sublayer representations. sublayer_cpb_params_present_flag may be encoded as VPS in the bitstream.

[0185] In step 1003, HRD reads the HRD parameters and sublayer_cpb_params_present_flag. HRD can then infer that when sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all lower sublayers / sublayer representations whose TemporalId is less than the maximum TemporalId are equal to the HRD parameters of the maximum sublayer / sublayer representation that has the maximum TemporalId. For example, multiple sublayers / sublayer representations may be associated with a time ID (TemporalId), such as time ID 622. The TemporalId of the maximum sublayer representation can be expressed as HRD maximum TemporalId (hrd_max_tid[i]), where i represents the i-th HRD parameter syntax structure. Thus, the TemporalId of lower sublayers / sublayer representations may range from 0 to hrd_max_tid[i]-1. hrd_max_tid[i] may be encoded in VPS. Inferable HRD parameters may include, for example, the fixed_pic_rate_general_flag[i] syntax element, the sublayer_hrd_parameters(i) syntax structure, and / or general_vcl_hrd_params_present_flag. fixed_pic_rate_general_flag[i] is a syntax element that indicates whether the temporal distance between HRD output times of consecutive images in output order is constrained. sublayer_hrd_parameters(i) is a syntax structure that contains HRD parameters for one or more sublayers. general_vcl_hrd_params_present_flag is a flag that indicates whether VCL HRD parameters relevant to the fitting point are present in the general HRD parameter syntax structure.

[0186] In step 1005, HRD can perform a set of bitstream conformance tests on the bitstream by using HRD parameters. Specifically, HRD can perform conformance tests on all sublayers / sublayer representations (including lower sublayers / representations) using HRD parameters from the maximum sublayer / sublayer representation.

[0187] In step 1007, the encoder can store a bitstream for communication with the decoder.

[0188] Figure 11 is a flowchart of an exemplary method 1100 for decoding a video sequence from a bitstream containing inferred HRD parameters, for use in bitstream compatibility testing with an HRD such as HRD500. Method 1100 may be employed by a decoder such as a codec system 200, a decoder 400, and / or a video coding device 900 when performing Method 100. Furthermore, Method 1100 can operate on a bitstream such as a bitstream 800 containing a multilayer video sequence 600 and / or a multilayer video sequence 700.

[0189] Method 1100 can be initiated when the decoder begins receiving a bitstream of coded data representing a multilayer video sequence, for example, as a result of Method 1000. In step 1101, the receiver receives a bitstream containing multiple sublayer / sublayer representations. The bitstream also contains HRD parameters and sublayer_cpb_params_present_flag. Setting sublayer_cpb_params_present_flag to 0 indicates that the HRD parameters for the top-level sublayer / sublayer representation apply to all sublayer / sublayer representations. sublayer_cpb_params_present_flag may be encoded in VPS in the bitstream.

[0190] In step 1103, when sublayer_cpb_params_present_flag is set to 0, the decoder infers that the HRD parameters of all lower sublayer representations whose TemporalId is less than the maximum TemporalId are equal to the HRD parameters of the maximum sublayer representation that has the maximum TemporalId. For example, multiple sublayers / sublayer representations may be associated with a TemporalId such as time ID 622. The TemporalId of the maximum sublayer representation can be expressed as HRD maximum TemporalId(hrd_max_tid[i]), where i represents the i-th HRD parameter syntax structure. Thus, the TemporalId of lower sublayers / sublayer representations may range from 0 to hrd_max_tid[i]-1. hrd_max_tid[i] may be encoded in VPS. Inferable HRD parameters may include, for example, the fixed_pic_rate_general_flag[i] syntax element, the sublayer_hrd_parameters(i) syntax structure, and / or general_vcl_hrd_params_present_flag. fixed_pic_rate_general_flag[i] is a syntax element that indicates whether the temporal distance between HRD output times of consecutive images in output order is constrained. sublayer_hrd_parameters(i) is a syntax structure that contains HRD parameters for one or more sublayers. general_vcl_hrd_params_present_flag is a flag that indicates whether VCL HRD parameters relevant to the fitting point are present in the general HRD parameter syntax structure.

[0191] In step 1105, the decoder decodes the image from the sublayer / sublayer representation. In step 1107, the decoder transfers the decoded image for display as part of the decoded video sequence.

[0192] Figure 12 is a schematic diagram of an exemplary system 1200 for coding a video sequence into a bitstream by including inferred HRD parameters. System 1200 may be implemented by encoders and decoders such as a codec system 200, encoder 300, decoder 400, and / or video coding device 900. Furthermore, system 1200 can employ HRD 500 to perform conformance testing against a multilayer video sequence 600, a multilayer video sequence 700, and / or bitstream 800. In addition, system 1200 may be employed when implementing methods 100, 1000, and / or 1100.

[0193] System 1200 includes a video encoder 1202. The video encoder 1202 includes an encoding module 1203 for encoding multiple sublayer representations into a bitstream. The encoding module 1203 further encodes HRD parameters and sublayer_cpb_params_present_flag into a bitstream. The video encoder 1202 further includes an inference module 1204 for inferring that when sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is less than the maximum TemporalId are equal to the HRD parameters of the largest sublayer representation having the maximum TemporalId. The video encoder 1202 further includes an HRD module 1205 for performing a set of bitstream conformance tests on the bitstream based on the HRD parameters. The video encoder 1202 further includes a storage module 1206 for storing the bitstream for communication toward the decoder. The video encoder 1202 further includes a transmission module 1207 for transmitting the bitstream toward the video decoder 1210. The video encoder 1202 may be further configured to perform any of the steps of method 1000.

[0194] System 1200 also includes a video decoder 1210. The video decoder 1210 includes a receive module 1211 for receiving a bitstream containing multiple sublayer representations, HRD parameters, and sublayer_cpb_params_present_flag. The video decoder 1210 further includes an inference module 1213 for inferring that when sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is less than the maximumTemporalId are equal to the HRD parameters of the largest sublayer representation having the maximumTemporalId. The video decoder 1210 further includes a decode module 1215 for decoding images from the sublayer representations. The video decoder 1210 further includes a transfer module 1217 for transferring images for display as part of a decoded video sequence. The video decoder 1210 may be further configured to perform any of the steps of method 1100.

[0195] If there are no intermediary components between the first and second components, except for lines, traces, or other media, the first component is directly coupled to the second component. If there are intermediary components other than lines, traces, or other media between the first and second components, the first component is indirectly coupled to the second component. The term "coupled" and its variations include both directly and indirectly coupled components. The use of the term "about" means a range including ±10% of the following number unless otherwise specified.

[0196] It should be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of the steps in such methods should be understood to be merely illustrative. Similarly, in methods consistent with various embodiments of this disclosure, additional steps may be included in such methods, and certain steps may be omitted or combined.

[0197] While several embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of this disclosure. These embodiments should be considered illustrative and not limiting, and the invention should not be limited to the details provided herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0198] In addition, the technologies, systems, subsystems, and methods described and illustrated in various embodiments, either discretely or separately, may be combined with or integrated with other systems, components, technologies, or methods without departing from the scope of this disclosure. Other examples of modifications, substitutions, and alternatives are readily apparent to those skilled in the art and can be made without departing from the spirit and scope disclosed herein. Other possible items [Item 1] A method implemented by a decoder, The steps include: receiving a bitstream containing multiple sublayer representations, virtual reference decoder (HRD) parameters, and a sublayer coding image buffer (CPB) parameter presence flag (sublayer_cpb_params_present_flag) via the decoder's receiver; The decoder's processor infers that when the sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is less than the MaximumTemporalId are equal to the HRD parameters of the Maximum Sublayer Representation having the MaximumTemporalId. The processor performs the steps of decoding an image from the plurality of sublayer representations, Methods that include... [Item 2] The method according to item 1, wherein the sublayer_cpb_params_present_flag is included in the video parameter set (VPS) in the bitstream. [Item 3] The method described in item 1 or 2, wherein the maximum TemporalId of the maximum sublayer representation is expressed as HRD Max TemporalId(hrd_max_tid[i]), where i represents the i-th HRD parameter syntax structure. [Item 4] The method according to any one of items 1 to 3, wherein the TemporalId, which is smaller than the maximum TemporalId, is in the range of 0 to hrd_max_tid[i]-1. [Item 5] The method according to any one of items 1 to 4, wherein the HRD parameter includes a fixed picture rate general flag (fixed_pic_rate_general_flag[i]) indicating whether the temporal distance between the HRD output times of consecutive images in output order is constrained. [Item 6] The method according to any one of items 1 to 4, wherein the HRD parameters include a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers. [Item 7] The method according to any one of items 1 to 4, wherein the HRD parameter includes a general video coding layer (VCL) HRD parameter presence flag (general_vcl_hrd_params_present_flag) indicating whether the VCL HRD parameter relating to the fitting point is present in the general HRD parameter syntax structure. [Item 8] A method performed by an encoder, The encoder's processor performs the steps of encoding multiple sublayer representations into a bitstream, The processor performs the steps of encoding the virtual reference decoder (HRD) parameters and the sublayer coding image buffer (CPB) parameter presence flag (sublayer_cpb_params_present_flag) into the bitstream, The processor infers that when the sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is less than the MaximumTemporalId are equal to the HRD parameters of the Maximum Sublayer Representation having the MaximumTemporalId, The steps include: The processor performing a set of bitstream conformance tests on the bitstream based on the HRD parameters; A method that includes this. [Item 9] The method according to item 8, wherein the sublayer_cpb_params_present_flag is encoded into the video parameter set (VPS) in the bitstream. [Item 10] The method according to item 8 or 9, wherein the maximum TemporalId of the maximum sublayer representation is represented as HRD maximum TemporalId(hrd_max_tid[i]), where i represents the i-th HRD parameter syntax structure. [Item 11] The method according to any one of items 8 to 10, wherein the TemporalId, which is less than the maximum TemporalId, is in the range of 0 to hrd_max_tid[i]-1. [Item 12] The method according to any one of items 8 to 11, wherein the HRD parameter includes a fixed picture rate general flag (fixed_pic_rate_general_flag[i]) indicating whether the temporal distance between the HRD output times of consecutive images in output order is constrained. [Item 13] The method according to any one of items 8 to 12, wherein the HRD parameters include a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers. [Item 14] The method according to any one of items 8 to 13, wherein the HRD parameter includes a general video coding layer (VCL) HRD parameter presence flag (general_vcl_hrd_params_present_flag) indicating whether a VCL HRD parameter relating to a fitting point exists in the general HRD parameter syntax structure. [Item 15] A video coding device, The system comprises a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform the method described in any one of items 1 to 14. Video encoding device. [Item 16] A non-transient computer-readable medium comprising a computer program product for use by a video coding device, wherein, when the computer program product is executed by a processor, the non-transient computer-readable medium comprises computer-executable instructions stored on the non-transient computer-readable medium to cause the video coding device to perform the method described in any one of items 1 to 14. [Item 17] It is a decoder, A receiving means for receiving a bitstream containing multiple sublayer representations, virtual reference decoder (HRD) parameters, and a sublayer coding image buffer (CPB) parameter presence flag (sublayer_cpb_params_present_flag), An inference means for inferring that when the sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is smaller than the MaximumTemporalId are equal to the HRD parameters of the Maximum Sublayer Representation having the MaximumTemporalId, Decoding means for decoding an image from the aforementioned multiple sublayer representations, A transfer means for transferring the image to be displayed as part of a decoded video sequence, A decoder equipped with a decoder. [Item 18] The decoder according to item 17, further configured to perform the method described in any one of items 1 to 7. [Item 19] It is an encoder, Encode multiple sublayer representations into a bitstream, The virtual reference decoder (HRD) parameters and the sublayer coding image buffer (CPB) parameter presence flag (sublayer_cpb_params_present_flag) are encoded into the bitstream. Encoding means for, An inference means for inferring that when the sublayer_cpb_params_present_flag is set to 0, the HRD parameters of all sublayer representations whose TemporalId is smaller than the MaximumTemporalId are equal to the HRD parameters of the Maximum Sublayer Representation having the MaximumTemporalId, HRD means for performing a set of bitstream conformance tests on the bitstream based on the HRD parameters, A storage means for storing the bitstream for communication to the decoder, An encoder equipped with the following features. [Item 20] The encoder described in item 19, further configured to perform the method described in any one of items 8 through 14.

Claims

1. A method for decoding a bitstream, wherein the method is The step of receiving a bitstream having a Video Parameter Set (VPS), wherein the VPS has a vps_extension_flag, and a vps_extension_flag equal to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and a vps_extension_flag equal to 1 specifies that the VPS Specify that the vps_extension_data_flag syntax element exists in the RBSP syntax structure; the VPS further includes ols_mode_idc, where ols_mode_idc equal to 0 specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both values, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are specified as output layers, and The steps include: parsing the VPS to obtain the vps_extension_flag and the ols_mode_idc, A step of decoding one or more of the layers based on the vps_extension_flag and the ols_mode_idc. A method that includes [a certain feature].

2. The method according to claim 1, wherein the bitstream further comprises sublayer_cpb_params_present_flag, and when sublayer_cpb_params_present_flag is equal to 0, the virtual reference decoder (HRD) parameter for a sublayer representation having a TemporalId in the range of 0 to hrd_max_temporal_id[i]-1, including the endpoints, is inferred to be the same as the HRD parameter for a sublayer representation having a TemporalId equal to hrd_max_temporal_id[i].

3. The method of claim 2, wherein if sublayer_cpb_params_present_flag is equal to 1, the i-th hrd_parameters() syntax structure includes an HRD parameter for the sublayer representation having a TemporalId in the range from 0 to the hrd_max_temporal_id[i], including both endpoints.

4. The method according to claim 2 or 3, wherein if sublayer_cpb_params_present_flag is equal to 0, the i-th hrd_parameters() syntax structure includes an HRD parameter for the sublayer representation having a TemporalId equal to only the hrd_max_temporal_id[i].

5. The method according to any one of claims 2 to 4, wherein the HRD parameters include a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers.

6. A method for encoding a bitstream, wherein the method is The stage of encoding the video parameter set (VPS) into a bitstream. Equipped with, The VPS has a vps_extension_flag, and a vps_extension_flag equal to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and a vps_extension_flag equal to 1 specifies that the VPS The method specifies that the RBSP syntax structure contains the vps_extension_data_flag syntax element; the VPS further includes ols_mode_idc, where ols_mode_idc equal to 0 specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both values, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are specified as output layers.

7. The method of claim 6, further comprising the step of encoding sublayer_cpb_params_present_flag into the bitstream, wherein if sublayer_cpb_params_present_flag is equal to 0, the virtual reference decoder (HRD) parameter for a sublayer representation having a TemporalId in the range of 0 to hrd_max_temporal_id[i]-1, including both endpoints, is inferred to be the same as the HRD parameter for a sublayer representation having a TemporalId equal to hrd_max_temporal_id[i].

8. The method according to claim 7, wherein if sublayer_cpb_params_present_flag is equal to 1, the i-th hrd_parameters() syntax structure includes HRD parameters for the sublayer representation having a TemporalId in the range from 0 to the hrd_max_temporal_id[i], including both endpoints.

9. The method according to claim 7 or 8, wherein if sublayer_cpb_params_present_flag is equal to 0, the i-th hrd_parameters() syntax structure includes an HRD parameter for the sublayer representation having a TemporalId equal to only the hrd_max_temporal_id[i].

10. The method according to any one of claims 7 to 9, wherein the HRD parameter includes a sublayer_hrd_parameters(i) syntax structure that includes HRD parameters for one or more sublayers.

11. A video decoding device, The system comprises a processor and a memory coupled to the processor, wherein the processor is configured to perform the method described in any one of claims 1 to 5. Video decoding device.

12. A video encoding device, The system comprises a processor and a memory coupled to the processor, wherein the processor is configured to perform the method described in any one of claims 6 to 10. Video encoding device.

13. A computer program for causing a video coding device to perform the method according to any one of claims 1 to 5, or any one of claims 6 to 10.

14. A device for storing bitstreams, comprising one or more storage media and a receiver, wherein the receiver is configured to receive one or more bitstreams, and the one or more bitstreams have a video parameter set (VPS); the VPS has a vps_extension_flag, and if the vps_extension_flag is equal to 0, the device identifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and if the vps_extension_flag is equal to 1, the device identifies the VPS Identify that the vps_extension_data_flag syntax element exists in the RBSP syntax structure; the VPS further includes ols_mode_idc, and if ols_mode_idc is equal to 0, the device identifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both ends, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are designated as output layers; The device wherein the one or more storage mediums are configured to store the one or more bitstreams.

15. The apparatus according to claim 14, further comprising a processor configured to retrieve the bitstream from one or more storage media and transmit the bitstream to another device.

16. A method for storing a bitstream, The stage of receiving the bitstream through the receiver, The step of storing the bitstream in one or more storage media, wherein the bitstream has a video parameter set (VPS); the VPS has a vps_extension_flag, where a vps_extension_flag equal to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and a vps_extension_flag equal to 1 specifies that the VPS Specify that the vps_extension_data_flag syntax element exists in the RBSP syntax structure; the VPS further includes ols_mode_idc, where ols_mode_idc equal to 0 specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both ends, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are designated as output layers, and A method that includes [a certain feature].

17. A device for transmitting a bitstream, wherein the device is At least one storage medium configured to store at least one bitstream, wherein the bitstream has a video parameter set (VPS); the VPS has a vps_extension_flag, and if the device sets the vps_extension_flag to equal to 0, the vps_extension_flag specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure; and if the device sets the vps_extension_flag to equal to 1, the vps_extension_flag is the VPS Specify that the vps_extension_data_flag syntax element exists in the RBSP syntax structure; the VPS further includes ols_mode_idc, and if the device sets ols_mode_idc to equal 0, then ols_mode_idc specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both ends, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are designated as output layers, and is provided for at least one storage medium; A processor comprising at least one processor that acquires one or more bitstreams from one of the at least one storage mediums, At least one processor configured to transmit one or more bitstreams to another device A device equipped with the following features.

18. A method for transmitting a bitstream, wherein the method is A step of obtaining one or more bitstreams from at least one storage medium, wherein the at least one storage medium stores at least one bitstream, and the bitstream has a video parameter set (VPS); the VPS has a vps_extension_flag, and a vps_extension_flag equal to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and a vps_extension_flag equal to 1 specifies that the VPS Specify that the vps_extension_data_flag syntax element exists in the RBSP syntax structure; the VPS further includes ols_mode_idc, where ols_mode_idc equal to 0 specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both values, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are specified as output layers, and The step of transmitting one or more bitstreams to another device. A method that includes [a certain feature].

19. A method for generating a bitstream, The encoder comprises a step of generating a bitstream having a video parameter set (VPS), wherein the VPS has a vps_extension_flag, and a vps_extension_flag equal to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS Raw Byte Sequence Payload (RBSP) syntax structure, and a vps_extension_flag equal to 1 specifies that the VPS A generation method that specifies that the RBSP syntax structure contains the vps_extension_data_flag syntax element; the VPS further includes ols_mode_idc, where ols_mode_idc equal to 0 specifies that the total number of output layer sets (OLS) specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers having layer indices from 0 to i, including both values, and for each OLS, only the top layer within the OLS is output; the OLS is a set of layers in which one or more layers are specified as output layers.

Images