HRD parameters for layers

Abstract

This invention provides a method for modifying virtual reference decoder (HRD) parameters and a video coding method to support efficient encoding and / or compliance testing of multilayer bitstreams. [Solution] A method for encoding a video sequence into a bitstream includes encoding a bitstream containing one or more layers of coded images, and encoding an HRD parameter syntax structure into the bitstream. The HRD parameter syntax structure specifies that all of the layers are associated with the same number of coded image buffer (CPB) delivery schedules. The method also includes performing a set of bitstream compatibility tests on the layers based on the CPB delivery schedules, and storing the bitstream for communication to a decoder.

Description

Technical Field

[0001] Cross-Reference to Related Applications This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 905,244, filed Sep. 24, 2019, by Ye-Kui Wang, entitled “Hypothetical Reference Decoder (HRD) for Multi-Layer Video Bitstreams,” which is hereby incorporated by reference in its entirety.

[0002] This disclosure relates generally to video coding, and more particularly to virtual reference decoder (HRD) parameter changes to support efficient encoding and / or compliance testing of multi-layer bitstreams.

Background Art

[0003] Even for relatively short videos, the amount of video data required for their depiction can be enormous, and thus there can be difficulties when streaming or otherwise communicating data over a communication network with limited bandwidth capacity. Therefore, video data is generally compressed before being communicated over modern telecommunications networks. Also, when storing video on a storage device, the size of the video can be a problem because memory resources may be limited. Video compression devices often use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. There is a desire for improved compression and decompression techniques that improve the compression ratio without sacrificing much or any of the image quality, given limited network resources and the increasing demand for higher quality video.

Summary of the Invention

[0004] In one embodiment, the disclosure includes a method implemented by a decoder, the method comprising: receiving a bitstream in the decoder's receiver that includes a syntax structure of virtual reference decoder (HRD) parameters and one or more layers of an encoded image, such that all of the layers are associated with the same number of encoded image buffer (CPB) delivery schedules as specified by the syntax structure of the HRD parameters; and decoding an image from one of the layers in the decoder's processor to obtain a decoded image.

[0005] Video coding systems employ various conformance tests to ensure that a bitstream is decodeable by a decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for potential decodeable outputs for conformance. Corresponding parameters are included in the bitstream to perform conformance checks. The HRD can read the parameters and perform the tests. Video can contain many layers and many different output layer sets (OLS). On request, the encoder transmits one or more layers of a selected OLS. For example, the encoder can transmit the best layer from the OLS that can be supported by the current network bandwidth. Problems can arise when video coding systems perform conformance checks on a bitstream. Video may be coded into multiple layers and / or sublayers, which can then be organized into OLS. Each layer and / or sublayer of each OLS is checked for conformance according to a delivery schedule. Each delivery schedule is associated with different CPB sizes and CPB delays to account for different transmission bandwidths and system capabilities. Some video coding systems allow each sublayer to define any number of delivery schedules. This can result in a large amount of signaling to support conformance checks, which in turn reduces the coding efficiency of the bitstream. The disclosed embodiment includes a mechanism for improving coding efficiency for video containing multiple layers. Specifically, all layers and / or sublayers are constrained to contain the same number of CPB delivery schedules. For example, the encoder can determine the maximum number of CPB delivery schedules that can also be used for any one layer and set the number of CPB delivery schedules for all layers to this maximum. The number of delivery schedules may then be signaled once, for example, as part of the HRD parameter in a VPS.This eliminates the need to signal several schedules for each layer / sublayer. In some examples, all layers / sublayers of the OLS can even share the same delivery schedule index. These changes reduce the amount of data used to signal data related to conformance checks. This results in a smaller bitstream size and, therefore, reduced utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0006] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the CPB delivery schedule is associated with a set of bitstream conformance tests performed by an HRD operating on the encoder.

[0007] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the number of CPB delivery schedules is specified by HRD CPB count - 1 (hrd_cpb_cnt_minus1), and hrd_cpb_cnt_minus1 is in the range of 0 to 31, including both endpoints.

[0008] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, multiple layers are organized into one or more output layer sets (OLS), and the corresponding CPB delivery schedule of the target OLS is referenced by a schedule index (ScIdx) ranging from 0 to the value hrd_cpb_cnt_minus1, including the endpoint values.

[0009] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, each of the CPB delivery schedules is described by a Network Abstraction Layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and a NAL initial CPB removal offset (nal_initial_cpb_removal_offset).

[0010] Optionally, in any of the embodiments described above, another embodiment is provided in which each layer is associated with a buffering period (BP) CPB count - 1 (bp_cpb_cnt_minus1) + 1, which specifies the number of pairs of nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset for each layer, where bp_cpb_cnt_minus1 is equal to the value of hrd_cpb_cnt_minus1.

[0011] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the syntax structure of the HRD parameter is included in the bitstream within the video parameter set (VPS) that specifies the OLS.

[0012] In one embodiment, the disclosure includes a method implemented by an encoder, the method comprising: the processor of the encoder encoding a bitstream containing one or more layers of an encoded image; the processor encoding a syntax structure of HRD parameters into the bitstream specifying that all of the layers are associated with the same number of CPB delivery schedules; and the processor performing a set of bitstream conformance tests on the layers based on the CPB delivery schedules.

[0013] Video coding systems employ various conformance tests to ensure that the bitstream is decodeable by the decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for potential decodeable outputs for conformance. Corresponding parameters are included in the bitstream to perform conformance checks. The HRD can read the parameters and perform the tests. Video can contain many layers and many different OLSs. Upon request, the encoder transmits one or more layers of selected OLSs. For example, the encoder can transmit the best layer from the OLSs that can be supported by the current network bandwidth. Problems can arise when video coding systems perform conformance checks on a bitstream. Video may be coded into multiple layers and / or sublayers, which can then be organized into OLSs. Each layer and / or sublayer of each OLS is checked for conformance according to a delivery schedule. Each delivery schedule is associated with different CPB sizes and CPB delays to account for different transmission bandwidths and system capabilities. Some video coding systems allow each sublayer to define any number of delivery schedules. This can result in a large amount of signaling to support conformance checks, which in turn reduces the coding efficiency of the bitstream. The disclosed embodiment includes a mechanism for improving coding efficiency for video containing multiple layers. Specifically, all layers and / or sublayers are constrained to contain the same number of CPB delivery schedules. For example, the encoder can determine the maximum number of CPB delivery schedules that can also be used for any one layer and set the number of CPB delivery schedules for all layers to this maximum. The number of delivery schedules may then be signaled once, for example, as part of the HRD parameter in a VPS.This eliminates the need to signal several schedules for each layer / sublayer. In some examples, all layers / sublayers of the OLS can even share the same delivery schedule index. These changes reduce the amount of data used to signal data related to conformance checks. This results in a smaller bitstream size and, therefore, reduced utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0014] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, a set of bitstream conformance tests is performed by an HRD operating on the processor.

[0015] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the number of CPB distribution schedules is specified by hrd_cpb_cnt_minus1, and hrd_cpb_cnt_minus1 is in the range of 0 to 31, including both endpoints.

[0016] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, multiple layers are organized into one or more OLS, and the corresponding CPB delivery schedule of the target OLS is referenced by ScIdx in the range of 0 to the value of hrd_cpb_cnt_minus1, including the endpoint values.

[0017] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, each of the CPB delivery schedules is described by nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset.

[0018] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, each layer is associated with bp_cpb_cnt_minus1+1 which specifies the number of pairs of nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset for each layer, and bp_cpb_cnt_minus1 is equal to the value of hrd_cpb_cnt_minus1.

[0019] Optionally, another embodiment of the embodiment is provided in which, in any of the embodiments described above, the syntax structure of the HRD parameters is encoded into a bitstream in the VPS specifying the OLS.

[0020] In one embodiment, the disclosure includes a video coding apparatus comprising a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform any of the methods described above.

[0021] In one embodiment, the disclosure includes a non-transient computer-readable medium comprising a computer program product for use by a video coding apparatus, the computer program product comprising computer-executable instructions stored on the non-transient computer-readable medium, which, when executed by a processor, cause the video coding apparatus to perform any of the methods described above.

[0022] In one embodiment, the disclosure includes a decoder comprising: receiving means for receiving a bitstream including a syntax structure of HRD parameters and one or more layers of an encoded image such that all of the layers are associated with the same number of CPB delivery schedules as specified by the HRD parameters; decoding means for decoding an image from one of the layers to obtain a decoded image; and transferring means for transferring the decoded image for display as part of a decoded video sequence.

[0023] Optionally, in any of the foregoing aspects, another embodiment of the aspect is provided in which the decoder is further configured to execute any of the methods of the foregoing aspects.

[0024] In one embodiment, the present disclosure includes an encoder comprising: encoding means for encoding a bitstream comprising one or more layers of an encoded image and encoding into the bitstream a syntax structure of HRD parameters specifying that all of the plurality of layers are associated with the same number of CPB delivery schedules; HRD means for performing a set of bitstream compliance tests on the plurality of layers based on the CPB delivery schedule; and storage means for storing the bitstream for communication to a decoder.

[0025] Optionally, in any of the foregoing aspects, another embodiment of the aspect is provided in which the encoder is further configured to execute any of the methods of the foregoing aspects.

[0026] For the sake of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0027] These and other features will be more clearly understood from the following detailed description in conjunction with the accompanying drawings and the claims.

[0028] To understand the present disclosure more fully, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description in which like reference numerals represent like parts.

Brief Description of the Drawings

[0029] [Figure 1] FIG. is a flowchart of an exemplary method of encoding a video signal.

[0030] [Figure 2] This is a schematic diagram illustrating an exemplary coding / decoding (codec) system for video coding.

[0031] [Figure 3] This is a schematic diagram illustrating an example video encoder.

[0032] [Figure 4] This is a schematic diagram illustrating an exemplary video decoder.

[0033] [Figure 5] This is a schematic diagram illustrating an exemplary virtual reference decoder (HRD).

[0034] [Figure 6] This is a schematic diagram showing an exemplary multilayer video sequence configured for interlayer prediction.

[0035] [Figure 7] This is a schematic diagram showing an exemplary multi-layer video sequence configured for temporal scalability.

[0036] [Figure 8] This is a schematic diagram showing an example bitstream.

[0037] [Figure 9] This is a schematic diagram of an exemplary video coding device.

[0038] [Figure 10] This is a flowchart illustrating an exemplary method for encoding a video sequence into a bitstream, where all layers in the bitstream have the same number of coded image buffer (CPB) delivery schedules.

[0039] [Figure 11]This is a flowchart illustrating an exemplary method for decoding a video sequence from a bitstream, where all layers in the bitstream have the same number of CPB delivery schedules.

[0040] [Figure 12] This is a schematic diagram of an exemplary system for coding a video sequence using a bitstream, where all layers in the bitstream have the same number of coded image buffer (CPB) delivery schedules. [Modes for carrying out the invention]

[0041] While exemplary embodiments of one or more embodiments are provided below, it should be understood from the outset that the disclosed systems and / or methods may be carried out using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary embodiments, drawings, and techniques shown below, including the exemplary designs and embodiments illustrated and described herein, and may be modified in the entirety of their equivalents within the scope of the appended claims.

[0042] The following terms are defined as set forth below, unless used in the opposite context herein. Specifically, the following definitions are intended to provide further clarity to this disclosure. However, terms may be described differently in different contexts. Therefore, the following definitions should be considered supplementary and should not be considered to limit any other definitions of such terms provided herein.

[0043] A bitstream is a sequence of bits containing compressed video data for transmission between an encoder and a decoder. An encoder is a device configured to employ an encoding process to compress video data into a bitstream. A decoder is a device configured to employ a decoding process to reconstruct video data from the bitstream for display. An image is an array of luma samples and / or chroma samples that make up a frame or its fields. An image being encoded or decoded is sometimes called the current picture for clarity. A Network Abstraction Layer (NAL) unit is a syntax structure containing data in the form of a raw byte sequence payload (RBSP), a representation of the data type, and interspersed anti-emulation bytes as needed. A Video Coding Layer (VCL) NAL unit is a NAL unit coded to contain video data, such as an encoded slice of an image. A non-VCL NAL unit is a NAL unit containing non-video data, such as syntax and / or parameters that support decoding video data, performing conformance checks, or other operations. An Access Unit (AU) is a set of NAL units related to one particular output time, associated with each other according to a given classification rule. A decoding unit (DU) is an AU or a subset of an AU and its associated non-VCL NAL units. For example, an AU contains VCL NAL units and any non-VCL NAL units associated with the VCL NAL units within the AU. Furthermore, a DU contains a set or subset of VCL NAL units from an AU, as well as any non-VCL NAL units associated with the VCL NAL units within the DU. A layer is a set of VCL NAL units and their associated non-VCL NAL units that share a specified characteristic (e.g., common resolution, frame rate, image size, etc.). The decoding order is the order in which the syntax elements are processed by the decoding process.A video parameter set (VPS) is a data unit containing parameters related to the entire video.

[0044] A time-scalable bitstream is a bitstream encoded in multiple layers that provide various time resolutions / frame rates (for example, each layer is encoded to support a different frame rate). A sublayer is a time-scalable layer of a time-scalable bitstream that contains VCL NAL units with a specific time identifier value and associated non-VCL NAL units. For example, a time sublayer is a layer that contains video data associated with a specified frame rate. A sublayer representation is a subset of the bitstream containing the NAL units of a particular sublayer and its lower sublayers. Thus, one or more time sublayers can be combined to achieve a sublayer representation that can be decoded into a video sequence with a specified frame rate. An output layer set (OLS) is a set of layers in which one or more layers are designated as output layers. An output layer is a layer designated for output (for example, to a display). An OLS index is an index that uniquely identifies a corresponding OLS. The zeroth (0th) OLS is an OLS that contains only the lowest layer (the layer with the lowest layer identifier) ​​and therefore only the output layer. A time identifier (ID) is a data element that indicates the time position of the data within the video sequence. The sub-bitstream extraction process removes NAL units from the bitstream that do not belong to the target set, as determined by the target OLS index and target highest time ID. The sub-bitstream extraction process outputs a sub-bitstream from the bitstream that contains the NAL units that are part of the target set.

[0045] HRD is a decoder model that operates on an encoder, checking the variability of the bitstream generated by the encoding process and verifying its compliance with specified constraints. Bitstream compliance testing is a test to determine whether the encoded bitstream conforms to standards such as VVC (Versatile Video Coding). HRD parameters are syntax elements that initialize and / or define the operating conditions of HRD. HRD parameters can be included in the syntax structure of HRD parameters. A syntax structure is a data object configured to contain multiple different parameters. A syntax element is a data object that contains one or more parameters of the same type. Therefore, a syntax structure can contain multiple syntax elements. Sequence-level HRD parameters are HRD parameters that apply to the entire encoded video sequence. The maximum HRD time ID (hrd_max_tid[i]) specifies the time ID of the highest sublayer representation in which the HRD parameter is included in the i-th OLS HRD parameter set. The general_hrd_parameters syntax structure is a syntax structure that contains sequence-level HRD parameters. An operating point (OP) is a time subset of the OLS, identified by the OLS index and highest time ID. The target OP is the OP selected in the HRD for conformance testing. The target OLS is the OLS selected to be extracted from the bitstream. The decoding unit HRD parameter presence flag (decoding_unit_hrd_params_present_flag) indicates whether the corresponding HRD parameter operates at the DU level or the AU level. The coded image buffer (CPB) is a first-in, first-out buffer in the HRD that contains coded images in decoding order for use during bitstream conformance verification. The decoded image buffer (DPB) is a buffer for holding decoded images for reference, output sorting, and / or output delay.

[0046] Supplemental Expansion Information (SEI) messages are semantically specified syntax structures that convey information not required by the decoding process to determine the sample values ​​of a decoded image. Scalable nesting SEI messages are messages containing multiple SEI messages corresponding to one or more OLS or one or more layers. Non-scalable nesting SEI messages are messages that are not nested and therefore contain a single SEI message. Buffering Period (BP) SEI messages are SEI messages containing HRD parameters for initializing HRD to manage CPB. Image Timing (PT) SEI messages are SEI messages containing HRD parameters for managing AU delivery information in CPB and / or DPB. Decoded Unit Information (DUI) SEI messages are SEI messages containing HRD parameters for managing delivery information for DU in CPB and / or DPB.

[0047] The CPB removal delay is the period during which the corresponding current AU can remain in the CPB before being removed and output to the DPB. The initial CPB removal delay is the default CPB removal delay for each image, AU, and / or DU in the bitstream, OLS, and / or layer. The CPB removal offset is the location within the CPB used to determine the boundary of the corresponding AU in the CPB. The initial CPB removal offset is the default CPB removal offset associated with each image, AU, and / or DU in the bitstream, OLS, and / or layer. The Decoded Image Buffer (DPB) output delay information is the period during which the corresponding AU can remain in the DPB before output. The CPB removal delay information is information related to the removal of the corresponding DU from the CPB. The delivery schedule specifies the timing for the delivery of video data to and from memory locations such as the CPB and / or DPB. The VPS layer ID (vps_layer_id) is a syntax element that indicates the layer ID of the i-th layer shown in the VPS. The number of output layer sets minus 1 (num_output_layer_sets_minus1) is a syntax element that specifies the total number of OLS specified by the VPS. The HRD-encoded image buffer count (hrd_cpb_cnt_minus1) is a syntax element that specifies the number of alternative CPB delivery schedules. The sublayer CPB parameter presence flag (sublayer_cpb_params_present_flag) is a syntax element that specifies whether the set of OLS HRD parameters includes the HRD parameters for the specified sublayer representation. The schedule index (ScIdx) is an index that identifies the delivery schedule. The BP CPB count minus 1 (bp_cpb_cnt_minus1) is a syntax element that specifies the number of pairs of initial CPB removal delays and offsets, and therefore the number of delivery schedules available for the time sublayer. The NAL unit header layer identifier (nuh_layer_id) is a syntax element that specifies the identifier of the layer containing the NAL unit.The fixed_pic_rate_general_flag syntax element specifies whether to constrain the temporal distance between HRD output times of consecutive images in output order. The sublayer_hrd_parameters syntax structure is a syntax structure that includes the HRD parameters of the corresponding sublayer. The general_vcl_hrd_params_present_flag flag specifies whether VCL HRD parameters are present in the syntax structure of general HRD parameters. The bp_max_sublayers_minus1 syntax element specifies the maximum number of time sublayers in which CPB removal delay and CPB removal offset are indicated in BP SEI messages. The vps_max_sublayers_minus1 syntax element specifies the maximum number of time sublayers that may exist in a layer specified by VPS. The Scalable Nesting OLS flag specifies whether a scalable nesting SEI message applies to a specific OLS or to a specific layer. Scalable Nesting OLS Number - 1 (num_olss_minus1) is a syntax element that specifies the number of OLS to which a scalable nesting SEI message applies. Nesting OLS Index (NestingOlsIdx) is a syntax element that specifies the OLS index of the OLS to which a scalable nesting SEI message applies. Target OLS Index (targetOlsIdx) is a variable that identifies the OLS index of the OLS to be decrypted. Total OLS Number - 1 (OLS-1) is a syntax element that specifies the total number of OLS specified in the VPS.

[0048] The following acronyms are used herein: Access Unit (AU), Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Codified Layer Video Sequence (CLVS), Codified Layer Video Sequence Start (CLVSS), Codified Video Sequence (CVS), Codified Video Sequence Start (CVSS), Joint Video Expert Team (JVET), Virtual Reference Decoder (HRD), Motion Constrained Tilesets (MCTS), Maximum Transmission Unit (MTU), Network Abstraction Layer (NAL), Output Layer Set (OLS), Image Order Count (POC), Random Access Point (RAP), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), Video Parameter Set (VPS), and General-Purpose Video Coding (VVC).

[0049] Many video compression techniques can be employed to minimize data loss and reduce the size of video files. For example, video compression techniques may include performing spatial (e.g., within an image) prediction and / or temporal (e.g., between images) prediction to reduce or eliminate data redundancy in a video sequence. In block-based video coding, a video slice (e.g., a video image or a portion of a video image) may be divided into video blocks, which are sometimes called tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coding (I) slice of an image are coded using spatial prediction against reference samples of adjacent blocks in the same image. Video blocks in an intercoding (P) or bidirectional prediction (B) slice of an image may be coded by employing spatial prediction against reference samples of adjacent blocks in the same image, or temporal prediction against reference samples of other reference images. Images may be called frames and / or images, and reference images may be called reference frames and / or reference images. Spatial or temporal prediction yields predicted blocks representing image blocks. Residual data represents the difference in pixels between the original image blocks and the predicted blocks. Thus, intercoded blocks are encoded according to motion vectors pointing to the reference sample blocks forming the predicted blocks and residual data showing the difference between the coded blocks and the predicted blocks. Intracoded blocks are encoded according to the intracoding mode and residual data. For further compression, the residual data may be transformed from the pixel region to the transformation region. This results in residual transformation coefficients that can be quantized. The quantized transformation coefficients may first be arranged in a two-dimensional array. The quantized transformation coefficients may be scanned to generate a one-dimensional vector of transformation coefficients. Entropy coding may be applied to achieve even greater compression. Such video compression techniques are described in more detail below.

[0050] To ensure that encoded video can be accurately decoded, video is encoded and decoded according to the corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding plus Depth (MVC+D), as well as 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC joint video experts team (JVET) began developing a video coding standard called Versatile Video Coding (VVC). VVC is included in Working Drafts (WDs), including JVET-O2001-v14.

[0051] Video coding systems employ various conformance tests to ensure that a bitstream is decodeable by a decoder. For example, conformance checks may include testing the entire bitstream for conformance, then testing each layer of the bitstream for conformance, and finally checking for potential decodeable outputs for conformance. Corresponding parameters are included in the bitstream to perform the conformance checks. A virtual reference decoder (HRD) can read the parameters and perform the tests. Video can contain many layers and many different output layer sets (OLS). Upon request, the encoder transmits one or more layers of a selected OLS. For example, the encoder can transmit the best layer from the OLS that can be supported by the current network bandwidth. The first problem with this technique is that a considerable number of layers are tested but are not actually transmitted to the decoder. However, parameters to support such tests may still be included in the bitstream, unnecessarily increasing the size of the bitstream.

[0052] In the first embodiment, a mechanism for applying bitstream conformance testing to only each OLS is disclosed herein. In this way, when testing the corresponding OLS, the entire bitstream, each layer, and the decodeable output are tested together. Thus, the number of conformance tests is reduced, thereby reducing the use of processor and memory resources in the encoder. Furthermore, by reducing the number of conformance tests, the number of associated parameters included in the bitstream can be reduced. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0053] The second problem is that the signaling process for HRD parameters used in HRD compliance testing in some video coding systems can become complex in a multi-layer context. For example, a set of HRD parameters may be signaled for each layer of each OLS. Such HRD parameters may be signaled at different locations within the bitstream, depending on the intended range of the parameters. As a result, the scheme becomes more complex as more layers and / or OLS are added. Furthermore, HRD parameters for different layers and / or OLS may contain redundant information.

[0054] In a second embodiment, a mechanism for signaling a global set of sequence-level HRD parameters for OLS and corresponding layers is disclosed herein. For example, all sequence-level HRD parameters applicable to all OLS and all layers contained within the OLS are signaled by a video parameter set (VPS). The VPS is signaled once with the bitstream, and therefore the sequence-level HRD parameters are signaled once. Furthermore, the sequence-level HRD parameters may be constrained to be identical across all OLS. In this way, redundant signaling is reduced and coding efficiency is improved. This technique also simplifies the HRD process. As a result, the usage of processor, memory, and / or network signaling resources is reduced in both the encoder and decoder.

[0055] A third problem can arise when the video coding system performs bitstream conformance checks. Video may be coded into multiple layers and / or sublayers, which can then be organized into an OLS. Each layer and / or sublayer in each OLS is checked for conformance according to a delivery schedule. Each delivery schedule is associated with different coded image buffer (CPB) sizes and CPB delays to account for different transmission bandwidths and system capabilities. Some video coding systems allow each sublayer to define any number of delivery schedules. This can result in a large amount of signaling to support conformance checks, which in turn reduces the efficiency of bitstream coding.

[0056] In a third embodiment, a mechanism for improving the coding efficiency of video containing multiple layers is disclosed herein. Specifically, all layers and / or sublayers are constrained to contain the same number of CPB delivery schedules. For example, the encoder can determine the maximum number of CPB delivery schedules to be used for any one layer and set the number of CPB delivery schedules for all layers to this maximum. The number of delivery schedules may then be signaled once, for example, as part of the HRD parameter in a VPS. This eliminates the need to signal several schedules per layer / sublayer. In some examples, all layers / sublayers of the OLS may also share the same delivery schedule index. These changes reduce the amount of data used to signal data related to compliance checks. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0057] A fourth issue can arise when video is coded into multiple layers and / or sublayers, which are then organized into an OLS. The OLS may include a zeroth (0th) OLS containing only the output layer. Supplemental Extension Information (SEI) messages may be included in the bitstream to inform the HRD of layer / OLS-specific parameters used to test multiple layers of the bitstream for compliance with the standard. Specifically, if an OLS is included in the bitstream, a scalable nesting SEI message is employed. A scalable nesting SEI message consists of a group of nested SEI messages that apply to one or more OLSs and / or one or more layers of an OLS. Each nested SEI message may include an indicator to show its association with the corresponding OLS and / or layer. Nested SEI messages are configured for use with multiple layers and may contain irrelevant information when applied to a 0th OLS containing a single layer.

[0058] In a fourth embodiment, a mechanism for improving the coding efficiency of a video including a zeroth OLS is disclosed herein. A non-scalable nested SEI message is employed in the zeroth OLS. The non-scalable nested SEI message is constrained to apply only to the zeroth OLS and therefore only to the output layers included in the zeroth OLS. In this way, irrelevant information such as nesting relationships and layer indications can be omitted from the SEI message. The non-scalable nested SEI message may be used as a buffering period (BP) SEI message, an image timing (PT) SEI message, a decoding unit (DU) SEI message, or a combination thereof. These modifications reduce the amount of data used to signal conformance check-related information to the zeroth OLS. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0059] A fifth problem can also arise when video is separated into multiple layers and / or sublayers. Encoders can encode these layers into bitstreams. Furthermore, encoders can use HRDs to perform conformance tests to check the bitstream for compliance with standards. Encoders may be configured to include layer-specific HRD parameters in the bitstream to support such conformance tests. Layer-specific HRD parameters may be encoded per layer in some video coding systems. In some cases, the layer-specific HRD parameters are the same per layer, resulting in redundant information that unnecessarily increases the size of the video encoding.

[0060] In a fifth embodiment, a mechanism for reducing HRD parameter redundancy for video employing multiple layers is disclosed herein. The encoder can encode the HRD parameters of the top layer. The encoder can also encode a sublayer CPB parameter presence flag (sublayer_cpb_params_present_flag). Setting sublayer_cpb_params_present_flag to 0 indicates that all lower layers should use the same HRD parameters as the top layer. In this context, the top layer is the layer with the largest layer identifier (ID), and lower layers are any layers with a layer ID smaller than the layer ID of the top layer. In this way, the HRD parameters of lower layers can be omitted from the bitstream. This reduces the bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0061] The sixth issue concerns the use of Sequence Parameter Sets (SPS), which contain syntax elements associated with each video sequence in a video. A video coding system can code video in layers and / or sublayers. Video sequences may behave differently in different layers and / or sublayers. Therefore, different layers may refer to different SPS. BP SEI messages can indicate which layers / sublayers are being checked for compliance with the standard. Some video coding systems may indicate that a BP SEI message applies to the layer / sublayer indicated by the SPS. This can lead to unexpected errors, as different layers may refer to different SPS, and such SPS may contain conflicting information.

[0062] In a sixth embodiment, a mechanism for addressing errors related to conformance checks when multiple layers are employed in a video sequence is disclosed herein. Specifically, the BP SEI message is modified to indicate that the conformance of any number of layers / sublayers described in the VPS can be checked. For example, the BP SEI message may include a bp_max_sublayers_minus1 syntax element, which indicates the number of layers / sublayers associated with the data in the BP SEI message. On the other hand, the vps_max_sublayers_minus1 syntax element in the VPS indicates the number of sublayers in the entire video. The bp_max_sublayers_minus1 syntax element can be set to any value from 0 to the value of the vps_max_sublayers_minus1 syntax element. In this way, the conformance of any number of layers / sublayers in the video can be checked while avoiding layer-based sequencing problems related to SPS inconstancy. Therefore, this disclosure avoids layer-based coding errors and thus improves the functionality of the encoder and / or decoder. Furthermore, this embodiment supports layer-based coding, which can improve coding efficiency. Thus, this embodiment supports a reduction in processor, memory, and / or network resource usage in the encoder and / or decoder.

[0063] The seventh issue concerns the layers included in the OLS. Each OLS includes at least one output layer configured to be displayed by a decoder. The encoder's HRD can check each OLS for compliance with the standard. A compliant OLS can always be decoded and displayed by a compliant decoder. The HRD process may be partially managed by SEI messages. For example, a scalable nesting SEI message may contain scalable nested SEI messages. Each scalable nested SEI message may contain data related to the corresponding layer. When performing a compliance check, the HRD can perform a bitstream extraction process on the target OLS. Data not related to the layers within the OLS is generally removed before compliance testing (e.g., before transmission) so that each OLS can be checked separately. Some video coding systems do not remove such messages during the sub-bitstream extraction process because scalable nesting SEI messages relate to multiple layers. Therefore, even when a scalable nesting SEI message is not relevant to any layer of the target OLS (the extracted OLS), it may remain in the bitstream after sub-bitstream extraction. This can increase the size of the final bitstream without providing any additional functionality.

[0064] In a seventh embodiment, a mechanism for reducing the size of a multilayer bitstream is disclosed herein. During sub-bitstream extraction, scalable nesting SEI messages may be considered for removal from the bitstream. If a scalable nesting SEI message is associated with one or more OLSs, the scalable nested SEI messages within the scalable nesting SEI message are checked. If a scalable nested SEI message is not associated with any layer of the target OLS, the entire scalable nesting SEI message can be removed from the bitstream. As a result, the size of the bitstream sent to the decoder is reduced. Thus, this embodiment improves coding efficiency and reduces the use of processor, memory, and / or network resources in both the encoder and decoder.

[0065] Figure 1 is a flowchart of an exemplary operation method 100 for coding a video signal. Specifically, the video signal is encoded by an encoder. The encoding process compresses the video signal by employing various mechanisms to reduce the video file size. Reducing the file size allows the compressed video file to be sent to the user while reducing the associated bandwidth overhead. The decoder decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process so that the decoder can consistently reconstruct the video signal.

[0066] In step 101, a video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. In another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. The video file may contain both audio and video components. The video component contains a series of image frames, which, when viewed sequentially, give the impression of visual movement. Each frame contains pixels, represented in this specification with respect to light, called the lumens component (or lumens sample), and color, called the chromens component (or color sample). In some examples, the frames may also include depth values ​​to support a three-dimensional view.

[0067] In step 103, the video is divided into blocks. The division involves subdividing the pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels × 64 pixels). A CTU contains both luminous and chroma samples. A coding tree can be employed to divide the CTUs into blocks, and then the blocks can be recursively subdivided until a configuration supporting further coding is achieved. For example, the luminous component of a frame may be subdivided until the individual blocks contain relatively uniform lighting values. Furthermore, the chroma component of a frame may be subdivided until the individual blocks contain relatively uniform color values. Thus, the division mechanism varies depending on the content of the video frame.

[0068] In step 105, various compression mechanisms are employed to compress the image blocks divided in step 103. For example, inter-prediction and / or intra-prediction can be employed. Inter-prediction is designed to take advantage of the fact that objects tend to appear in consecutive frames within a common scene. Therefore, it is not necessary to repeatedly describe the block representing the object in the reference frame in adjacent frames. Specifically, objects such as tables may remain in the same position across multiple frames. Therefore, a table is described once, and adjacent frames can refer to the reference frame. A pattern matching mechanism can be used to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to the movement of the object or the movement of the camera. A particular example is a video that shows a car moving across the screen across multiple frames. Such motion can be described using motion vectors. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in a frame to the coordinates of an object in a reference frame. Therefore, inter-prediction can encode the image block in the current frame as a set of motion vectors that show the offset from the corresponding block in the reference frame.

[0069] Intra-prediction encodes blocks within a common frame. It leverages the fact that luma and chroma components tend to cluster within a frame. For example, some green patches in a tree tend to be adjacent to similar green patches. Intra-prediction employs multiple directional prediction modes (e.g., 33 in HEVC), a planar mode, and a DC mode. Directional modes indicate that the current block is similar / identical to samples of adjacent blocks in the corresponding direction. Planar modes indicate that a sequence of blocks along a row / column (e.g., a plane) can be interpolated based on adjacent blocks at the edges of the row. Planar modes effectively show smooth light / color transitions across rows / columns by using a relatively constant slope when changing values. DC modes are employed for boundary smoothing and indicate that a block is similar / identical to the mean associated with samples of all adjacent blocks associated with the angular direction of the directional prediction mode. Thus, intra-predicted blocks can represent image blocks not as actual values, but as various relational prediction mode values. Furthermore, the interpretation block can represent the image block as a motion vector value rather than its actual value. In either case, the prediction block may not accurately represent the image block in some cases. Any differences are stored in the residual block. Transformations can be applied to the residual block to further compress the file.

[0070] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. Block-based prediction, as described above, can generate blocky images in the decoder. Furthermore, block-based prediction may reconstruct the encoded blocks after encoding them for later use as reference blocks. In-loop filtering repeatedly applies noise suppression filters, deblocking filters, adaptive loop filters, and sample-adaptive offset (SAO) filters to blocks / frames. These filters reduce such blocking artifacts, allowing the encoded file to be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference blocks, making it less likely that these artifacts will generate additional artifacts in subsequent blocks encoded based on the reconstructed reference blocks.

[0071] Once the video signal has been split, compressed, and filtered, in step 109 the resulting data is encoded into a bitstream. The bitstream contains the data described above, as well as any signaling data desired to support appropriate video signal reconstruction in the decoder. For example, such data may include split data, prediction data, residual blocks, and various flags that give coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. Alternatively, the bitstream may be broadcast and / or multicast to multiple decoders. The creation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 may be performed sequentially and / or simultaneously across many frames and blocks. The order shown in Figure 1 is presented for clarity and ease of discussion and is not intended to restrict the video coding process to a specific order.

[0072] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder employs an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. In step 111, the decoder uses the syntax data from the bitstream to determine the division of frames. This division should match the result of the block division in step 103. Next, the entropy coding / decoding employed in step 111 is described. The encoder makes many choices during the compression process, such as selecting a block division scheme from several possible options based on the spatial arrangement of values ​​in the input image. Signaling the precise selection may involve using a number of bins. As used herein, a bin is a binary value (e.g., a bit value that may change depending on the context) treated as a variable. Entropy coding allows the encoder to discard any options that are obviously not feasible for a particular case, leaving a set of acceptable options. Each acceptable option is then assigned a codeword. The length of the codeword is based on the number of allowed options (e.g., one bin for two options, two bins for three or four options, etc.). The encoder then encodes the codeword for the selected option. This method reduces the size of the codeword because the desired size is sufficient to uniquely indicate a selection from a small subset of allowed options, rather than uniquely indicating a selection from a potentially large set of all possible options. The decoder then decodes this selection by determining the set of allowed options, similar to the encoder. By determining the set of allowed options, the decoder can read the codeword and determine the selection made by the encoder.

[0073] In step 113, the decoder performs block decoding. Specifically, the decoder generates residual blocks using the inverse transform. The decoder then reconstructs the image blocks according to the partitioning using the residual blocks and the corresponding prediction blocks. The prediction blocks may include both intra-prediction blocks and inter-prediction blocks generated by the encoder in step 105. The reconstructed image blocks are then placed into the frames of the reconstructed video signal according to the partitioning data determined in step 111. The syntax of step 113 may also be signaled in the bitstream via entropy coding, as described above.

[0074] In step 115, the frames of the reconstructed video signal are filtered in a manner similar to that in step 107 of the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be applied to the frames to remove blocking artifacts. Once the frames are filtered, the video signal may be output to a display for viewing by the end user in step 117.

[0075] Figure 2 is a schematic diagram of an exemplary coding / decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functions to support the implementation of operation method 100. The codec system 200 is generalized to show components used in both the encoder and the decoder. The codec system 200 receives and divides the video signal, as described with respect to steps 101 and 103 of operation method 100, resulting in the divided video signal 201. The codec system 200 then compresses the divided video signal 201 into an encoded bitstream, as described with respect to steps 105, 107, and 109 of method 100. When operating as a decoder, the codec system 200 generates an output video signal from the bitstream, as described with respect to steps 111, 113, 115, and 117 of operation method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-image estimation component 215, an intra-image prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter-controlled analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. These components are coupled together as shown in the figure. In Figure 2, black lines indicate the movement of data being encoded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All components of the codec system 200 may reside within the encoder. The decoder may contain a subset of the components of the codec system 200.For example, the decoder may include an intra-image prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.

[0076] The partitioned video signal 201 is a captured video sequence that has been divided into blocks of pixels by a coding tree. The coding tree subdivides blocks of pixels into smaller blocks of pixels using various partitioning modes. These blocks can then be subdivided into even smaller blocks. Blocks are sometimes called nodes on the coding tree. Larger parent nodes are subdivided into smaller child nodes. The number of times a node is subdivided is called the node / coding tree depth. The partitioned blocks may sometimes be contained within a coding unit (CU). For example, a CU can be a sub-part of a CTU, including a luma block, a red difference chroma (Cr) block, and a blue difference chroma (Cb) block, along with the corresponding syntax instructions for the CU. Partitioning modes can include binary trees (BT), ternary trees (TT), and quadary trees (QT), which are employed to subdivide nodes into two, three, or four child nodes of various shapes, depending on the partitioning mode used. The divided video signal 201 is transferred for compression to a general coder control component 211, a transformation scaling and quantization component 213, an intra-image estimation component 215, a filter control analysis component 227, and a motion estimation component 221.

[0077] The general coder control component 211 is configured to make decisions related to coding the video sequence image into a bitstream according to application constraints. For example, the general coder control component 211 manages the optimization of bitrate / bitstream size for reconstruction quality. Such decisions can be made based on memory space / bandwidth availability and image resolution requests. The general coder control component 211 also manages buffer utilization considering the transmission speed and mitigates buffer underrun and overrun issues. To manage these issues, the general coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the general coder control component 211 can dynamically increase the complexity of compression to improve resolution and increase bandwidth usage, or decrease the complexity of compression to reduce resolution and bandwidth usage. Thus, the general coder control component 211 controls other components of the codec system 200 to balance the relationship between video signal reconstruction quality and bitrate. The general coder control component 211 creates control data that controls the operation of other components. The control data is also transferred to the header formatting and CABAC component 231, encoded into a bitstream, and signaling parameters for decoding by the decoder.

[0078] The divided video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for interprediction. A frame or slice of the divided video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform relative interprediction coding of the received video blocks for one or more blocks within one or more reference frames, providing time predictions. The codec system 200 can perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0079] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors can, for example, represent the relative displacement of the coded object with respect to the predicted block. A predicted block is a block that is found to closely match the block being coded in terms of pixel differences. Predicted blocks are sometimes called reference blocks. Such pixel differences may be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. HEVC employs several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be split into CTBs, and then the CTBs can be split into CBs to be included in the CUs. A CU can be coded as a prediction unit (PU) containing prediction data and / or a transformation unit (TU) containing residual data transformed for the CUs. The motion estimation component 221 generates motion vectors, PUs, and TUs by using rate-distortion analysis as part of the rate-distortion optimization process. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame and select the reference blocks, motion vectors, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of video reconstruction (e.g., the amount of data loss due to compression) and coding efficiency (e.g., the size of the final encoding).

[0080] In some examples, the codec system 200 can calculate values ​​for sub-integer pixel positions of the reference image stored in the decoded image buffer component 223. For example, the video codec system 200 can interpolate values ​​for quarter-pixel, eighth-pixel, or other fractional-pixel positions of the reference image. Thus, the motion estimation component 221 can perform a relative motion search between full-pixel positions and fractional-pixel positions and output a motion vector with fractional-pixel precision. The motion estimation component 221 calculates the motion vector of the PU of the reference image in the video block during the intercoded slice by comparing the PU position with the predicted block position. The motion estimation component 221 outputs the calculated motion vector as motion data to the header formatting and CABAC component 231 for encoding and outputs the motion to the motion compensation component 219.

[0081] Motion compensation performed by the motion compensation component 219 may include fetching or generating a predicted block based on a motion vector determined by the motion estimation component 221. In this case as well, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated in some examples. Upon receiving a motion vector for the PU of the current video block, the motion compensation component 219 can determine the location of the predicted block pointed to by the motion vector. The residual video block is then formed by subtracting the pixel values ​​of the predicted block from the coded pixel values ​​of the current video block to form a pixel difference value. Generally, the motion estimation component 221 performs motion estimation for the lumens component, and the motion compensation component 219 uses a motion vector calculated based on the lumens component for both the chromens and lumens components. The predicted and residual blocks are then transferred to the transformation scaling and quantization component 213.

[0082] The divided video signal 201 is also sent to the intra-image estimation component 215 and the intra-image prediction component 217. Similar to the motion estimation component 221 and the motion compensation component 219, the intra-image estimation component 215 and the intra-image prediction component 217 may be highly integrated, but are shown separately for conceptual purposes. The intra-image estimation component 215 and the intra-image prediction component 217 intra-predict the current block for the block in the current frame, as an alternative to the inter-prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames, as described above. In particular, the intra-image estimation component 215 determines the intra-prediction mode to use for encoding the current block. In some examples, the intra-image estimation component 215 selects an appropriate intra-prediction mode from several tested intra-prediction modes to encode the current block. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0083] For example, the intra-image estimation component 215 calculates rate-distortion values ​​for various tested intra-prediction modes using rate-distortion analysis and selects the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original unencoded block encoded to produce the encoded block, as well as the bit rate (e.g., number of bits) used to produce the encoded block. The intra-image estimation component 215 calculates a ratio from the distortion and rate of various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block. In addition, the intra-image estimation component 215 may be configured to encode depth blocks of the depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

[0084] The intra-image prediction component 217, when implemented on an encoder, can generate residual blocks from prediction blocks based on a selected intra-prediction mode determined by the intra-image estimation component 215, or, when implemented on a decoder, can read residual blocks from a bitstream. The residual blocks contain the difference in values ​​between the prediction blocks and the original blocks and are represented as a matrix. The residual blocks are then transferred to the transformation scaling and quantization component 213. The intra-image estimation component 215 and the intra-image prediction component 217 can operate on both luma and chroma components.

[0085] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transform, to the residual block to generate a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms can also be used. The transform can convert the residual information from the pixel value domain to a transformation domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bitrate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the transformation scaling and quantization component 213 can then perform a scan of a matrix containing the quantized transformation coefficients. The quantized transformation coefficients are then transferred to the header formatting and CABAC component 231 to be encoded into a bitstream.

[0086] The scaling and inverse transform component 229 supports motion estimation by applying the inverse operations of the transform scaling and quantization component 213. The scaling and inverse transform component 229 reconstructs the residual block in the pixel region by applying inverse scaling, transform, and / or quantization for later use as a reference block that may become a prediction block for another current block, for example. The motion estimation component 221 and / or motion compensation component 219 can compute the reference block by adding the residual block to the corresponding prediction block for use in motion estimation of a later block / frame. A filter is applied to the reconstructed reference block to mitigate artifacts generated during scaling, quantization, and transform. Such artifacts can normally lead to inaccurate predictions (and generate additional artifacts) when predicting subsequent blocks.

[0087] The filter-controlled analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, a transformed residual block from the scaling and inverse transform component 229 can be combined with the corresponding predicted block from the intra-image prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. A filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block instead. Like the other components in Figure 2, the filter-controlled analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are shown separately for conceptual purposes. A filter applied to a reconstructed reference block is applied to a specific spatial region and includes several parameters to adjust how such a filter is applied. The filter-controlled analysis component 227 analyzes the reconstructed reference block to determine where such a filter is applied and sets the corresponding parameters. Such data is transferred to the header formatting and CABAC component 231 as filter-controlled data for encoding. The in-loop filter component 225 applies such a filter based on the filter-controlled data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Depending on the embodiment, such filters may be applied in the spatial / pixel domain (e.g., on reconstructed pixel blocks) or in the frequency domain.

[0088] When operating as an encoder, the filtered and reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for later use in the motion estimation described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and transfers them to the display as part of the output video signal. The decoded image buffer component 223 may be any memory device capable of storing the prediction blocks, residual blocks, and / or reconstructed image blocks.

[0089] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to the decoder. Specifically, the header formatting and CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. Furthermore, prediction data, including intra-prediction and motion data, as well as residual data in the form of quantized transformation coefficient data, are all encoded into a bitstream. The final bitstream contains all the information desired by the decoder to reconstruct the original segmented video signal 201. Such information may include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, indications of the most probable intra-prediction mode, and indications of segmentation information. Such data can be encoded using entropy coding. For example, information can be encoded by employing context-adaptive variable-length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), stochastic interval-partitioned entropy (PIPE) coding, or other entropy coding techniques. Following entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0090] Figure 3 is a block diagram showing an exemplary video encoder 300. The video encoder 300 can be employed to perform the encoding function of the codec system 200 and / or to perform steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 splits the input video signal, resulting in a split video signal 301 that is substantially similar to the split video signal 201. The split video signal 301 is then compressed by the components of the encoder 300 and encoded into a bitstream.

[0091] Specifically, the segmented video signal 301 is transferred to the intra-image prediction component 317 for intra-prediction. The intra-image prediction component 317 may be substantially the same as the intra-image estimation component 215 and the intra-image prediction component 217. The segmented video signal 301 is also transferred to the motion compensation component 321 for intra-prediction based on a reference block in the decoding image buffer component 323. The motion compensation component 321 may be substantially the same as the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-image prediction component 317 and the motion compensation component 321 are transferred to the transformation and quantization component 313 for transformation and quantization of the residual blocks. The transformation and quantization component 313 may be substantially the same as the transformation scaling and quantization component 213. The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are transferred to the entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially the same as the header formatting and CABAC component 231.

[0092] The transformed and quantized residual blocks and / or corresponding predicted blocks are also transferred from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into reference blocks for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. Depending on the embodiment, the in-loop filter of the in-loop filter component 325 is also applied to the residual blocks and / or the reconstructed reference blocks. The in-loop filter component 325 may be substantially similar to the filter-controlled analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters as described with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded image buffer component 323 for use as reference blocks by the motion compensation component 321. The decoded image buffer component 323 may be substantially similar to the decoded image buffer component 223.

[0093] Figure 4 is a block diagram showing an exemplary video decoder 400. The video decoder 400 can be employed to perform the decoding function of the codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of the operation method 100. The decoder 400 receives a bitstream from, for example, the encoder 300 and generates an output video signal reconstructed based on the bitstream for display to the end user.

[0094] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may employ header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, division information, motion data, prediction data, and quantized transformation coefficients from residual blocks. The quantized transformation coefficients are transferred to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0095] The reconstructed residual blocks and / or predicted blocks are transferred to the intra-image prediction component 417 for reconstruction into image blocks based on intra-predictive operation. The intra-image prediction component 417 may be similar to the intra-image estimation component 215 and the intra-image prediction component 217. Specifically, the intra-image prediction component 417 uses a prediction mode to locate the position of a reference block in the frame and applies the residual block to the result to reconstruct the intra-predicted image block. The reconstructed intra-predicted image block and / or residual block, along with the corresponding inter-predictive data, are transferred to the decoded image buffer component 423 via the in-loop filter component 425, which may be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or predicted block, and such information is stored in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is transferred to the motion compensation component 421 for inter-prediction. The motion compensation component 421 may be substantially the same as the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 generates a predicted block using a motion vector from a reference block and reconstructs the image block by applying a residual block to the result. The resulting reconstructed block may also be transferred to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 continues to store additional reconstructed image blocks that can be reconstructed into frames via segmentation information. Such frames can also be placed in a sequence. The sequence is output to a display as a reconstructed output video signal.

[0096] Figure 5 is a schematic diagram showing an exemplary HRD500. The HRD500 can be employed in an encoder such as the codec system 200 and / or encoder 300. The HRD500 can check the bitstream created in step 109 of method 100 before the bitstream is transferred to a decoder such as the decoder 400. In some examples, the bitstream may be transferred sequentially through the HRD500 as the bitstream is encoded. If a portion of the bitstream does not conform to the associated constraints, the HRD500 can indicate such failure to the encoder, causing the encoder to re-encode the corresponding section of the bitstream using a different mechanism.

[0097] The HRD500 includes a Virtual Stream Scheduler (HSS) 541. The HSS 541 is a component configured to run a virtual distribution mechanism. The virtual distribution mechanism is used to check the conformity of a bitstream or decoder with respect to the timing and data flow of the bitstream 551 input to the HRD500. For example, the HSS 541 can receive the bitstream 551 output from an encoder and manage the conformity testing process for the bitstream 551. In a specific example, the HSS 541 can control the rate at which an encoded image moves through the HRD500 and verify that the bitstream 551 does not contain any non-conforming data.

[0098] HSS541 can transfer bitstream 551 to CPB543 at a predetermined rate. HRD500 can manage data in decoding unit (DU) 553. DU553 is an AU or a subset of AUs, and associated non-video coding layer (VCL) network abstraction layer (NAL) units. Specifically, an AU contains one or more images associated with output time. For example, an AU can contain a single image in a single-layer bitstream, or an image for each layer in a multi-layer bitstream. Each image in an AU can be divided into slices, each contained in a corresponding VCL NAL unit. Thus, DU553 can contain one or more images, one or more slices of images, or a combination thereof. Parameters used to decode the AU, images, and / or slices can also be contained in non-VCL NAL units. Therefore, DU553 contains non-VCL NAL units containing the data necessary to support the decoding of VCL NAL units in DU553. CPB543 is a first-in, first-out buffer in HRD500. The CPB543 contains DU553s with video data in the decoded order. The CPB543 stores the video data for use during bitstream conformance verification.

[0099] CPB543 transfers DU553 to the decoding process component 545. The decoding process component 545 is a component compliant with the VVC standard. For example, the decoding process component 545 can emulate a decoder 400 used by an end user. The decoding process component 545 decodes DU553 at a rate achievable by an exemplary end-user decoder. If the decoding process component 545 cannot decode DU553 fast enough to prevent an overflow in CPB543, the bitstream 551 is non-compliant with the standard and needs to be re-encoded.

[0100] The decoding process component 545 decodes DU553 and produces the decoded DU555. The decoded DU555 contains the decoded image. The decoded DU555 is transferred to DPB547. DPB547 may be substantially similar to the decoded image buffer components 223, 323, and / or 423. To support interpretation, an image marked to be used as a reference image 556 obtained from the decoded DU555 is returned to the decoding process component 545 to support further decoding. DPB547 outputs the decoded video sequence as a series of images 557. Images 557 are reconstructed images that are a mirror image of the images encoded by the encoder into the bitstream 551.

[0101] Image 557 is transferred to the output cropping component 549. The output cropping component 549 is configured to apply a suitable cropping window to image 557. This yields the cropped output image 559. The cropped output image 559 is a fully reconstructed image. Thus, the cropped output image 559 mimics what the end user would expect to see when decoding the bitstream 551. In this way, the encoder can review the cropped output image 559 to ensure that the encoding is satisfactory.

[0102] The HRD500 is initialized based on HRD parameters in the bitstream 551. For example, the HRD500 can read HRD parameters from VPS, SPS, and / or SEI messages. The HRD500 can then perform conformity testing operations on the bitstream 551 based on the information in such HRD parameters. Specifically, the HRD500 can determine one or more CPB delivery schedules 561 from the HRD parameters. The delivery schedules specify the timing for the delivery of video data to and from memory locations such as CPB and / or DPB. Thus, the CPB delivery schedule 561 specifies the timing for the delivery of AU, DU553, and / or images to and from the CPB543. For example, the CPB delivery schedule 561 can describe the bitrate and buffer size of the CPB543, such bitrate and buffer size corresponding to a particular class of decoder and / or network conditions. Thus, the CPB delivery schedule 561 can indicate how long data can remain in the CPB543 before evacuation. The inability of the HRD500 to maintain the CPB distribution schedule 561 during conformance testing indicates that the decoder corresponding to the CPB distribution schedule 561 is unable to decode the corresponding bitstream. Note that the HRD500 can use a DPB distribution schedule similar to the CPB distribution schedule 561 for the DPB547.

[0103] Video may be coded into different layers and / or OLS for use by decoders with varying levels of hardware capabilities and for various network conditions. The CPB distribution schedule 561 is selected to reflect these challenges. Thus, the sub-bitstreams of the upper layers are specified for optimal hardware and network conditions, and the upper layers can receive one or more CPB distribution schedules 561 using a large amount of memory in the CPB 543 and a short delay for the transfer of the DU 553 toward the DPB 547. Similarly, the sub-bitstreams of the lower layers are specified for limited decoder hardware capabilities and / or poor network conditions. Thus, the lower layers can receive one or more CPB distribution schedules 561 using a small amount of memory in the CPB 543 and a longer delay for the transfer of the DU 553 toward the DPB 547. The OLS, layers, sublayers, or combinations thereof can then be tested according to the corresponding distribution schedule 561 to ensure that the resulting sub-bitstreams can be correctly decoded under the conditions expected for the sub-bitstreams. Each CPB delivery schedule 561 is associated with a schedule index (ScIdx) 563. ScIdx 563 is an index that identifies the delivery schedule. Therefore, the HRD parameters in bitstream 551 can not only indicate the CPB delivery schedule 561 by ScIdx 563, but can also contain enough data to enable HRD 500 to determine the CPB delivery schedule 561 and correlate it with the corresponding OLS, layer, and / or sublayer.

[0104] Figure 6 is a schematic diagram showing an exemplary multilayer video sequence 600 configured for interlayer prediction 621. The multilayer video sequence 600 may be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Furthermore, the multilayer video sequence 600 can be checked for compliance with standards by an HRD such as HRD 500. The multilayer video sequence 600 is included to illustrate an exemplary application of layers in an encoded video sequence. The multilayer video sequence 600 is an arbitrary video sequence using multiple layers such as layer N631 and layer N+1632.

[0105] In one example, a multilayer video sequence 600 may employ interlayer prediction 621. Interlayer prediction 621 is applied between images 611, 612, 613, and 614 of different layers and images 615, 616, 617, and 618. In the illustrated example, images 611, 612, 613, and 614 are part of layer N+1 632, and images 615, 616, 617, and 618 are part of layer N631. Layers such as layer N631 and / or layer N+1 632 are groups of images all associated with similar characteristics such as similar size, quality, resolution, signal-to-noise ratio, and capability. A layer can be formally defined as a set of VCL NAL units and associated non-VCL NAL units. A VCL NAL unit is a NAL unit coded to contain video data, such as an coded slice of an image. A non-VCL NAL unit is a NAL unit that contains non-video data such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations.

[0106] In the illustrated example, layer N+1 632 is associated with a larger image size than layer N631. Therefore, in this example, images 611, 612, 613, and 614 of layer N+1 632 have a larger image size (e.g., greater height and width, and thus more samples) than images 615, 616, 617, and 618 of layer N631. However, such images can be separated between layer N+1 632 and layer N631 by other characteristics. Although only two layers, layer N+1 632 and layer N631, are shown, a set of images can be separated into any number of layers based on associated characteristics. Layers N+1 632 and N631 may also be represented by layer IDs. A layer ID is a data item associated with an image that indicates that the image is part of a specified layer. Therefore, each image 611-618 can be associated with a corresponding layer ID to indicate which layer N+1 632 or layer N631 contains the corresponding image. For example, a layer ID can include a NAL unit header layer identifier (nuh_layer_id), which is a syntax element that specifies the identifier of a layer containing NAL units (e.g., slices and / or parameters of the image within the layer). Layers associated with lower quality / bitstream sizes, such as layer N631, are generally assigned lower layer IDs and are called lower layers. Furthermore, layers associated with higher quality / bitstream sizes, such as layer N+1 632, are generally assigned higher layer IDs and are called upper layers.

[0107] Images 611-618 from different layers 631-632 are configured to be displayed alternately. Therefore, images from different layers 631-632 can share a time ID 622, as long as these images are included in the same AU. A time ID 622 is a data element that indicates that the data corresponds to a time position in the video sequence. An AU is a set of NAL units associated with one particular output time, related to each other according to a given classification rule. For example, an AU can contain one or more images in different layers, such as images 611 and 615, if such images are associated with the same time ID 622. Specifically, the decoder can decode and display image 615 at the current display time if a smaller image is desired, or the decoder can decode and display image 611 at the current display time if a larger image is desired. Thus, images 611-614 from the upper layer N+1 632 contain substantially the same image data as the corresponding images 615-618 from the lower layer N631 (despite the difference in image size). Specifically, image 611 contains substantially the same image data as image 615, image 612 contains substantially the same image data as image 616, and so on.

[0108] Images 611-618 can be coded by referencing other images 611-618 from the same layer N631 or N+1 632. Coding an image by referencing another image from the same layer results in an interpretation 623. Interpretations 623 are depicted by solid arrows. For example, image 613 may be coded by employing an interpretation 623 that references one or two of images 611, 612, and / or 614 from layer N+1 632, one image referenced for unidirectional interpretation and / or two images referenced for bidirectional interpretation. Furthermore, image 617 may be coded by employing an interpretation 623 that references one or two of images 615, 616, and / or 618 from layer N531, one image referenced for unidirectional interpretation and / or two images referenced for bidirectional interpretation. When performing interprediction 623, if an image is used as a reference to another image on the same layer, that image is sometimes called a reference image. For example, image 612 may be a reference image used to encode image 613 according to interprediction 623. Interprediction 623 is sometimes called intralayer prediction in a multilayer context. Therefore, interprediction 623 is a mechanism for encoding a sample of the current image by referencing a specified sample in a reference image that is different from the current image, and the reference image and the current image are on the same layer.

[0109] Images 611-618 can also be coded by referencing other images 611-618 in different layers. This process is known as inter-layer prediction 621 and is illustrated by a dashed arrow. Inter-layer prediction 621 is a mechanism for coding a sample of the current image by referencing an indicated sample in a reference image, where the current image and the reference image are in different layers and therefore have different layer IDs. For example, an image in lower layer N631 can be used as a reference image to code the corresponding image in upper layer N+1 632. Specifically, image 611 can be coded by referencing image 615 according to inter-layer prediction 621. In such a case, image 615 is used as the inter-layer reference image. The inter-layer reference image is the reference image used in inter-layer prediction 621. In most cases, inter-layer prediction 621 is constrained so that the current image, such as image 611, can only use inter-layer reference images in lower layers, such as image 615, that are contained within the same AU. If multiple layers (e.g., three or more) are available, the inter-layer prediction 621 can encode / decode the current image based on multiple inter-layer reference images at lower levels than the current image.

[0110] The video encoder can encode images 611-618 using a multilayer video sequence 600 via many different combinations and / or permutations of inter-prediction 623 and inter-layer prediction 621. For example, image 615 may be coded according to intra-prediction. Then images 616-618 can be coded according to inter-prediction 623 by using image 615 as a reference image. Furthermore, image 611 may be coded according to inter-layer prediction 621 by using image 615 as an inter-layer reference image. Then images 612-614 can be coded according to inter-prediction 623 by using image 611 as a reference image. Thus, the reference image can play the role of both a single-layer reference image and an inter-layer reference image for different coding mechanisms. By coding the images of upper layers N+1 632 based on the images of lower layers N 631, upper layers N+1 632 can avoid employing intra-prediction, which is far less coding efficient than inter-prediction 623 and inter-layer prediction 621. Therefore, the low coding efficiency of intra prediction can be limited to minimum / lowest quality images, and thus to coding of the minimum amount of video data. Images used as reference images and / or inter-layer reference images can be indicated by entries in the reference image list contained in the reference image list structure.

[0111] To perform this operation, layers such as layer N631 and layer N+1 632 may be included in one or more OLS625 and 626. Specifically, images 611-618 are encoded as layers 631-632 of bitstream 600, and then each layer 631-632 of the images is assigned to one or more of OLS625 and 626. Then, OLS625 and / or 626 can be selected, and depending on the capabilities of the decoder and / or network conditions, the corresponding layers 631 and / or 632 can be sent to the decoder. OLS625 is a set of layers in which one or more layers are designated as output layers. Output layers are layers designated to be output (e.g., to a display). For example, layer N631 may be included only to support inter-layer prediction 621 and may never be output. In such a case, layer N+1 632 is decoded based on layer N631 and output. In such cases, OLS625 includes layer N+1 632 as output layers. When an OLS contains only output layers, it is called the 0th OLS626. The 0th OLS626 is an OLS that contains only the lowest layer (the layer having the lowest layer identifier), and therefore an OLS that contains only output layers. In other cases, OLS625 may contain many layers in different combinations. For example, the output layers of OLS625 can be coded according to inter-layer prediction 621 based on one, two, or many lower layers. Furthermore, OLS625 can contain two or more output layers. Thus, OLS625 can contain one or more output layers and any support layers necessary to reconstruct the output layers. Although only two OLS625 and 626 are shown, a multi-layer video sequence 600 may be coded by employing many different OLS625 and / or 626, each employing a different combination of layers. OLS625 and 626 are each associated with OLS index 629, which is an index that uniquely identifies the corresponding OLS625 and 626.

[0112] Checking the compliance of a multilayer video sequence 600 with the HRD500 can be complex depending on the number of layers 631-632 and OLS 625 and 626. The HRD500 can separate the multilayer video sequence 600 into a series of operating points 627 for testing. OLS 625 and / or 626 are identified by OLS index 629. An operating point 627 is a temporal subset of OLS 625 / 626. An operating point 627 can be identified by both the OLS index 629 and the highest time ID 622 of the corresponding OLS 625 / 626. For example, a first operating point 627 may contain all images in the first OLS 625 from time ID 0 to time ID 200, a second operating point 627 may contain all images in the first OLS 625 from time ID 201 to time ID 400, and so on. In this case, the first operating point 627 is described by the OLS index 629 of the first OLS 625 and a time ID of 200. Furthermore, the second operating point 627 is described by the OLS index 629 of the first OLS 625 and a time ID of 400. The operating point 627 selected to be tested at a specified moment is called the target Op. Thus, the target Op is the operating point 627 selected for conformity testing in HRD500.

[0113] Figure 7 is a schematic diagram showing an exemplary multilayer video sequence 700 configured for time scalability. The multilayer video sequence 700 can be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Furthermore, the multilayer video sequence 700 can be checked for compliance with standards by an HRD such as HRD 500. The multilayer video sequence 700 is included to demonstrate another exemplary application for layers in coded video sequences. For example, the multilayer video sequence 700 may be employed as a separate embodiment or in combination with the techniques described with respect to the multilayer video sequence 600.

[0114] The multilayer video sequence 700 includes sublayers 710, 720, and 730. A sublayer is a time-scalable layer of a time-scalable bitstream containing VCL NAL units (e.g., images) with specific time identifier values, as well as associated non-VCL NAL units (e.g., support parameters). For example, layers such as layer N631 and / or layer N+1632 may be further divided into sublayers 710, 720, and 730 to support time scalability. Sublayer 710 may be called the base layer, and sublayers 720 and 730 may be called enhancement layers. As illustrated, sublayer 710 contains images 711 at a first frame rate, such as 30 frames per second. Since sublayer 710 contains the base / minimum frame rate, sublayer 710 is the base layer. Sublayer 720 contains images 721 that are time-offset from images 711 of sublayer 710. As a result, sublayers 710 and 720 can be combined, resulting in a higher overall frame rate than that of sublayer 710 alone. For example, sublayers 710 and 720 together can have a frame rate of 60 frames per second. Thus, sublayer 720 improves the frame rate of sublayer 710. Furthermore, sublayer 730 includes image 731 which is temporally offset from images 721 and 711 of sublayers 720 and 710. Thus, sublayer 730 can be combined with sublayers 720 and 710 to further enhance sublayer 710. For example, sublayers 710, 720, and 730 together can have a frame rate of 90 frames per second.

[0115] A sublayer representation 740 can be dynamically created by combining sublayers 710, 720, and / or 730. A sublayer representation 740 is a subset of a bitstream containing NAL units of a particular sublayer and its lower sublayers. In the illustrated example, the sublayer representation 740 includes image 741, which is a composite image 711, 721, and 731 of sublayers 710, 720, and 730. Thus, a multilayer video sequence 700 can be temporally scaled to a desired frame rate by selecting a sublayer representation 740 containing a desired set of sublayers 710, 720, and / or 730. The sublayer representation 740 may also be created by employing an OLS that includes sublayers 710, 720, and / or 730 as layers. In such a case, the sublayer representation 740 is selected as the output layer. Therefore, temporal scalability is one of several mechanisms that can be achieved using the multilayer mechanism.

[0116] Figure 8 is a schematic diagram showing an exemplary bitstream 800. For example, bitstream 800 can be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, bitstream 800 may include multilayer video sequences 600 and / or 700. In addition, bitstream 800 may include various parameters for controlling the operation of an HRD, such as HRD500. Based on such parameters, the HRD can check bitstream 800 for compliance with standards before sending it to the decoder for decoding.

[0117] Bitstream 800 includes a VPS811, one or more SPS813s, multiple Image Parameter Sets (PPS)815s, multiple slice headers817s, image data820, and SEI messages819. The VPS811 contains data relevant to the entire bitstream 800. For example, the VPS811 may include data related to the OLS, layers, and / or sublayers used in the bitstream 800. The SPS813 contains sequence data common to all images within the coded video sequences included in the bitstream 800. For example, each layer may contain one or more coded video sequences, and each coded video sequence may reference the SPS813 for its corresponding parameters. Parameters in the SPS813 may include image sizing, bit depth, coding tool parameters, bitrate limits, etc. While each sequence references the SPS813, note that in some examples, a single SPS813 may contain data for multiple sequences. The PPS815 contains parameters applicable to the entire image. Therefore, each image in a video sequence can reference a PPS815. While each image references a PPS815, it should be noted that in some examples, a single PPS815 can contain data for multiple images. For example, multiple similar images may be coded according to similar parameters. In such cases, a single PPS815 can contain data for such similar images. The PPS815 can indicate the coding tools, quantization parameters, offsets, etc., available for the corresponding image slices.

[0118] The slice header 817 contains parameters specific to each slice in the image. Therefore, there may be one slice header 817 for each slice in a video sequence. The slice header 817 can include slice type information, POC, reference image list, prediction weights, tile entry points, deblocking parameters, etc. It should be noted that in some examples, the bitstream 800 may also include an image header, which is a syntactic structure containing parameters applicable to all slices in a single image. For this reason, the image header and slice header 817 may be used interchangeably in some contexts. For example, certain parameters may be moved between the slice header 817 and the image header depending on whether such parameters are common to all slices in the image.

[0119] Image data 820 includes video data encoded according to inter-prediction and / or intra-prediction, as well as corresponding transformed and quantized residual data. For example, image data 820 may include AU821, DU822, and / or image 823. AU821 is a set of NAL units related to one particular output time, associated with one another according to a given classification rule. DU822 is an AU or a subset of an AU, and associated non-VCL NAL units. Image 823 is an array of lumens and / or chromens that make up a frame or its fields. Simply put, AU821 includes various video data that may appear at a given moment in a video sequence, as well as supporting syntax data. Thus, AU821 may include a single image 823 in a single-layer bitstream, or multiple images from multiple layers, all associated at the same moment in a multi-layer bitstream. Image 823, on the other hand, is a coded image that may be output for display or used to support the coding of other images 823 for output. DU822 can contain one or more images 823 and any supporting syntax data necessary for decoding. For example, DU822 and AU821 may be used interchangeably in a simple bitstream (for example, if the AU contains a single image). However, in more complex multi-layer bitstreams, DU822 may contain only a portion of the video data from AU821. For example, AU821 may contain images 823 in several layers and / or sublayers, some of which are associated with different OLSs. In such cases, DU822 may contain only images 823 from the specified OLS and / or specified layers / sublayers.

[0120] Image 823 contains one or more slices 825. A slice 825 may be defined as an integer number of complete tiles (e.g., within a tile) or an integer number of consecutive complete coding tree unit (CTU) rows in Image 823, where a tile or CTU row is exclusively contained within a single NAL unit 829. Thus, a slice 825 is also contained within a single NAL unit 829. A slice 825 is further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a group of samples of a predefined size that can be divided by a coding tree. A CTB is a subset of a CTU and contains the lumen or chroma components of the CTU. A CTU / CTB is further divided into coding blocks based on the coding tree. The coding blocks can then be encoded / decoded according to a prediction mechanism.

[0121] Bitstream 800 is a sequence of NAL units 829. A NAL unit 829 is a container for video data and / or supporting syntax. A NAL unit 829 can be a VCL NAL unit or a non-VCL NAL unit. A VCL NAL unit is a NAL unit 829 coded to contain video data, such as a coded slice 825 and an associated slice header 817. A non-VCL NAL unit is a NAL unit 829 containing non-video data, such as syntax and / or parameters that support decoding the video data, performing conformance checks, or other operations. For example, a non-VCL NAL unit may contain VPS811, SPS813, PPS815, SEI messages 819, or other supporting syntax.

[0122] SEI message 819 is a semantically specified syntactic structure that conveys information not required by the decoding process to determine the sample values ​​of the decoded image. For example, an SEI message may include data to support the HRD process or other supporting data not directly related to decoding bitstream 800 in the decoder. SEI message 819 can include scalable nesting SEI messages and / or non-scalable nested SEI messages. A scalable nesting SEI message is a message containing multiple SEI messages corresponding to one or more OLS or one or more layers. A non-scalable nested SEI message is a message that is not nested and therefore contains a single SEI message. SEI message 819 may include a BP SEI message containing HRD parameters for initializing HRD to manage CPB. SEI message 819 may also include a PT SEI message containing HRD parameters for managing delivery information about AU821 in CPB and / or DPB. SEI message 819 may also include a DUI SEI message containing HRD parameters for managing distribution information about DU822 in the CPB and / or DPB.

[0123] Bitstream 800 contains an integer(i) set of sequence-level HRD parameters 833, which are syntax elements that initialize and / or define the operating conditions of an HRD, such as HRD500. In some examples, the general_hrd_parameters syntax structure may include sequence-level HRD parameters 833 that apply to all OLS specified by VPS811. Sequence-level HRD parameters are HRD parameters that apply to the entire coded video sequence. A coded video sequence is one or more images 823. In one example, an encoder may encode the video sequence into layers. Each layer may contain a coded video sequence. The encoder can then encode the sequence-level HRD parameters 833 into a bitstream to properly configure the HRD to perform conformance checks on the sequence in the OLS. HRD parameters 833 may also indicate to the decoder that the decoder can decode the bitstream according to the delivery schedule. Sequence-level HRD parameters 833 are included in VPS811. Additional parameters used to configure the HRD may also be included in SEI message 819.

[0124] As mentioned above, a video stream can contain many OLSs and many layers, such as OLS625, layer N631, layer N+1 632, sublayer 710, sublayer 720, and / or sublayer 730. Furthermore, some layers may be contained within multiple OLSs. Each layer and / or sublayer of each OLS is checked for suitability by the HRD according to the delivery schedule. Each delivery schedule is associated with different CPB sizes and CPB delays to account for different transmission bandwidths and system capabilities. Some video coding systems allow each sublayer to define any number of delivery schedules. This can result in a large amount of signaling to support suitability checks, which in turn reduces the coding efficiency of the bitstream. Moreover, such flexibility creates significant complexity, resulting in a substantial increase in processor and memory resource usage in the encoder.

[0125] This disclosure includes a mechanism for simplifying HRD signaling for videos containing multiple layers and improving coding efficiency. Specifically, all layers and / or sublayers are constrained to contain the same number of CPB delivery schedules. For example, an encoder can determine the maximum number of CPB delivery schedules used for any one layer and set the number of CPB delivery schedules for all layers to this maximum. The number of delivery schedules may then be signaled once, for example, as part of the HRD parameters in a VPS. This eliminates the need to signal several schedules per layer / sublayer. Also, all layers / sublayers within the OLS can share the same delivery schedule index. These changes reduce the amount of data used to signal data related to compliance checks. This results in a smaller bitstream size and therefore reduces the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0126] Specifically, the HRD parameter 833 of VPS811 can contain an HRD CPB count, which can be expressed as hrd_cpb_cnt_minus1 831. hrd_cpb_cnt_minus1 831 is a syntax element that specifies the number of CPB delivery schedules used in bitstream 800. minus1 indicates that the value of hrd_cpb_cnt_minus1 831 is 1 less than the indicated value, and that +1 must be added to obtain the true value. hrd_cpb_cnt_minus1 831 may be in the range of 0 to 31, including both ends. This indicates that 1 to 32 CPB delivery schedules can be specified for bitstream 800.

[0127] SEI message 819 may contain data describing the CPB delivery schedule indicated in HRD parameter 833. For example, a BP SEI message may contain BP CPB count -1 (bp_cpb_cnt_minus1) 835, initial CPB removal delay (nal_initial_cpb_removal_delay) 837, and initial CPB removal offset (nal_initial_cpb_removal_offset) 839, which can be used to describe the CPB delivery schedule for the corresponding layer. Each CPB delivery schedule is described by the pair of nal_initial_cpb_removal_delay837 and nal_initial_cpb_removal_offset839. nal_initial_cpb_removal_delay837 is the default removal delay used when removing data units from the CPB in HRD when checking images, AUs, and / or DUs in bitstreams, OLS, and / or layers. Therefore, nal_initial_cpb_removal_delay837 describes the timing of the CPB delivery schedule. nal_initial_cpb_removal_offset839 is the default CPB removal offset associated with each image, AU, and / or DU within the bitstream, OLS, and / or layer. For example, the offset can indicate the size of the data units removed from the CPB in HRD. Thus, nal_initial_cpb_removal_offset839 describes the cache size used in the CPB delivery schedule. Therefore, HRD can determine the CPB delivery schedule based on nal_initial_cpb_removal_delay837 and nal_initial_cpb_removal_offset839. Thus, such pairs of values ​​are used to define each CPB delivery schedule, such as CPB delivery schedule 561.

[0128] bp_cpb_cnt_minus1 835 is a syntax element that specifies the number of pairs of nal_initial_cpb_removal_delay837 and nal_initial_cpb_removal_offset839 used for the corresponding time sublayers, such as sublayers 710, 720, and / or 730. Thus, bp_cpb_cnt_minus1 835 specifies the number of delivery schedules available per layer / sublayer. The value of bp_cpb_cnt_minus1 835 can be set to be equal to the value of hrd_cpb_cnt_minus1 831. This ensures that the number of delivery schedules for each layer is the same as the total number of CPB delivery schedules. Thus, all layers have the same number of CPB delivery schedules. Ensuring that all layers have the same number of CPB delivery schedules reduces complex and / or redundant signaling within bitstream 800, thereby increasing coding efficiency. This technique also simplifies the HRD process. As a result, the usage of processor, memory, and / or network signaling resources is reduced in both the encoder and decoder. It should also be noted that each CPB delivery schedule in the target layer / OLS can be referenced by a unique schedule index (ScIdx). ScIdx can include values ​​ranging from 0 to hrd_cpb_cnt_minus1 831, including both end values. This means that ScIdx can vary from a value of 0 for the first CPB delivery schedule to a value of hrd_cpb_cnt_minus1 for the last CPB delivery schedule. Since each layer / sublayer contains the same number of CPB delivery schedules, and ScIdx can uniquely identify any CPB delivery schedule, layers / sublayers within the OLS can also share the same set of ScIdx values. This further reduces the complexity of the HRD process. Thus, the mechanism described for bitstream 800 can enhance the functionality of the encoder and / or decoder.Furthermore, the mechanism described for Bitstream 800 can support improved coding efficiency and / or a reduction in processor, memory, and / or network communication resources in the encoder and / or decoder.

[0129] Hereinafter, the information described above will be explained in more detail below in this specification. Layered video coding is also called scalable video coding or video coding with scalability. Scalability in video coding can be supported by using multilayer coding techniques. A multilayer bitstream includes a base layer (BL) and one or more enhancement layers (EL). Examples of scalability include spatial scalability, signal-to-noise ratio (SNR) scalability, multiview scalability, and frame rate scalability. When using multilayer coding techniques, an image or part thereof may be coded without using a reference image (intra-prediction), coded by referencing a reference image in the same layer (inter-prediction), and / or coded by referencing a reference image in another layer (inter-layer prediction). The reference image used for inter-layer prediction of the current image is called the interlayer reference image (ILRP). Figure 6 shows an example of multilayer coding for spatial scalability where images in different layers have different resolutions.

[0130] Several video coding families offer support for profile scalability, separated from profiles for single-layer coding. Scalable Video Coding (SVC) is a scalable extension of Advanced Video Coding (AVC) that supports spatial, temporal, and quality scalability. In SVC, each macroblock (MB) in an EL image is signaled with a flag indicating whether the EL MB is predicted using a collocation block from a lower layer. Predictions from a collocation block can include texture, motion vectors, and / or coding modes. An implementation of SVC cannot directly reuse an unmodified implementation of AVC in the design. The syntax and decoding processes for SVC EL macroblocks differ from those for AVC.

[0131] Scalable HEVC (SHVC) is an extension of HEVC that supports spatial and quality scalability. Multiview HEVC (MV-HEVC) is an extension of HEVC that supports multiview scalability. 3D HEVC (3D-HEVC) is an extension of HEVC that supports more advanced and efficient 3D video coding than MV-HEVC. Temporal scalability may be included as an integral part of the single-layer HEVC codec. In the multi-layer extensions of HEVC, the decoded images used for inter-layer prediction are only those from the same AU and are treated as long-term reference images (LTRPs). Such images, along with other temporal reference images of the current layer, are assigned a reference index in the reference image list. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the reference index value to refer to the inter-layer reference image in the reference image list. Spatial scalability is achieved by resampling the reference image or a portion thereof if the ILRP has a different spatial resolution than the current image being encoded or decoded. Resampling of a reference image can be achieved at either the image level or the coding block level.

[0132] VVC can also support layered video coding. A VVC bitstream can contain multiple layers. All layers may be independent of each other. For example, each layer can be coded without using inter-layer prediction. In this case, the layer is also called a simulcast layer. In some cases, some of the layers are coded using ILP. A flag in the VPS can indicate whether a layer is a simulcast layer or whether some layers use ILP. If some layers use ILP, the layer dependencies between layers are also signaled in the VPS. Unlike SHVC and MV-HEVC, VVC may not specify an OLS. The OLS contains a specified set of layers, one or more of which are specified as output layers. The output layers are the layers of the output OLS. In some embodiments of VVC, if a layer is a simulcast layer, only one layer may be selected for decoding and output. In some embodiments of VVC, the entire bitstream containing all layers is specified to be decoded if any of the layers use ILP. In addition, a specific layer among the layers is specified as the output layer. The output layers may be instructed to be only the top layer, all layers, or the top layer plus a set of specified lower layers.

[0133] Video coding standards can specify HRDs for verifying bitstream conformance through designated HRD conformance tests. SHVC and MV-HEVC employ three sets of bitstream conformance tests to check bitstream conformance. The bitstream is referred to as the entire bitstream and is denoted as entireBitstream. The first set of bitstream conformance tests is for testing the conformance of the entire bitstream and its corresponding time subset. Such tests are employed regardless of whether there is a layer set specified by the active VPS containing all nuh_layer_id values ​​for VCL NAL units present in the entire bitstream. Therefore, the entire bitstream is always checked for conformance, even if one or more layers are not included in the output set. The second set of bitstream conformance tests is employed to test the conformance of the layer set and associated time subset specified by the active VPS. For all of these tests, only the base layer image (e.g., an image where nuh_layer_id is equal to 0) is decoded and output. Other images are ignored by the decoder when the decoding process is invoked. A third set of bitstream compliance tests is employed to test the compliance of the OLS and associated time subsets specified by the VPS extension portion of the active VPS, based on the OLS and bitstream partitioning. Bitstream partitioning includes one or more layers of the OLS of a multilayer bitstream.

[0134] The aforementioned embodiments involve certain problems. For example, the first two sets of conformance testing may apply to layers that are not decoded and are not output. For instance, layers other than the lowest layer are not decoded and are not output. In a real application, the decoder can only receive data that is decoded. Therefore, using the first two sets of conformance testing both complicates the codec design and can waste bits for carrying both sequence-level and image-level parameters used to support conformance testing. A third set of conformance testing involves bitstream splitting. Such splitting may relate to one or more layers of the OLS of a multilayer bitstream. Alternatively, if conformance testing always operates separately for each layer, the HRD can be greatly simplified.

[0135] Signaling of sequence-level HRD parameters can be complex. For example, sequence-level HRD parameters may be signaled in multiple locations, such as both SPS and VPS. Furthermore, sequence-level HRD parameter signaling can involve redundancy. For instance, information that might be generally the same across the entire bitstream may be repeated at each layer of each OLS. Additionally, exemplary HRD schemes allow for the selection of different delivery schedules per layer. Such delivery schedules may be selected from a list of signaled schedules for each layer at each operating point, where the operating point is an OLS or a time subset of an OLS. Such systems are complex. Furthermore, exemplary HRD schemes allow for the association of incomplete AUs with buffered-period SEI messages. An incomplete AU is an AU that lacks an image for all layers present in the CVS. However, HRD initialization in such AUs can be problematic. For example, HRD may not be properly initialized for layers that have layer access units that are not present in the incomplete AU. In addition, the demultiplexing process for deriving layer bitstreams may not be sufficiently efficient at removing nested SEI messages that do not apply to the target layer. Layer bitstreams occur when the bitstream splitting contains only one layer. Furthermore, the applicable OLS for non-scalable nested buffering periods, image timings, and decoding unit information SEI messages may be specified for the entire bitstream. However, the non-scalable nested buffering period should instead be applicable to the 0th OLS.

[0136] Furthermore, depending on the VVC implementation, it may fail to infer HDR parameters when sub_layer_cpb_params_present_flag is equal to 0. Such inference may enable proper HRD operation. In addition, the values ​​of bp_max_sub_layers_minus1 and pt_max_sub_layers_minus1 may need to be equal to the value of sps_max_sub_layers_minus1. However, buffering period and image timing SEI messages may be nested and applicable to multiple OLSs and multiple layers of each of multiple OLSs. In such contexts, the layers involved may refer to multiple SPSs. Therefore, it may be difficult for the system to track which SPS corresponds to each layer. Thus, the values ​​of these two syntax elements should instead be constrained based on the value of vps_max_sub_layers_minus1. Furthermore, since different layers can have different numbers of sublayers, the values ​​of these two syntax elements are not always equal to a specific value in all buffering period and image timing SEI messages.

[0137] Furthermore, the following issues are associated with HRD design in both SHVC / MV-HEVC and VVC: The sub-bitstream extraction process may not remove SEI NAL units containing nested SEI messages that are not required for the target OLS.

[0138] Generally, this disclosure describes a technique for scalable nesting of SEI messages for output layer sets in a multilayer video bitstream. The description of this technique is based on VVC. However, this technique is also applicable to layered video coding based on other video codec specifications.

[0139] One or more of the above-mentioned problems can be solved as follows. Specifically, this disclosure includes a method and related embodiments for HRD design that enables efficient signaling of HRD parameters using HRD operation that is much simpler compared to SHVC and MV-HEVC. Each of the solutions described below addresses the above-mentioned problems. For example, instead of requiring three sets of conformance tests, this disclosure requires only one set of conformance tests to test the conformance of an OLS specified by the VPS. Furthermore, instead of a bitstream partitioning-based design, the disclosed HRD mechanism can always operate separately for each layer of the OLS. In addition, sequence-level HRD parameters that are global for all layers and sublayers of all OLSs only need to be signaled once in the VPS, for example. In addition, a single number of delivery schedules need to be signaled for all layers and sublayers of all OLSs. The same delivery schedule index can also be applied to all layers of the OLS. Furthermore, incomplete AUs do not need to be associated with buffering period SEI messages. An incomplete AU is an AU that does not contain images for all layers present in the CVS. This ensures that the HRD is always properly initialized for all layers of the OLS. Furthermore, a mechanism is disclosed for efficiently removing nested SEI messages that are not applicable to the target layer in the OLS. This supports a demultiplexing process for deriving the layer bitstream. In addition, the OLS to which non-scalable nested buffering periods, image timings, and decoded unit information SEI messages are applicable may be specified as the 0th OLS. Furthermore, if sub_layer_cpb_params_present_flag is equal to 0, HDR parameters can be inferred, potentially enabling proper HRD operation. The values ​​of bp_max_sub_layers_minus1 and pt_max_sub_layers_minus1 may need to be in the range of 0 to vps_max_sub_layers_minus1.Thus, these parameters do not need to be specific values ​​for all buffering periods and image timing SEI messages. Furthermore, the sub-bitstream extraction process can remove SEI NAL units containing nested SEI messages that are not applicable to the target OLS.

[0140] An exemplary embodiment of the aforementioned mechanism is as follows: The output layer is the output layer from the output layer set. The OLS is a set of layers containing a specified set of layers, one or more layers in the set of layers are designated as output layers. The OLS layer index is the index of a layer in the OLS to the list of layers in the OLS. The sub-bitstream extraction process is a specified process in which NAL units in a bitstream that do not belong to the target set, determined by the target OLS index and the target highest TemporalId, are removed from the bitstream, and the output sub-bitstream contains NAL units in the bitstream that belong to the target set.

[0141] The syntax for an example video parameter set is as follows: [Table 1]

[0142] The syntax for an exemplary sequence parameter set RBSP is as follows: [Table 2]

[0143] The syntax for an example DPB parameter is as follows: [Table 3]

[0144] The syntax for an example of a general HRD parameter is as follows: [Table 4]

[0145] The semantics of the exemplary video parameter set RBSP are as follows: Setting each_layer_is_an_ols_flag to equal 1 specifies that each output layer set contains only one layer, and each layer in the bitstream is itself an output layer set, with the single layer it contains being the only output layer. Setting each_layer_is_an_ols_flag to equal 0 specifies that an output layer set can contain two or more layers. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 1. Otherwise, if vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 0.

[0146] Setting ols_mode_idc to 0 specifies that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including both endpoints), and for each OLS, only the top layer of the OLS is output. Setting ols_mode_idc to 1 specifies that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including both endpoints), and for each OLS, all layers of the OLS are output. Setting ols_mode_idc to 2 specifies that the total number of OLS specified by the VPS is explicitly signaled, and for each OLS, the top layer of the OLS and a set of explicitly signaled lower layers are output. The value of ols_mode_idc should be in the range of 0 to 2 (including both endpoints). The value of ols_mode_idc 3 is reserved. If vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, then the value of ols_mode_idc is presumed to be equal to 2. num_output_layer_sets_minus1+1 specifies the total number of OLS specified by the VPS, assuming ols_mode_idc is equal to 2.

[0147] The variable TotalNumOlss, which specifies the total number of OLS designated by the VPS, is derived as follows:

number

[0148] The `layer_included_flag[i][j]` specifies whether to include the j-th layer (the layer whose `nuh_layer_id` is equal to `vps_layer_id[j]`) in the i-th OLS when `ols_mode_idc` is equal to 2. Setting `layer_included_flag[i][j]` to equal 1 specifies that the j-th layer should be included in the i-th OLS. Setting `layer_included_flag[i][j]` to equal 0 specifies that the j-th layer should not be included in the i-th OLS.

[0149] The variables NumLayersInOls[i], which specify the number of layers in the i-th OLS, and LayerIdInOls[i][j], which specify the nuh_layer_id value of the j-th layer of the i-th OLS, are derived as follows:

number

[0150] The variable LayerIdInOls[i][j], which specifies the OLS layer index of the layer where nuh_layer_id is equal to OlsLayeIdx[i][j], is derived as follows:

number

[0151] The lowest layer in each OLS is assumed to be an independent layer. In other words, for each i in the range from 0 to TotalNumOlss-1, including both endpoints, the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] is equal to 1. Each layer is assumed to be contained in at least one OLS specified by the VPS. In other words, for each k in the range from 0 to vps_max_layers_minus1, including both endpoints, for each layer where a particular value of nuh_layer_id nuhLayerId is equal to one of vps_layer_id[k], there may be at least one pair of i and j values, where i is in the range from 0 to TotalNumOlss-1, including both endpoints, j is in the range up to NumLayersInOls[i]-1, including both endpoints, and the value of LayerIdInOls[i][j] is equal to nuhLayerId. Any layer in OLS shall be an output layer of OLS, or a (direct or indirect) reference layer of an output layer of OLS.

[0152] The `vps_output_layer_flag[i][j]` specifies whether the j-th layer of the i-th OLS is output when `ols_mode_idc` is equal to 2. `vps_output_layer_flag[i]` equal to 1 specifies that the j-th layer of the i-th OLS is output. Setting `vps_output_layer_flag[i]` equal to 0 specifies that the j-th layer of the i-th OLS is not output. When `vps_all_independent_layers_flag` is equal to 1 and `each_layer_is_an_ols_flag` is equal to 0, the value of `vps_output_layer_flag[i]` is presumed to be equal to 1. The variable `OutputLayerFlag[i][j]`, which specifies that the j-th layer of the i-th OLS is output when valued at 1 and not output when valued at 0, is derived as follows:

number

[0153] Setting vps_extension_flag to 0 specifies that the vps_extension_data_flag syntax element does not exist in the VPS RBSP syntax structure. Setting vps_extension_flag to 1 specifies that the vps_extension_data_flag syntax element exists in the VPS RBSP syntax structure. vps_extension_data_flag can have any value. The presence and value of vps_extension_data_flag do not affect the decoder's suitability for the specified profile. The decoder shall ignore all vps_extension_data_flag syntax elements.

[0154] The semantics of an example DPB parameter are as follows: The dpb_parameters() syntax structure provides DPB size information and, optionally, maximum image sort count and maximum latency (MRML) information. Each SPS contains one or more dpb_parameters() syntax structures. The first dpb_parameters() syntax structure of an SPS contains both DPB size information and MRML information. If present, the second dpb_parameters() syntax structure of an SPS contains only DPB size information. The MRML information of the first dpb_parameters() syntax structure of an SPS applies to the layer referencing the SPS, regardless of whether that layer is an OLS output layer. The DPB size information of the first dpb_parameters() syntax structure of an SPS applies to the layer referencing the SPS if that layer is an OLS output layer. The DPB size information contained in the second dpb_parameters() syntax structure of the SPS, if present, applies to the layer referencing the SPS if that layer is a non-output layer of the OLS. If the SPS contains only one dpb_parameters() syntax structure, it is inferred that the DPB size information for the layer as a non-output layer is the same as that for the layer as an output layer.

[0155] The semantics of exemplary general HRD parameters are as follows: The general_hrd_parameters() syntax structure provides the HRD parameters used in HRD operation. Setting sub_layer_cpb_params_present_flag equal to 1 specifies that the i-th layer_level_hrd_parameters() syntax structure contains HRD parameters for sublayer representations where TemporalId ranges from 0 to hrd_max_temporal_id[i], including both endpoints. Setting sub_layer_cpb_params_present_flag equal to 0 specifies that the i-th layer_level_hrd_parameters() syntax structure contains HRD parameters for sublayer representations where TemporalId is equal only to hrd_max_temporal_id[i]. If vps_max_sub_layers_minus1 is equal to 0, the value of sub_layer_cpb_params_present_flag is inferred to be equal to 0. If sub_layer_cpb_params_present_flag is equal to 0, the HRD parameters for sublayer representations where TemporalId is in the range of 0 to hrd_max_temporal_id[i]-1, including both endpoints, are presumed to be the same as those for sublayer representations where TemporalId is equal to hrd_max_temporal_id[i]. These include the HRD parameters in the layer_level_hrd_parameters syntax structure, starting from the fixed_pic_rate_general_flag[i] syntax element up to the sub_layer_hrd_parameters(i) syntax structure immediately below the conditional statement if(general_vcl_hrd_params_present_flag). num_layer_hrd_params_minus1+1 specifies the number of layer_level_hrd_parameters() syntax structures present in the general_hrd_parameters() syntax structure.The value of num_layer_hrd_params_minus1 shall be in the range of 0 to 63, including the endpoints. hrd_cpb_cnt_minus1+1 specifies the number of alternative CPB specifications in the CVS bitstream. The value of hrd_cpb_cnt_minus1 shall be in the range of 0 to 31, including the endpoints. hrd_max_temporal_id[i] specifies the TemporalId of the top-level sublayer representation in which the HRD parameters are included in the i-th layer_level_hrd_parameters() syntax structure. The value of hrd_max_temporal_id[i] shall be in the range of 0 to vps_max_sub_layers_minus1, including the endpoints. If vps_max_sub_layers_minus1 is equal to 0, the value of hrd_max_temporal_id[i] is presumed to be equal to 0. `layer_level_hrd_idx[i][j]` specifies the index of the `layer_level_hrd_parameters()` syntax structure applied to the j-th layer of the i-th OLS. The value of `layer_level_hrd_idx[[i][j]]`, including both endpoints, must be in the range of 0 to `num_layer_hrd_params_minus1`. If it does not exist, the value of `layer_level_hrd_idx[[0][0]` is presumed to be equal to 0.

[0156] An exemplary sub-bitstream extraction process is as follows: The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx, and the target highest TemporalId value tIdTarget. The output of this process is the sub-bitstream outBitstream. The bitstream conformance requirement for the input bitstream is that any output sub-bitstream is a conforming bitstream, which is the output of the process specified in this section with respect to the bitstream, with input targetOlsIdx equal to the index in the list of OLS specified by the VPS, and tIdTarget equal to any value in the range of 0 to 6, including the end values, and satisfies the following conditions: The output sub-bitstream should contain at least one VCL NAL unit where nuh_layer_id is equal to each of the nuh_layer_id values ​​in LayerIdInOls[targetOlsIdx]. The output sub-bitstream should contain at least one VCL NAL unit where TemporalId is equal to tIdTarget. A compatible bitstream contains one or more coded slice NAL units whose TemporalId is equal to 0, but does not need to contain any coded slice NAL units whose nuh_layer_id is equal to 0.

[0157] The output sub-bitstream OutBitstream is derived as follows: The bitstream outBitstream is set to be identical to the bitstream inBitstream. Remove all NAL units from outBitstream whose TemporalId is greater than tIdTarget. Remove all NAL units from outBitstream whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx]. Remove all SEI NAL units from outBitstream that contain scalable nesting SEI messages whose value of i is not in the range of 0 to nesting_num_olss_minus1, including both endpoints, such that nesting_ols_flag is equal to 1 and NestingOlsIdx[i] is equal to targetOlsIdx. If targetOlsIdx is greater than 0, remove all SEI NAL units from the outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information).

[0158] A typical example of an HRD is as follows. This section specifies HRDs and their use for checking bitstream and decoder conformance. A set of bitstream conformance tests is used to check the conformance of a bitstream called the entire bitstream, denoted as entireBitstream. A set of bitstream conformance tests is for testing the conformance of each OLS and time subsets of each OLS specified by the VPS. For each test, the following ordered steps are applied in the order listed.

[0159] The operating point under test, denoted as targetOp, is selected by selecting a target OLS having an OLS index opOlsIdx and the highest TemporalId value opTid. The value of opOlsIdx, including both endpoints, is in the range of 0 to TotalNumOlss-1. The value of opTid, including both endpoints, is in the range of 0 to vps_max_sub_layers_minus1. The values ​​of opOlsIdx and opTid are such that the sub-bitstream BitstreamToDecode, which is the output of calling the sub-bitstream extraction process with entireBitstream, opOlsIdx, and opTid as inputs, satisfies the following conditions: There is at least one VCL NAL unit where nuh_layer_id is equal to each of the nuh_layer_id values ​​of LayerIdInOls[opOlsIdx] in BitstreamToDecode. There is at least one VCL NAL unit where TemporalId is equal to opTid in BitstreamToDecode.

[0160] The values ​​of TargetOlsIdx and Htid are set to be equal to opOlsIdx and opTid of targetOp, respectively. The value of ScIdx is selected. The selected ScIdx shall be in the range of 0 to hrd_cpb_cnt_minus1, including both endpoints. The access unit in BitstreamToDecode associated with the buffering period SEI message applicable to TargetOlsIdx (present in TargetLayerBitstream or available through an external mechanism not specified herein) is selected as the HRD initialization point and is referred to as access unit 0 for each layer of the target OLS.

[0161] The subsequent steps apply to each layer in the target OLS that has an OLS layer index, TargetOlsLayerIdx. If the target OLS has only one layer, the layer bitstream under test, TargetLayerBitstream, is set to be identical to BitstreamToDecode. Otherwise, TargetLayerBitstream is derived by calling a demultiplex process to derive the layer bitstream, taking BitstreamToDecode, TargetOlsIdx, and TargetOlsLayerIdx as inputs, and the output is assigned to TargetLayerBitstream.

[0162] The `layer_level_hrd_parameters()` and `sub_layer_hrd_parameters()` syntax structures applicable to `TargetLayerBitstream` are selected as follows: The `layer_level_hrd_idx[TargetOlsIdx][TargetOlsLayerIdx]`-th `layer_level_hrd_parameters()` syntax structure within the VPS (or provided through an external mechanism such as user input) is selected. In the selected `layer_level_hrd_parameters()` syntax structure, if `BitstreamToDecode` is a type I bitstream, the `sub_layer_hrd_parameters(Htid)` syntax structure immediately following the condition `if(general_vcl_hrd_params_present_flag)` is selected, and the variable `NalHrdModeFlag` is set to equal 0. Otherwise (if BitstreamToDecode is a type II bitstream), the sub_layer_hrd_parameters(Htid) syntax structure immediately following either the condition if(general_vcl_hrd_params_present_flag) (in which case the variable NalHrdModeFlag is set to equal 0) or the condition if(general_nal_hrd_params_present_flag) (in which case the variable NalHrdModeFlag is set to equal 1) is selected. If BitstreamToDecode is a type II bitstream and NalHrdModeFlag is equal to 0, all non-VCL NAL units except filler data NAL units, as well as all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte stream from the NAL unit stream, are discarded from TargetLayerBitstream if they exist, and the remaining bitstream is allocated to TargetLayerBitstream.

[0163] If decoding_unit_hrd_params_present_flag is equal to 1, the CPB is scheduled to operate either at the access unit level (in which case the variable DecodingUnitHrdFlag is set to equal to 0) or at the decoding unit level (in which case the variable DecodingUnitHrdFlag is set to equal to 1). Otherwise, DecodingUnitHrdFlag is set to equal to 0, and the CPB is scheduled to operate at the access unit level. For each access unit in TargetLayerBitstream starting from access unit 0, a buffering period SEI message associated with the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected; an image timing SEI message associated with the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected; and if DecodingUnitHrdFlag is equal to 1 and decoding_unit_cpb_params_in_pic_timing_sei_flag is equal to 0, a decoding unit information SEI message associated with the decoding unit in the access unit and applied to TargetOlsIdx and TargetOlsLayerIdx (either present in TargetLayerBitstream or available through an external mechanism) is selected.

[0164] Each conformance test includes a combination of one option in each of the steps described above. If there are two or more options for a given step, only one option is selected for any particular conformance test. All possible combinations of all steps form the entire set of conformance tests. For each operating point under test, the number of bitstream conformance tests performed is equal to n0*n1*n2*n3, where the values ​​of n0, n1, n2, and n3 are specified as follows: n1 is equal to hrd_cpb_cnt_minus1+1. n1 is the number of access units in BitstreamToDecode associated with the buffering period SEI message. n2 is derived as follows: n0 is equal to 1 if BitstreamToDecode is a type I bitstream. Otherwise (if BitstreamToDecode is a type II bitstream), n0 is equal to 2. n3 is derived as follows: n3 is equal to 1 if decoding_unit_hrd_params_present_flag is equal to 0. Otherwise, n3 is equal to 2.

[0165] HRD includes a bitstream demultiplexer (optionally present), coded image buffers (CPBs) for each layer, instantaneous decoding processes for each layer, decoded image buffers (DPBs) containing sub-DPBs for each layer, and output cropping.

[0166] In one example, the HRD operates as follows: The HRD is initialized to 0 in the decoding unit, and each CPB and each sub-DPB of the DPB is set to be empty. The sub-DPB fullness of each sub-DPB is set to equal 0. After initialization, the HRD is not initialized again by subsequent buffering period SEI messages. Data associated with decoding units that have flowed into each CPB according to the specified arrival schedule is delivered by the HSS. The data associated with each decoding unit is removed by the instantaneous decoding process during the decoding unit's CPB removal time and decoded instantaneously. Each decoded image is placed in the DPB. Decoded images are removed from the DPB when they are no longer needed for interpredictive reference and no longer needed for output.

[0167] In one example, the demultiplex process for deriving the layer bitstream is as follows: The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the target OLS layer index targetOlsLayerIdx. The output of this process is the layer bitstream outBitstream. The output layer bitstream outBitstream is derived as follows: The bitstream outBitstream is set to be identical to the bitstream inBitstream. All NAL units whose nuh_layer_id is not equal to LayerIdInOls[targetOlsIdx][targetOlsLayerIdx] are removed from outBitstream. Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 1 and i and j values ​​do not fall within the ranges of 0 to nesting_num_ols_minus1 (including both endpoints) and nesting_num_olss_layers_minus1[i] (including both endpoints), so that NestingOlsLayerIdx[i][j] is equal to targetOlsLayerIdx. Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 1 and i and j values ​​fall within the ranges of 0 to nesting_num_olss_minus1 (including both endpoints) and nesting_num_ols_layers_minus1[i] (including both endpoints), so that NestingOlsLayerIdx[i][j] is less than targetOlsLayerIdx.Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 0 and i has no value in the range from 0 to LayerIdInOls-1, including both endpoints, so that NestingLayerId[i] is equal to NestingNumLayers[targetOlsIdx][targetOlsLayerIdx]. Remove all SEI NAL units from the outBitstream that contain scalable nesting SEI messages where nesting_ols_flag is equal to 0 and i has at least one value in the range from 0 to LayerIdInOls-1, including both endpoints, so that NestingLayerId[i] is less than NestingNumLayers[targetOlsIdx][targetOlsLayerIdx].

[0168] An example of buffered-period SEI message syntax is as follows: [Table 5]

[0169] An example of scalable nesting SEI message syntax is as follows: [Table 6]

[0170] The following are exemplary general SEI payload semantics. The following applies to the applicable layers (in the context of OLS, or generally) of a non-scalable nested SEI message. For a non-scalable nested SEI message, if payloadType is equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information), the non-scalable nested SEI message applies only to the lowest layer in the context of the 0th OLS. For a non-scalable nested SEI message, if payloadType is equal to any value in VclAssociatedSeiList, the non-scalable nested SEI message applies only to layers where the nuh_layer_id of the VCL NAL unit is equal to the nuh_layer_id of the SEI NAL unit containing the SEI message.

[0171] The semantics of an exemplary buffering period SEI message are as follows: The buffering period SEI message provides initial CPB removal delay and initial CPB removal delay offset information for initializing the HRD at the position of the associated access unit in decoding order. If a buffering period SEI message is present, an image whose TemporalId is equal to 0 and is not a RASL or RADL (random access decodable leading) image is said to be a notDiscardablePic image. If the current image is not the first image in the bitstream in decoding order, prevNonDiscardablePic is said to be a preceding image in decoding order whose TemporalId is equal to 0 and is not a RASL or RADL image.

[0172] The presence of a buffering period SEI message is specified as follows: If NalHrdBpPresentFlag is equal to 1 or VclHrdBpPresentFlag is equal to 1, the following applies to each access unit in CVS: If the access unit is an IRAP or Gradual Decoder Refresh (GDR) access unit, the buffering period SEI message applicable to the operating point shall be associated with the access unit. Otherwise, if the access unit contains notDiscardablePic, the buffering period SEI message applicable to the operating point may or may not be associated with the access unit. Otherwise, the access unit shall not be associated with the buffering period SEI message applicable to the operating point. Otherwise (if both NalHrdBpPresentFlag and VclHrdBpPresentFlag are equal to 0), no access unit in CVS shall be associated with the buffering period SEI message. In some applications, it may be desirable to have buffered-period SEI messages frequently (for example, in the case of random access to IRAP or non-IRAP images, or in the case of bitstream splicing). If the image of an access unit is associated with a buffered-period SEI message, the access unit shall have an image for each layer present in the CVS, and each image of the access unit shall be associated with a buffered-period SEI message.

[0173] bp_max_sub_layers_minus1+1 specifies the maximum number of time sublayers for which the CPB removal delay and CPB removal offset are indicated in the buffering period SEI message. The value of bp_max_sub_layers_minus1, including both endpoints, should be in the range of 0 to vps_max_sub_layers_minus1. bp_cpb_cnt_minus1+1 specifies the number of syntax element pairs of nal_initial_cpb_removal_delay[i][j] and nal_initial_cpb_removal_offset[i][j] in the i-th time sublayer when bp_nal_hrd_params_present_flag is equal to 1, and the number of syntax element pairs of vcl_initial_cpb_removal_delay[i][j] and vcl_initial_cpb_removal_offset[i][j] in the i-th time sublayer when bp_vcl_hrd_params_present_flag is equal to 1. The value of bp_cpb_cnt_minus1 shall be in the range of 0 to 31, including the endpoints. The value of bp_cpb_cnt_minus1 shall be equal to the value of hrd_cpb_cnt_minus1.

[0174] The semantics of an exemplary image timing SEI message are as follows: An image timing SEI message provides CPB removal delay and DPB output delay information for the access unit associated with the SEI message. If the bp_nal_hrd_params_present_flag or bp_vcl_hrd_params_present_flag of a buffering period SEI message applicable to the current access unit is equal to 1, the variable CpbDpbDelaysPresentFlag is set to equal to 1. Otherwise, CpbDpbDelaysPresentFlag is set to equal to 0. The existence of an image timing SEI message is specified as follows: If CpbDpbDelaysPresentFlag is equal to 1, the image timing SEI message is associated with the current access unit. Otherwise (CpbDpbDelaysPresentFlag is equal to 0), there is no image timing SEI message associated with the current access unit. The TemporalId in the image timing SEI message syntax is the TemporalId of the SEI NAL unit containing the image timing SEI message. pt_max_sub_layers_minus1+1 specifies the TemporalId of the top-level sublayer representation in which CPB removal delay information is included in the image timing SEI message. The value of pt_max_sub_layers_minus1, including both endpoints, should be in the range of 0 to vps_max_sub_layers_minus1.

[0175] The semantics of an exemplary scalable nesting SEI message are as follows: A scalable nesting SEI message provides a mechanism for associating an SEI message with a specific layer within the context of a particular OLS, or with a specific layer that is not within the context of an OLS. A scalable nesting SEI message contains one or more SEI messages. SEI messages contained within a scalable nesting SEI message are also called scalable nested SEI messages. The bitstream conformance requirement is that the following restrictions apply to the inclusion of SEI messages within a scalable nesting SEI message: SEI messages with payloadType equal to 132 (decoded image hash) or 133 (scalable nesting) shall not be included in a scalable nesting SEI message. If a scalable nesting SEI message includes a buffering period, image timing, or decoding unit information SEI message, the scalable nesting SEI message shall not contain any other SEI messages whose payloadType is not equal to 0 (buffering period), 1 (image timing), or 130 (decoding unit information).

[0176] The bitstream conformance requirement is that the following restrictions apply to the value of nal_unit_type for SEI NAL units containing scalable nesting SEI messages: If a scalable nesting SEI message contains an SEI message whose payloadType is equal to 0 (buffering period), 1 (image timing), 130 (decoded unit information), 145 (dependent RAP instruction), or 168 (frame field information), then the SEI NAL unit containing the scalable nesting SEI message shall have nal_unit_type equal to PREFIX_SEI_NUT. If a scalable nesting SEI message contains an SEI message whose payloadType is equal to 132 (decoded image hash), then the SEI NAL unit containing the scalable nesting SEI message shall have nal_unit_type equal to SUFFIX_SEI_NUT.

[0177] Setting nesting_ols_flag to 1 specifies that scalable nesting SEI messages apply to a specific layer in the context of a particular OLS. Setting nesting_ols_flag to 0 specifies that scalable nesting SEI messages apply to a specific layer in general (not in the context of an OLS). Bitstream compatibility requirements are that the following restrictions apply to the value of nesting_ols_flag: If a scalable nesting SEI message contains an SEI message with payloadType equal to 0 (buffering period), 1 (image timing), or 130 (decoded unit information), the value of nesting_ols_flag shall be equal to 1. If a scalable nesting SEI message contains an SEI message with payloadType equal to a value of VclAssociatedSeiList, the value of nesting_ols_flag shall be equal to 0. nesting_num_olss_minus1+1 specifies the number of OLS to which scalable nesting SEI messages apply. The value of nesting_num_olss_minus1 is assumed to be in the range of 0 to TotalNumOlss-1, including both endpoints. nesting_ols_idx_delta_minus1[i] is used to derive the variable NestingOlsIdx[i], which specifies the OLS index of the i-th OLS to which the scalable nested SEI message is applied when nesting_ols_flag is equal to 1. The value of nesting_ols_idx_delta_minus1[i] is assumed to be in the range of 0 to TotalNumOlss-2, including both endpoints. The variable NestingOlsIdx[i] is derived as follows:

number

[0178] nesting_num_ols_layers_minus1[i]+1 specifies the number of layers to which the scalable nested SEI message is applied in the context of the NestingOlsIdx[i]-th OLS. The value of nesting_num_ols_layers_minus1[i], including both endpoints, is in the range of 0 to NumLayersInOls[NestingOlsIdx[i]]-1. nesting_ols_layer_idx_delta_minus1[i][j] is used to derive the variable NestingOlsLayerIdx[i][j], which specifies the OLS layer index of the j-th layer to which the scalable nested SEI message is applied in the context of the NestingOlsIdx[i]-th OLS, when nesting_ols_flag is equal to 1. The value of nesting_ols_layer_idx_delta_minus1[i], including both endpoints, is assumed to be in the range of 0 to NumLayersInOls[nestingOlsIdx[i]]-2. The variable NestingOlsLayerIdx[i][j] is derived as follows:

number

[0179] The smallest value among all the values ​​of LayerIdInOls[NestingOlsIdx[i]][NestingOlsLayerIdx[i][0]] for i in the range from 0 to nesting_num_olss_minus1, including both endpoints, shall be equal to the nuh_layer_id of the current SEI NAL unit (the SEI NAL unit containing the scalable nesting SEI message). nesting_all_layers_flag is set to 1 to specify that the scalable nesting SEI message applies generally to all layers whose nuh_layer_id is greater than or equal to the nuh_layer_id of the current SEI NAL unit. nesting_all_layers_flag is set to 0 to specify that the scalable nesting SEI message may or may not apply generally to all layers whose nuh_layer_id is greater than or equal to the nuh_layer_id of the current SEI NAL unit. nesting_num_layers_minus1+1 specifies the number of layers to which scalable nested SEI messages are generally applied. The value of nesting_num_layers_minus1, including both ends, is in the range of 0 to vps_max_layers_minus1-GeneralLayerIdx[nuh_layer_id], where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit. nesting_layer_id[i] specifies the nuh_layer_id value of the i-th layer to which scalable nested SEI messages are generally applied when nesting_all_layers_flag is equal to 0. The value of nesting_layer_id[i] is greater than nuh_layer_id, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.If nesting_ols_flag is equal to 0, the variable NestingNumLayers specifies the number of layers to which scalable nested SEI messages are generally applied, and NestingLayerId[i] is a list of nuh_layer_id values ​​for i in the range from 0 to NestingNumLayers-1, including both ends, which specifies a list of nuh_layer_id values ​​for the layers to which scalable nested SEI messages are generally applied, and is derived as follows, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

number

[0180] nesting_num_seis_minus1+1 specifies the number of scalable nested SEI messages. The value of nesting_num_seis_minus1, including both ends, should be in the range of 0 to 63. nesting_0_bit should be equal to 0.

[0181] Figure 9 is a schematic diagram of an exemplary video coding apparatus 900. The video coding apparatus 900 is suitable for carrying out the disclosed embodiments / models described herein. The video coding apparatus 900 comprises a transceiver unit (Tx / Rx) 910 including a downstream port 920, an upstream port 950, and / or a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding apparatus 900 also includes a processor 930 including a logic unit and / or a central processing unit (CPU) for processing data, and memory 932 for storing data. The video coding apparatus 900 may also include electrical, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 950 and / or downstream port 920 for communicating data over an electrical, optical, or wireless communication network. The video coding apparatus 900 may also include input and / or output (I / O) devices 960 for communicating data with a user. The I / O device 960 may include output devices such as a display for showing video data and a speaker for outputting audio data. The I / O device 960 may also include input devices such as a keyboard, mouse, or trackball, and / or corresponding interfaces for interacting with such output devices.

[0182] The processor 930 is implemented by hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 930 communicates with downstream ports 920, Tx / Rx 910, upstream ports 950, and memory 932. The processor 930 includes a coding module 914. The coding module 914 implements embodiments disclosed herein, such as methods 100, 1000, and 1100, which can employ a multi-layer video sequence 600, a multi-layer video sequence 700, and / or bitstream 800. The coding module 914 can also implement any other methods / mechanisms described herein. Furthermore, the coding module 914 can implement a codec system 200, an encoder 300, a decoder 400, and / or HRD 500. For example, the coding module 914 may be used to implement HRD. Furthermore, the coding module 914 may be used to encode parameters into a bitstream and support the HRD conformance check process. Thus, the coding module 914 may be configured to perform a mechanism to address one or more of the aforementioned problems. Thus, the coding module 914 provides additional functionality and / or coding efficiency to the video coding device 900 when coding video data. Thus, the coding module 914 improves the functionality of the video coding device 900 and also addresses problems specific to video coding technology. Furthermore, the coding module 914 results in a transformation of the video coding device 900 into different states. Alternatively, the coding module 914 can be implemented as instructions stored in memory 932 and executed by the processor 930 (for example, as a computer program product stored on a non-transient medium).

[0183] Memory 932 comprises one or more memory types, such as disks, tape drives, solid-state drives, read-only memory (ROM), random-access memory (RAM), flash memory, terminally associative memory (TCAM), and static random-access memory (SRAM). Memory 932 is used as an overflow data storage device and can store programs when they are selected for execution, and can store instructions and data read during program execution.

[0184] Figure 10 is a flowchart of an exemplary method 1000 for encoding a video sequence into a bitstream such as bitstream 800, where all layers within the bitstream have the same number of CPB distribution schedules, such as CPB distribution schedule 561. Method 1000 may be employed by an encoder such as a codec system 200, encoder 300, and / or video coding device 900 when performing Method 100. Furthermore, Method 1000 can operate on HRD 500 and thus can perform conformance testing for multilayer video sequences 600 and / or 700.

[0185] Method 1000 can be initiated when an encoder receives a video sequence and decides to encode that video sequence into a multilayer bitstream, for example, based on user input. In step 1001, the encoder encodes a bitstream containing encoded images of one or more layers, such as layer N631, layer N+1632, sublayer 710, sublayer 720, and / or sublayer 730. In some examples, the multiple layers are organized into one or more OLS, such as OLS625.

[0186] In step 1003, the encoder encodes the syntax structure of the HRD parameters into a bitstream. The syntax structure of the HRD parameters is a data object containing one or more HRD parameters, which may be included in the syntax elements within the syntax structure. The HRD parameters in the HRD syntax structure specify that all layers are associated with the same number of CPB delivery schedules. For example, the number of CPB delivery schedules is specified by the HRD CPB count - 1 (hrd_cpb_cnt_minus1) syntax element. minus1 indicates that the value of hrd_cpb_cnt_minus1 is 1 less than the indicated value, and that +1 should be added to obtain the true value. hrd_cpb_cnt_minus1 may be in the range of 0 to 31, including both ends. This indicates that 1 to 32 CPB delivery schedules may be specified, and that all layers are associated with the same number of CPB delivery schedules. The syntax structure of the HRD parameters can be encoded into a bitstream in the VPS specifying the OLS.

[0187] In some examples, each corresponding CPB delivery schedule in a target OLS can be referenced by a unique schedule index (ScIdx). ScIdx can include values ​​ranging from 0 to hrd_cpb_cnt_minus1, including both endpoints. This indicates that ScIdx can vary from a value of 0 for the first CPB delivery schedule to a value of hrd_cpb_cnt_minus1 for the last CPB delivery schedule. In some examples, each CPB delivery schedule is described by a pair of initial CPB removal delays and initial CPB removal offsets. The initial CPB removal delay (nal_initial_cpb_removal_delay) is the default removal delay used when removing data units from the CPB in the HRD when checking the bitstream, OLS, and / or images, AUs, and / or DUs in the layer. The initial CPB removal offset (nal_initial_cpb_removal_offset) is the default CPB removal offset associated with each image, AU, and / or DU in the bitstream, OLS, and / or layer. A layer may also be associated with the BP CPB count - 1 (bp_cpb_cnt_minus1) + 1 syntax element. BP CPB count - 1 is a syntax element that specifies the number of pairs of initial CPB removal delays and initial CPB removal offsets, and therefore the number of delivery schedules available for each layer / sublayer. The value of bp_cpb_cnt_minus1 may be set to be equal to the value of hrd_cpb_cnt_minus1. This ensures that the number of delivery schedules for each layer is the same as the total number of CPB delivery schedules. Thus, all layers have the same number of CPB delivery schedules. Syntax elements describing CPB delivery schedules may be included in the SEI message. For example, nal_initial_cpb_removal_delay, nal_initial_cpb_removal_offset, and bp_cpb_cnt_minus1 may be included in the BP SEI message.

[0188] In step 1005, HRD retrieves HRD parameters from the syntax structure of the HRD parameters and performs a set of bitstream conformance tests for multiple layers based on the CPB delivery schedule. For example, HRD can retrieve hrd_cpb_cnt_minus1 to determine the number of CPB delivery schedules. HRD can also retrieve bp_cpb_cnt_minus1 to determine the number of CPB delivery schedules for each layer. HRD can also retrieve nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset to determine each of the CPB delivery schedules. HRD can then perform conformance tests for multiple layers based on the CPB delivery schedule. HRD can run on the same processor as the encoder. In another example, HRD can run on a processor in the same chassis as the encoder. In yet another example, HRD can run on a processor in the same network as the encoder.

[0189] In step 1007, the encoder stores a bitstream for communication with the decoder.

[0190] Figure 11 is a flowchart of an exemplary method 1100 for decoding a video sequence from a bitstream such as bitstream 800, where all layers in the bitstream have the same number of CPB delivery schedules, e.g., CPB delivery schedule 561, as checked for compatibility by an HRD such as HRD 500. Method 1100 may be employed by a decoder such as a codec system 200, a decoder 400, and / or a video coding device 900 when performing Method 100. Furthermore, Method 1100 may operate on a bitstream containing multilayer video sequences 600 and / or 700.

[0191] Method 1100 can be initiated when the decoder begins receiving a bitstream of coded data representing a multilayer video sequence, for example, as a result of Method 1000. In step 1101, the decoder can receive the bitstream. The bitstream contains a syntax structure of HRD parameters. The syntax structure of HRD parameters is a data object containing one or more HRD parameters, which may be contained within the syntax elements in the syntax structure. The bitstream may also contain one or more layers of the coded image, such as layer N631, layer N+1632, sublayer 710, sublayer 720, and / or sublayer 730. In some examples, multiple layers are organized into one or more OLS, such as OLS625. The HRD parameters specify that all of the multiple layers are associated with the same number of CPB delivery schedules. The CPB delivery schedules may be associated with a set of bitstream conformance tests performed by the HRD operating on the encoder. The HRD parameters may contain VPS specifying the OLS. In one example, the number of CPB delivery schedules is specified by the HRD CPB count - 1 (hrd_cpb_cnt_minus1) syntax element. minus1 indicates that the value of hrd_cpb_cnt_minus1 is 1 less than the specified value, and that +1 should be added to obtain the true value. hrd_cpb_cnt_minus1 may range from 0 to 31, including both extreme values. This indicates that 1 to 32 CPB delivery schedules can be specified, and that all layers are associated with the same number of CPB delivery schedules.

[0192] In some examples, each corresponding CPB delivery schedule in a target OLS can be referenced by a unique schedule index (ScIdx). ScIdx can include values ​​ranging from 0 to hrd_cpb_cnt_minus1, including both endpoints. This indicates that ScIdx can vary from a value of 0 for the first CPB delivery schedule to a value of hrd_cpb_cnt_minus1 for the last CPB delivery schedule. In some examples, each CPB delivery schedule is described by a pair of initial CPB removal delays and initial CPB removal offsets. The initial CPB removal delay (nal_initial_cpb_removal_delay) is the default removal delay used when removing data units from the CPB in the HRD when checking the bitstream, OLS, and / or images, AUs, and / or DUs in the layer. The initial CPB removal offset (nal_initial_cpb_removal_offset) is the default CPB removal offset associated with each image, AU, and / or DU in the bitstream, OLS, and / or layer. Each layer may also be associated with a BP CPB count - 1 (bp_cpb_cnt_minus1) + 1 syntax element. BP CPB count - 1 is a syntax element that specifies the number of pairs of initial CPB removal delays and initial CPB removal offsets, and therefore the number of delivery schedules available to each layer / sublayer. The value of bp_cpb_cnt_minus1 may be set to be equal to the value of hrd_cpb_cnt_minus1. This ensures that the number of delivery schedules for each layer is equal to the total number of CPB delivery schedules. Thus, all layers have the same number of CPB delivery schedules. Syntax elements describing CPB delivery schedules may be included in the SEI message. For example, nal_initial_cpb_removal_delay, nal_initial_cpb_removal_offset, and bp_cpb_cnt_minus1 may be included in the BP SEI message.

[0193] In step 1103, the decoder can decode the image / encoded image from one of the layers and / or OLS to obtain the decoded image. The decoder can also transfer the decoded image in step 1105 for display as part of the decoded video sequence.

[0194] Figure 12 is a schematic diagram of an exemplary system 1200 for coding a video sequence using a bitstream in which all layers of the bitstream have the same number of CPB distribution schedules, such as CPB distribution schedule 561. System 1200 may be implemented by encoders and decoders such as a codec system 200, encoder 300, decoder 400, and / or video coding device 900. Furthermore, system 1200 can employ HRD 500 to perform conformance testing for multilayer video sequence 600, multilayer video sequence 700, and / or bitstream 800. In addition, system 1200 may be employed when implementing methods 100, 1000, and / or 1100.

[0195] System 1200 includes a video encoder 1202. The video encoder 1202 includes an encoding module 1203 for encoding a bitstream containing one or more layers of coded images. The encoding module 1203 further encodes into the bitstream a syntax structure of HRD parameters specifying that all of the multiple layers are associated with the same number of coded image buffer (CPB) delivery schedules. The video encoder 1202 further includes an HRD module 1205 for performing a set of bitstream conformance tests on the multiple layers based on the CPB delivery schedules. The video encoder 1202 further includes a storage module 1206 for storing the bitstream for communication toward the decoder. The video encoder 1202 further includes a transmission module 1207 for transmitting the bitstream toward the video decoder 1210. The video encoder 1202 may further be configured to perform any of the steps of Method 1000.

[0196] System 1200 also includes a video decoder 1210. The video decoder 1210 includes a receive module 1211 for receiving a bitstream comprising the syntax structure of HRD parameters and one or more layers of coded images, such that all of the multiple layers are associated with the same number of CPB delivery schedules as specified by the syntax structure of the HRD parameters. The video decoder 1210 further includes a decode module 1213 for decoding an image from one of the layers to obtain a decoded image. The video decoder 1210 further includes a transfer module 1215 for transferring the decoded image for display as part of a decoded video sequence. The video decoder 1210 may be further configured to perform any of the steps of method 1100.

[0197] If there are no intermediary components between the first and second components, except for lines, traces, or other media, the first component is directly coupled to the second component. If there are intermediary components other than lines, traces, or other media between the first and second components, the first component is indirectly coupled to the second component. The term "coupled" and its variations include both directly and indirectly coupled components. The use of the term "about" means a range including ±10% of the following number unless otherwise specified.

[0198] It should be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of the steps in such methods should be understood to be merely illustrative. Similarly, in methods consistent with various embodiments of this disclosure, additional steps may be included in such methods, and certain steps may be omitted or combined.

[0199] While several embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of this disclosure. These embodiments should be considered illustrative and not limiting, and the invention should not be limited to the details provided herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0200] In addition, the technologies, systems, subsystems, and methods described and illustrated in various embodiments, either discretely or separately, may be combined with or integrated with other systems, components, technologies, or methods without departing from the scope of this disclosure. Other examples of modifications, substitutions, and alternatives are readily apparent to those skilled in the art and can be made without departing from the spirit and scope disclosed herein. Other possible claims (Item 1) A method implemented by a decoder, The steps include receiving a bitstream comprising a syntax structure of virtual reference decoder (HRD) parameters and one or more layers of an encoded image, such that all of the layers are associated with the same number of encoded image buffer (CPB) delivery schedules specified by the syntax structure of the HRD parameters, The decoder's processor decodes an image from one of the multiple layers to obtain a decoded image. Methods that include... (Item 2) The method according to item 1, wherein the CPB delivery schedule is associated with a set of bitstream conformance tests performed by an HRD operating on the encoder. (Item 3) The method described in item 1 or 2, wherein the number of CPB delivery schedules is specified by HRD CPB count - 1 (hrd_cpb_cnt_minus1), and hrd_cpb_cnt_minus1 is in the range of 0 to 31, including both endpoints. (Item 4) The method according to any one of items 1 to 3, wherein the aforementioned layers are organized into one or more output layer sets (OLS), and the corresponding CPB delivery schedule of the target OLS is referenced by a schedule index (ScIdx) ranging from 0 to the value hrd_cpb_cnt_minus1, including the end values. (Item 5) The method according to any one of items 1 to 4, wherein each of the CPB distribution schedules is described by a Network Abstraction Layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and a NAL initial CPB removal offset (nal_initial_cpb_removal_offset). (Item 6) The method according to any one of items 1 to 5, wherein each layer is associated with a buffering period (BP) CPB count - 1 (bp_cpb_cnt_minus1) + 1, which specifies the number of pairs of nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset for each layer, and bp_cpb_cnt_minus1 is equal to the value of hrd_cpb_cnt_minus1. (Item 7) The method according to any one of items 1 to 6, wherein the HRD parameter is included in the bitstream within the video parameter set (VPS) that specifies the plurality of OLS. (Item 8) A method performed by an encoder, The encoder's processor performs the steps of encoding a bitstream containing one or more layers of the encoded image, The process involves the processor encoding into the bitstream a syntax structure of virtual reference decoder (HRD) parameters that specify that all of the multiple layers are associated with the same number of coded image buffer (CPB) delivery schedules, The processor performs a set of bitstream conformance tests on the multiple layers based on the CPB distribution schedule, Methods that include... (Item 9) The method according to item 8, wherein the set of bitstream conformance tests is performed by an HRD operating on the processor. (Item 10) The method described in item 8 or 9, where the number of CPB delivery schedules is specified by HRD CPB count - 1 (hrd_cpb_cnt_minus1), and hrd_cpb_cnt_minus1 is in the range of 0 to 31, including both endpoints. (Item 11) The method according to any one of items 8 to 10, wherein the plurality of layers are organized into one or more output layer sets (OLS), and the corresponding CPB delivery schedule of the target OLS is referenced by a schedule index (ScIdx) ranging from 0 to the value hrd_cpb_cnt_minus1, including the end values. (Item 12) The method described in any one of items 8 to 11, wherein each of the CPB delivery schedules is described by a Network Abstraction Layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and a NAL initial CPB removal offset (nal_initial_cpb_removal_offset). (Item 13) The method according to any one of items 8 to 12, wherein each layer is associated with a buffering period (BP) CPB count - 1 (bp_cpb_cnt_minus1) + 1, which specifies the number of pairs of nal_initial_cpb_removal_delay and nal_initial_cpb_removal_offset for each layer, and bp_cpb_cnt_minus1 is equal to the value of hrd_cpb_cnt_minus1. (Item 14) The method according to any one of items 8 to 13, wherein the syntax structure of the HRD parameter is encoded into the bitstream in a video parameter set (VPS) specifying the plurality of OLS. (Item 15) A video coding device, The system comprises a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform the method described in any one of items 1 to 14. Video coding device. (Item 16) A non-transient computer-readable medium comprising a computer program product for use by a video coding device, wherein, when the computer program product is executed by a processor, the non-transient computer-readable medium comprises computer-executable instructions stored on the non-transient computer-readable medium to cause the video coding device to perform the method described in any one of items 1 to 14. (Item 17) It is a decoder, Receiving means for receiving a bitstream comprising a syntax structure of virtual reference decoder (HRD) parameters and one or more layers of an encoded image, wherein all of the plurality of layers are associated with the same number of encoded image buffer (CPB) delivery schedules specified by the syntax structure of the HRD parameters, A decoding means for decoding an image from one of the aforementioned multiple layers and obtaining a decoded image, A transfer means for transferring the decoded image for display as part of the decoded video sequence, A decoder equipped with a decoder. (Item 18) The decoder according to item 17, further configured to perform the method described in any one of items 1 to 7. (Item 19) It is an encoder, Encode a bitstream having one or more layers of coded images, The syntax structure of virtual reference decoder (HRD) parameters specifying that all of the aforementioned layers are associated with the same number of coded image buffer (CPB) delivery schedules is encoded into the bitstream. Encoding means for, HRD means for performing a set of bitstream conformity tests on the multiple layers based on the CPB distribution schedule, A storage means for storing the bitstream for communication to the decoder, An encoder equipped with the following features. (Item 20) The encoder described in item 19, further configured to perform the method described in any one of items 8 through 14.

Claims

1. A decoding method, The step of receiving a bitstream that includes multiple time sublayers, a syntax structure of virtual reference decoder (HRD) parameters, and a buffering period syntax structure, In the syntax structure of the HRD parameter, a first single number is signaled only once, and the first single number plus 1 specifies the number of coded image buffer (CPB) delivery schedules in the bitstream. In the buffering period syntax structure, a second single number is signaled only once, and the second single number plus 1 specifies the number of syntax element pairs for each of the plurality of time sublayers, wherein each syntax element pair includes a network abstraction layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and an NAL initial CPB removal offset (nal_initial_cpb_removal_offset), The second single number is equal to the first single number, Stages and The step of decoding one of the aforementioned multiple time sublayers to obtain a decoded image. A method for providing this.

2. The first single number is signaled by a first single syntax element specified as hrd_cpb_cnt_minus1 in the syntax structure of the HRD parameter. The method according to claim 1.

3. The first single number is in the range of 0 to 31, including both endpoints. The method according to claim 1 or 2.

4. The aforementioned second single number is signaled by a second single syntax element designated as bp_cpb_cnt_minus1 in the buffering period syntax structure. The method according to any one of claims 1 to 3.

5. The syntax structure of the HRD parameters is included in the video parameter set (VPS) within the bitstream. The method according to any one of claims 1 to 4.

6. The aforementioned time sublayers are organized into one or more output layer sets (OLS), and the corresponding CPB delivery schedule of the target OLS is referenced by a schedule index (ScIdx) ranging from 0 to the first single number, including the endpoints. The method according to any one of claims 1 to 5.

7. An encoding method, The step involves encoding multiple time sublayers, a syntax structure of virtual reference decoder (HRD) parameters, and a buffering period syntax structure into a bitstream, In the syntax structure of the HRD parameter, a first single number is signaled only once, and the first single number plus 1 specifies the number of coded image buffer (CPB) delivery schedules in the bitstream. In the buffering period syntax structure, a second single number is signaled only once, and the second single number plus 1 specifies the number of syntax element pairs for each of the plurality of time sublayers, wherein the syntax element pairs include a network abstraction layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and an NAL initial CPB removal offset (nal_initial_cpb_removal_offset), The second single number is equal to the first single number, Stages and The steps include: performing a set of bitstream conformance tests on the multiple time sublayers based on the CPB distribution schedule; A method for providing this.

8. The first single number is signaled by a first single syntax element specified as hrd_cpb_cnt_minus1 in the syntax structure of the HRD parameter. The method according to claim 7.

9. The first single number is in the range of 0 to 31, including both endpoints. The method according to claim 7 or 8.

10. The aforementioned second single number is signaled by a second single syntax element designated as bp_cpb_cnt_minus1 in the buffering period syntax structure. The method according to any one of claims 7 to 9.

11. The syntax structure of the HRD parameters is included in the video parameter set (VPS) within the bitstream. The method according to any one of claims 7 to 10.

12. The aforementioned time sublayers are organized into one or more output layer sets (OLS), and the corresponding CPB delivery schedule of the target OLS is referenced by a schedule index (ScIdx) ranging from 0 to the first single number, including the endpoints. The method according to any one of claims 7 to 11.

13. A non-temporary computer-readable storage medium that holds program code, which, when executed by a computer device, causes the computer device to perform the method according to any one of claims 1 to 12.

14. A computer program, which, when executed on a computer or processor, includes program code for performing the method described in any one of claims 1 to 12.

15. A decoding device, At least one processor, A computer-readable storage medium coupled to the at least one processor and storing instructions for execution by the at least one processor, wherein when the instructions are executed by the at least one processor, the computer-readable storage medium causes the decoding device to perform the method according to any one of claims 1 to 6. A decoding device equipped with the following features.

16. The processing circuit comprises a method for performing the method described in any one of claims 1 to 6. Decoding device.

17. An encoding device, At least one processor, A computer-readable storage medium coupled to the at least one processor and storing instructions for execution by the at least one processor, wherein when the instructions are executed by the at least one processor, the computer-readable storage medium causes the encoding device to perform the method according to any one of claims 7 to 12. An encoding device equipped with the following features.

18. The processing circuit comprises a method for performing the method described in any one of claims 7 to 12. Encoding device.

19. A non-temporary computer-readable storage medium storing a bitstream including multiple time sublayers, a syntax structure of virtual reference decoder (HRD) parameters, and a buffering period syntax structure, The syntax structure of the HRD parameters and the buffering period syntax structure are determined by the processor. In the syntax structure of the HRD parameter, a first single number is obtained only once, where the first single number plus 1 specifies the number of coded image buffer (CPB) delivery schedules in the bitstream. From the bitstream, obtain a number of CPB distribution schedules equal to the first single number plus one. In the buffering period syntax structure, a second single number is obtained only once, where the second single number plus 1 specifies the number of syntax element pairs for each of the plurality of time sublayers, and the syntax element pair includes the Network Abstraction Layer (NAL) initial CPB removal delay (nal_initial_cpb_removal_delay) and the NAL initial CPB removal offset (nal_initial_cpb_removal_offset). From the bitstream, obtain a number of syntax element pairs equal to the second single number of the initial NAL CPB removal delay and the initial NAL CPB removal offset plus one, and A decoded image is obtained by decoding one of the plurality of time sublayers according to the CPB distribution schedule, which is the first single number plus one, and the syntax element pairs, which is the second single number plus one, of the NAL initial CPB removal delay and the NAL initial CPB removal offset. Used for, The second single number is equal to the first single number, Computer-readable storage medium.

20. The first single number is signaled by a first single syntax element designated as hrd_cpb_cnt_minus1 in the syntax structure of the HRD parameter, and the second single number is signaled by a second single syntax element designated as bp_cpb_cnt_minus1 in the buffering period syntax structure. The computer-readable storage medium according to claim 19.

Images