Cross-layer reference constraints

Abstract

The present disclosure provides a video encoding and decoding method using inter-layer prediction or reference. A video decoder receives data from a bitstream carrying video pictures in multiple different layers. At least one of the multiple layers includes a temporal sub-layer corresponding to a level in a hierarchical temporal prediction structure. Each of the temporal sub-layers is associated with a temporal identifier. The video decoder receives an inter-layer prediction restriction parameter that restricts the maximum temporal sub-layer used in inter-layer prediction. The video decoder performs inter-layer prediction by referencing data of a second picture in a second layer to reconstruct a first picture in a first layer. The temporal identifier of the reference data satisfies the received inter-layer prediction restriction parameter.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 978,360 filed on February 19, 2020, No. 63 / 006,658 filed on April 7, 2020, and No. 63 / 011,849 filed on April 17, 2020, which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to video coding and decoding, and more particularly to inter-layer reference and prediction. Background Art

[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0005] High-efficiency video coding (HEVC) is an international video coding standard developed by the Joint Collaboration on Video Coding (JCT-VC). The input video signal is predicted from a reconstructed signal derived from the coded picture region. The prediction residual signal is processed by a linear transform. The transform coefficients are quantized and entropy coded along with other auxiliary information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after an inverse transform of the dequantized transform coefficients. The reconstructed signal is further processed by loop filtering to remove coding artifacts. The decoded picture is stored in a frame buffer and used to predict future pictures in the input video signal.

[0006] In HEVC, coded pictures are divided into non-overlapping square block areas represented by associated coding tree units (CTUs). A CTU can be divided into multiple non-overlapping coding units (CUs) using a recursive quadtree (QT) structure to accommodate various local motion and texture characteristics. A CTU can also be split into one or more smaller CUs using a quadtree with nested multi-type trees using binary and ternary partitioning. The generated CU partitions can be square or rectangular.

[0007] One or more prediction units (PUs) are specified for each CU. The prediction unit, together with the associated CU syntax, is used as the basic unit for notifying the predictor information. The specified prediction process is used to predict the values ​​of the relevant pixel samples within the PU. The CU can be further partitioned using the Residential Quadtree (RQT) structure to represent the associated prediction residual signal. The leaf nodes of the RQT correspond to the transform units (TUs). The transform unit consists of a transform block (TB) of luminance samples of size 8x8, 16x16 or 32x32, or four transform blocks of luminance samples of size 4x4 for a 4:2:0 color format picture, and two corresponding transform blocks of chrominance samples. An integer transform is applied to a transform block, and the average value of the quantization coefficient is entropy encoded and decoded in the bitstream together with other auxiliary information.

[0008] The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined to designate a two-dimensional array of samples for a color component corresponding to a CTU, CU, PU, ​​and TU, respectively. Thus, a CTU consists of one luma CTB, two chroma CTBs, and associated syntax elements. Similar relationships apply to CUs, PUs, and TUs. Tree partitioning generally applies to both luma and chroma, with exceptions occurring when chroma reaches certain minimum sizes.

[0009] A Network Abstraction Layer (NAL) unit is a logical data unit that encapsulates a syntax structure in a coded video bitstream. Therefore, the logical data unit may contain an indication of the type of subsequent data and a bit group containing data in the form of a raw byte sequence payload (RBSP). Each NAL unit has a NAL unit header (NUH). NAL units used to carry compressed video data for the video coding layer (VCL) are called VCL NAL units. NAL units can also be used to carry only parameters without actual video data; these are called non-VCL NAL units.

[0010] The header of each NAL unit also indicates the type of NAL unit in the syntax element nal_unit_type. For example, nal_unit_type can be used to indicate the type of picture carried by the NAL unit, such as gradual decoding refresh (GDR) pictures, intra random access pictures (IRAP), instantaneous decoding refresh (IDR) pictures, random access decodable leading (RADL) pictures, random access skipped leading (RASL) pictures, etc.

[0011] The coded video can have multiple layers of data in multiple substreams to implement scalable video coding. Different substreams carrying different layers are packaged in different NAL units. The header (NUH) of each NAL unit identifies the layer of the NAL unit in the syntax element nuh_layer_id. Different values ​​of nuh_layer_id represent different layers. VCL NAL units and associated non-VCL NAL units belonging to the same layer have the same nuh_layer_id. Summary of the Invention

[0012] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected, but not all, implementations are further described in the detailed description below. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0013] Some embodiments of the present disclosure provide a video encoding method using inter-layer prediction or reference. A video decoder receives data from a bitstream carrying video pictures in multiple different layers. At least one of the multiple layers includes multiple temporal sub-layers, and the multiple temporal sub-layers correspond to multiple levels in a hierarchical temporal prediction structure. Each of the temporal sub-layers is associated with a temporal identifier. The video decoder receives an inter-layer prediction restriction parameter that restricts the maximum temporal sub-layer used in the inter-layer prediction. The video decoder performs inter-layer prediction by referring to reference data of a second picture in a second layer to reconstruct a first picture in a first layer. The temporal identifier corresponding to the reference data satisfies the restrictions of the received inter-layer prediction restriction parameter.

[0014] In some embodiments, when a specific flag in a video parameter set (VPS) indicates the presence of an inter-layer prediction restriction parameter, the inter-layer prediction restriction parameter is signaled in the video parameter set applicable to the first picture. In some embodiments, when the temporal identifier is less than or equal to a value indicated by the inter-layer prediction restriction parameter, the temporal identifier of the reference data satisfies the constraints of the received inter-layer prediction restriction parameter. In some embodiments, when the inter-layer prediction restriction parameter is set to zero, the video decoder does not allow inter-layer prediction for the first picture.

[0015] When the third picture is associated with a temporal identifier that is greater than a value indicated by the inter-layer prediction restriction parameter, the video decoder may exclude the third picture from being used as a reference for inter-layer prediction of the first picture.

[0016] In some embodiments, the inter-layer prediction restriction parameters apply only when the second layer is used as a reference layer for inter-layer prediction of the first layer. The inter-layer prediction restriction parameters do not apply when another layer is used as a reference layer for inter-layer prediction of the first layer. In some embodiments, the entries in the inter-layer reference prediction (ILRP) list for the first picture (slice) are intra random access point pictures (IRAPs) or reference pictures in a reference layer with temporal identifiers that satisfy the restrictions of the inter-layer prediction restriction parameters applicable to the first layer and the reference layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of this disclosure. The accompanying drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. It will be understood that the drawings are not necessarily drawn to scale, as in actual implementations, some components may be shown out of scale in order to clearly illustrate the concepts of the present disclosure.

[0018] Figure 1 Describes the contents of a preset quantization matrix with preset values.

[0019] Figure 2 It is described that the basic scaling matrix is ​​upsampled to derive a quantization matrix with larger blocks.

[0020] Figure 3 Shows an example of luma mapping with chroma scaling (LMCS).

[0021] Figure 4 Conceptually illustrate the limitations of inter-layer prediction.

[0022] Figure 5 An example of an example video encoder that performs inter-layer prediction is illustrated.

[0023] Figure 6 A partial video encoder that implements inter-layer prediction restrictions is described.

[0024] Figure 7 Conceptually illustrates the process of encoding video pictures using inter-layer prediction.

[0025] Figure 8 An example of a video decoder performing inter-layer prediction is illustrated.

[0026] Figure 9 A partial video decoder that implements inter-layer prediction restrictions is described.

[0027] Figure 10 Conceptually illustrates the process of decoding video pictures using inter-layer prediction.

[0028] Figure 11 The electronic system for implementing the embodiments of the present invention is conceptually shown. DETAILED DESCRIPTION

[0029] In the following detailed description, many specific details are set forth by way of example to provide a thorough understanding of the relevant teachings. Any variations, derivations, and / or extensions based on the teachings described herein are within the scope of protection of the present disclosure. In some cases, well-known methods, processes, components, and / or circuits related to one or more example implementations disclosed herein may be described at a relatively high level without detail to avoid unnecessarily obscuring aspects of the teachings of the present disclosure.

[0030] I. Quantization Matrix

[0031] The block-based hybrid video coding scheme with residual signal transform coding uses frequency-dependent scaling to control the distribution of quantization distortion at different frequencies in a transform unit (TU). In order to achieve perceptually uniform quantization across spatial frequencies, the quantization matrix weights each frequency channel according to the perceptual sensitivity of each frequency channel associated with the transform coefficient in its relevant frequency range, so that the lower frequency coefficients in the transform block are quantized with a finer quantization step than the higher frequency coefficients. At the decoder, the corresponding quantization matrix inversely weights the dequantized transform coefficients of each frequency channel. The quantization matrix has been successfully applied to various video coding standards such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding) as well as numerous commercial products to improve the subjective quality of video content.

[0032] Quantization matrices (QM) are used to implement frequency-dependent scaling. Therefore, quantization matrices can also be called scaling matrices, scaling lists, or scaling list matrices. For HEVC, frequency-dependent scaling can be enabled by using the syntax element scaling_list_enabled_flag in the sequence parameter sets (SPS). When this flag is enabled, additional flags in the SPS and picture parameter sets (PPS) control whether a preset quantization matrix or a non-preset quantization matrix is ​​used. Figure 1 The contents of the preset quantization matrices of size 4x4 and 8x8 with preset values ​​are described. Non-preset quantization matrices can optionally be transmitted in the bitstream in the SPS or PPS.

[0033] For H.265 / HEVC, the following quantization matrices are supported, depending on the transform block size and type. Luma: Intra4x4, Inter4x4, Intra8x8, Inter8x8, Intra16x16, Inter16x16, Intra32x32, Inter32x32. Cb: Intra4x4, Inter4x4, Intra8x8, Inter8x8, Intra16x16, Inter16x16. Cr: Intra4x4, Inter4x4, Intra8x8, Inter8x8, Intra16x16, Inter16x16.

[0034] To reduce the memory required to store the quantization matrix, 8x8 matrices are used to derive the quantization matrices for larger transform blocks (e.g., 16x16, 32x32, 64x64, etc.). Therefore, the 8x8 pre-set quantization matrix is ​​called the base scaling matrix. For example, the quantization matrices for transform blocks of size 16x16 and 32x32 are derived from the same type of 8x8 base scaling matrix using upsampling replication.

[0035] Figure 2 The basic scaling matrix (8x8 quantization matrix) is upsampled to derive a quantization matrix for a larger block. As shown in the figure, the 1x1 region of the 8x8 quantization matrix 210 is upsampled to a 2x2 region in the 16x16 quantization matrix 220 and a 4x4 region in the 32x32 quantization matrix 230.

[0036] II. Luma Mapping with Chroma Scaling

[0037] Luma Mapping with Chroma Scaling (LMCS) is a video codec tool that includes: 1) a luma mapping process for mapping input luma code values ​​to a set of new code values ​​for use within the codec loop; and 2) a chroma scaling process for scaling chroma residual values ​​that depend on the luma component. The first luma mapping process aims to improve the coding efficiency of standard and high dynamic range video signals by better utilizing the range of luma code values ​​allowed by a specified bit depth. The second chroma scaling process is used to manage the relative compression efficiency of the luma and chroma components of the video signal. The luma mapping process of LMCS is applied at the pixel sample level and is implemented using a piecewise linear model. The chroma scaling process is applied at the chroma block level and is implemented using scaling factors derived from the reconstructed neighboring luma samples of the chroma block.

[0038] When LMCS is applied, the video samples are encoded, decoded and reconstructed in the integer domain before loop filtering. The integer domain reconstructed samples are converted to the raw domain using an inverse mapping. The loop filtered raw domain reconstructed samples are stored in the decoded picture buffer (DPB). For inter mode, the motion compensated (MC) predictor is converted to the integer domain using forward luminance mapping. Figure 3 An example of the luminance mapping and chroma residual scaling process under LMCS is shown.

[0039] III. Signaling LMCS and QM

[0040] An Adaptation Parameter Set (APS) is a syntax structure in coded video that contains parameters applicable to a slice identified by syntax elements found in the slice header. Table 1 shows the scaling list, LMCS, and adaptive loop filter (ALF) parameters specified by an Adaptation Parameter Set (APS):

[0041] Table 1:

[0042]

[0043] The syntax element adaptation_parameter_set_id provides an identifier of the APS for reference by other syntax elements. The syntax element aps_parameter_type specifies the type of the APS parameter. When aps_params_type is equal to ALF_APS or SCALING_APS, the value of adaptation_parameter_set_id shall be in the range of 0 to 7, inclusive. When aps_params_type is equal to LMCS_APS, the value of adaptation_parameter_set_id shall be in the range of 0 to 3, inclusive.

[0044] In some embodiments, APS NAL units with a specific aps_params_type value use the same value space for adaptation_parameter_set_id regardless of the value of nuh_layer_id. APS NAL units with different aps_params_type values ​​use separate value spaces for adaptation_parameter_set_id.

[0045] In some embodiments, all APS NAL units with a specific aps_params_type value may reference each other if the following two conditions are met: (1) the value of the nuh_layer_id of the referenced layer is less than or equal to the value of the nuh_layer_id of the current layer, and the nuh_layer_id of the reference layer is included in at least one output layer parameter set (OLS) that includes the current layer, and (2) the temporal identifier (hereinafter referred to as TemporalId) of the referenced APS NAL unit should be less than or equal to the TemporalId of the picture or coded slice NAL unit associated with the PH.

[0046] Table 2 shows the picture header (PH) that specifies ALF, LMCS, and scaling list parameters at the picture level.

[0047] Table 2:

[0048]

[0049]

[0050] The syntax element ph_pic_parameter_set_id specifies the value of pps_pic_parameter_set_id for the PPS in use. The value of ph_pic_parameter_set_id shall be in the range of 0 to 63, inclusive. Based on the bitstream conformance requirement, the TemporalId value of the PH must be greater than or equal to the TemporalId value of the PPS, where the pps_pic_parameter_set_id of the PPS is equal to ph_pic_parameter_set_id.

[0051] In some embodiments, if the value of nuh_layer_id in the PPS NAL unit whose pps_pic_parameter_set_id is equal to ph_pic_parameter_set_id is less than the value of nuh_layer_id of the coded picture NAL unit, then the value of TemporalId of the PPS whose pps_pic_parameter_set_id is equal to ph_pic_parameter_set_id is restricted to be less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1[curPicVpsLayerId][refPicVpsLayerId]), where refPicVpsLayerId is equal to the ID of the VPS layer of the reference PPS (and curPicVpsLayerId is equal to the VPS layer ID of the picture associated with the PH).

[0052] The syntax element ph_alf_aps_id_luma[i] specifies the adaptation_parameter_set_i of the i-th ALF APS referred to by the luma component of the slice associated with the PH. The value of alf_luma_filter_signal_flag of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_luma[i] shall be equal to 1. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_luma[i] shall be less than or equal to the TemporalId of the picture associated with the PH.

[0053] The syntax element ph_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALFAPS to which the chroma components of the slice associated with the PH refer. The value of alf_chroma_filter_signal_flag of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be equal to 1. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be less than or equal to the TemporalId of the picture associated with the PH.

[0054] The syntax element ph_cc_alf_cb_aps_id specifies the adaptation_parameter_set_id of the ALFAPS to which the Cb chroma component of the slice associated with the PH refers. The value of alf_cc_cb_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_cc_alf_cb_aps_id.

[0055] The syntax element ph_cc_alf_cr_aps_id specifies the adaptation_parameter_set_id of the ALF APS to which the Cr chroma components of the slice associated with the PH refer. The value of alf_cc_cr_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_cc_alf_cr_aps_id.

[0056] The syntax element ph_lmcs_aps_id specifies the adaptation_parameter_set_id of the LMCS APS referenced by the slice associated with the PH.The TemporalId of the APS NAL unit with aps_params_type equal to LMCS_APS and adaptation_parameter_set_id equal to ph_lmcs_aps_id shall be less than or equal to the TemporalId of the picture associated with the PH.

[0057] The syntax element ph_scaling_list_aps_id specifies the adaptation_parameter_set_id of the scaling list APS. The TemporalId of the APS NAL unit with aps_params_type equal to SCALING_APS and adaptation_parameter_set_id equal to ph_scaling_list_aps_id shall be less than or equal to the TemporalId of the picture associated with the PH.

[0058] Table 3 shows a slice header (SH) that specifies ALF, LMCS, and scaling list parameters at the slice level.

[0059] Table 3:

[0060]

[0061] The syntax element slice_alf_aps_id_luma[i] specifies the adaptation_parameter_set_id of the i-th ALF APS to which the luma component of the slice refers. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_luma[i] shall be less than or equal to the TemporalId of the coded slice NAL unit. When slice_alf_enabled_flag is equal to 1 and slice_alf_aps_id_luma[i] is not present, the value of slice_alf_aps_id_luma[i] is inferred to be equal to the value of ph_alf_aps_id_luma[i]. The value of alf_luma_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_luma[i].

[0062] The syntax element slice_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice refer. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be less than or equal to the TemporalId of the coded slice NAL unit. When slice_alf_enabled_flag is equal to 1 and slice_alf_aps_id_chroma is not present, the value of slice_alf_aps_id_chroma is inferred to be equal to the value of ph_alf_aps_id_chroma. The value of alf_chroma_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma.

[0063] The syntax element slice_cc_alf_cb_aps_id specifies the adaptation_parameter_set_id to which the Cb chroma component of the slice refers. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_cc_alf_cb_aps_id shall be less than or equal to the TemporalId of the coded slice NAL unit. When slice_cc_alf_cb_enabled_flag is equal to 1 and slice_cc_alf_cb_aps_id is not present, the value of slice_cc_alf_cb_aps_id is inferred to be equal to the value of ph_cc_alf_cb_aps_id. The value of alf_cc_cb_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_cc_alf_cb_aps_id.

[0064] The syntax element slice_cc_alf_cr_aps_id specifies the adaptation_parameter_set_id to which the Cr chroma components of the slice refer. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_cc_alf_cr_aps_id shall be less than or equal to the TemporalId of the coded slice NAL unit. When slice_cc_alf_cr_enabled_flag is equal to 1 and slice_cc_alf_cr_aps_id is not present, the value of slice_cc_alf_cr_aps_id is inferred to be equal to the value of ph_cc_alf_cr_aps_id. The value of alf_cc_cr_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_cc_alf_cr_aps_id.

[0065] APS is a syntax structure encapsulated in a non-VCL NAL unit (or APS NAL). apsLayerId is set to the nuh_layer_id value of a specific APS NAL unit, and vclLayerId is set to the nuh_layer_id value of a specific VCL NAL unit. In some embodiments, a specific VCL NAL unit is restricted from referencing a specific APS NAL unit unless apsLayerId is less than or equal to vclLayerId and a layer with a nuh_layer_id equal to apsLayerId is included in at least one output layer set (OLS), and the OLS includes a layer with a nuh_layer_id equal to vclLayerId.

[0066] IV. Inter-Layer Referencing

[0067] In some embodiments, pictures in a video sequence can be encoded and decoded according to a hierarchical temporal prediction structure, since multiple layers of video data are used to implement temporally adaptable video. In an inter-layer reference or prediction scheme, pictures in different levels of the hierarchy can reference each other. Each layer of the hierarchical temporal prediction structure can correspond to a sublayer or temporally adaptable layer of a temporally adaptable bitstream, consisting of a VCL NAL unit with a specific temporal identifier (TemporalId in NUH or syntax element nuh_temporal_id_plus1). Each NAL unit is associated with a temporal identifier to indicate a level in the hierarchical temporal prediction structure. The temporal identifier of a NAL unit can also be referred to as a temporal identifier of the video content (e.g., a video picture or a slice of a video picture) carried by the NAL unit.

[0068] In inter-layer reference or prediction, the current picture can be encoded by referring to another picture in another layer and / or another temporal sub-layer. Such a reference picture is called an inter-layer reference picture or ILRP. In some embodiments, the ILRP is in the same access unit (AU) as the current picture, has a nuh_layer_id that is smaller than the nuh_layer_id of the current picture, and is marked as "used for long-term reference". When the video data of the current layer is encoded or decoded by referring to data from another layer as a reference layer, the data of the reference layer can be video data from a VCL NAL unit, or parameters from a non-VCL NAL unit (such as an APS NAL).

[0069] In some embodiments, the inter-layer reference is restricted based on parameters associated with the current layer and the reference layer. Specifically, when decoding the first layer, restrictions are applied to determine whether a picture or APS from the second layer can be used as a reference. For example, if the value of the nuh_layer_id of the reference layer is less than the value of the nuh_layer_id of the current layer and the referenced layer is included in at least one OLS including the current layer, all APS NAL units can be referenced. If the value of the nuh_layer_id of the reference layer is equal to the value of the nuh_layer_id of the current layer, the TemporalId of the referenced APS NAL unit should be less than or equal to the TemporalId of the picture associated with the PH or coded slice NAL unit. Otherwise, if the value of the nuh_layer_id of the reference layer is less than the value of the nuh_layer_id of the current layer, the referenced APS NAL unit is valid regardless of the TemporalId. An APS NAL unit with a specific value of aps_params_type can be ALF_APS, LMCS_APS, or SCALING_APS. This restriction on the relationship between the reference layer and the current layer applies to all syntax elements in PH and SH that are used to indicate the APS identifiers referenced in PH and SH, such as ph_alf_aps_id_luma[], ph_alf_aps_id_chroma, ph_cc_alf_cb_aps_id, ph_cc_alf_cr_aps_id, slice_alf_aps_id_luma[], slice_alf_aps_id_chroma, slice_cc_alf_cb_aps_id, slice_cc_alf_cr_aps_id, ph_lmcs_aps_id and ph_scaling_list_aps_id.

[0070] For example, the syntax element ph_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice associated with the PH refer. The value of alf_chroma_filter_signal_flag of the APS NAL unit with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be equal to 1. If the value of nuh_layer_id of the APS NAL unit is equal to the value of the picture associated with the PH_layer_id, then the TemporalId of the APS NAL unit with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be less than or equal to the TemporalId of the picture associated with the PH.

[0071] As another example, the syntax element slice_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice refer. If the nuh_layer_id value of the APS NAL unit is equal to the nuh_layer_id value of the coded slice NAL unit, then the TemporalId of the APS NAL unit with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be less than or equal to the TemporalId of the coded slice NAL unit. When slice_alf_enabled_flag is equal to 1 and slice_alf_aps_id_chroma is not present, the value of slice_alf_aps_id_chroma is inferred to be equal to the value of ph_alf_aps_id_chroma. The value of alf_chroma_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma.

[0072] In some embodiments, inter-layer prediction is subject to a restriction on the maximum temporal sub-layer or temporal identifier. Table 4A shows the syntax of the video parameter set (VPS), which specifies the restriction on the maximum temporal sub-layer:

[0073] Table 4A:

[0074]

[0075] A value of 1 for max_tid_ref_present_flag[i] specifies that the syntax element max_tid_il_ref_pics_plus1[i] is present. A value of 0 for max_tid_ref_present_flag[i] specifies that the syntax element max_tid_il_ref_pics_plus1[i] is not present.

[0076] The syntax element max_tid_il_ref_pics_plus1[i] sent at the VPS is used to limit the maximum temporal sub-layer allowed as a reference for inter-layer prediction of layer i. A value of max_tid_il_ref_pics_plus1[i] of 0 specifies that non-IRAP pictures of layer i do not use inter-layer prediction. A value of max_tid_il_ref_pics_plus1[i] greater than 0 specifies that for picture decoding of layer i, no pictures with a TemporalId greater than max_tid_il_ref_pics_plus1[i]-1 are used as ILRPs. When not present, the value of max_tid_il_ref_pics_plus1[i] is inferred to be equal to 7. In some embodiments, when not present, the value of max_tid_il_ref_pics_plus1[i] is inferred to be equal to vps_max_sublayers_minus1 plus 2.

[0077] The value of the syntax element vps_max_sublayers_minus1 plus 1 specifies the maximum number of temporal sublayers that can be present in a layer in each CVS of the referenced VPS. The value of vps_max_sublayers_minus1 is constrained to be in the range of 0 to 6, inclusive.

[0078] In some embodiments, the value of max_tid_il_ref_pics_plus1[i] is restricted to the range of 0 to vps_max_sublayers_minus1+1, inclusive. When vps_max_sublayers_minus1 is equal to 0, the value of max_tid_il_ref_pics_plus1[i] is inferred to be equal to 0. When vps_max_sublayers_minus1 is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, the value of max_tid_il_ref_pics_plus1[i] is inferred to be equal to vps_max_sublayers_minus1 plus 1. In some embodiments, max_tid_il_ref_pics_plus1 is sent as only one (common) value for all layers in the CVS of the reference VPS. In this case, the value is restricted to be no greater than the minimum number of sublayers in all layers of the CVS of the reference VPS. In some embodiments, when vps_max_sublayers_minus1 is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, only one value of max_tid_il_ref_pics_plus1 is sent for all layers in the CVS.

[0079] The syntax element max_tid_il_ref_pics_plus1[i] in the VPS of Table 4A specifies a restriction on the pictures that can be used as reference (or ILRP) for decoding the i-th layer in inter-layer prediction. In some embodiments, the signaling of the restriction on the maximum temporal sub-layer allowed for inter-layer prediction can be further generalized. For example, for decoding the i-th layer in inter-layer prediction, the syntax element max_tid_il_ref_pics_plus1[i][j] restricts whether pictures from the j-th layer can be used as reference in a two-dimensional manner. Specifically, a value of max_tid_il_ref_pics_plus1[i][j] equal to 0 specifies that the j-th layer is not used as a reference layer for inter-layer prediction of non-IRAP pictures of the i-th layer. A value of max_tid_il_ref_pics_plus1[i][j] greater than 0 indicates that for decoding pictures of the i-th layer, pictures from the j-th layer with a TemporalId greater than max_tid_il_ref_pics_plus1[i][j]-1 are not used as ILRPs. When not present, the value of max_tid_il_ref_pics_plus1[i][j] is inferred to be equal to 7. Table 4B shows a VPS that specifies restrictions on inter-layer prediction for decoding and reference layers:

[0080] Table 4B:

[0081]

[0082] A value of max_tid_ref_present_flag[i] equal to 1 specifies that the syntax element max_tid_il_ref_pics_plus1[i][j] is present. A value of max_tid_ref_present_flag[i] equal to 0 specifies that the syntax element max_tid_il_ref_pics_plus1[i][j] is not present.

[0083] Figure 4 This conceptual illustration illustrates the restrictions on inter-layer prediction. The diagram above illustrates a current picture 0400 to be encoded or decoded using inter-layer prediction. Specifically, the current picture 0400 has at least one slice containing video content in layer 0410, which is to be encoded or decoded with reference to video content in layer 0420. Layer 0420 has video content in different temporal sublayers 0421-0426, which are assigned corresponding temporal identifiers (TemporalId). The different temporal sublayers correspond to different levels in a hierarchical temporal prediction structure (for temporally adaptable video).

[0084] Whether the video content of a sub-layer can be referenced by the current picture 0400 for inter-layer prediction is subject to a restriction 0430 regarding the maximum temporal identifier between the i-th layer 0410 and the j-th layer 0420 (e.g., the syntax element max_tid_il_ref_pics_plus1[i][j]). In this example, sub-layers 0421-0423 have temporal identifiers that satisfy restriction 0430 (≤ maximum temporal identifier), and pictures in these sub-layers can be used as reference pictures (ILRP) for encoding or decoding the current picture 0400 for inter-layer prediction. On the other hand, the temporal identifiers of sub-layers 0424-0426 do not satisfy restriction 0430 (> maximum temporal identifier), and pictures in these sub-layers cannot be used as reference pictures for encoding or decoding the current picture 0400 for inter-layer prediction.

[0085] The above diagram also shows the kth layer 0415. In some embodiments, under the constraint of the maximum temporal identifier between the i-th layer and the k-th layer, the pictures in the temporal sub-layer of the k-th layer can be used as reference layers for inter-layer prediction of the i-th layer 0410 (e.g., max_tid_il_ref_pics_plus1[i][k]).

[0086] In some embodiments, the reference picture list construction is subject to a set of constraints. For example, in some embodiments, the pictures referenced by each ILRP entry in RefPicList[0] and / or RefPicList[1] of the slice of the current picture are constrained to have a TemporalId less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_picLayerId][refPicVpsLayerId]), where refPicVpsLayerId is equal to the GeneralLayerIdx[refVpsLayerId] of the reference picture (and curPicVpsLayerId is equal to GeneralLayerIdx[curVpsLayerId]) or is an IRAP. For example, the picture referenced by each ILRP entry in RefPicList[0] and / or RefPicList[1] of the slice of the current picture has a TemporalId less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1[curPic], VpsLayerrefLayerVic), where refPicVpsLayerId is equal to the VPS layer ID of the reference picture (curPicVpsLayerId is equal to the VPS layer ID of the current picture), or is an IRAP. Figure 4 In FIG, based on the maximum temporal identifier limit 0430, the ILRP entry of (a slice of) the current picture 0400 can refer to pictures in sub-layers 0421-0423 instead of pictures in sub-layers 0424-0426.

[0087] In some embodiments, restrictions on the referenced APS NAL units may also be applied. For example, if the value of nuh_layer_id of an APS NAL unit is less than the value of nuh_layer_id of the picture associated with the PH or coded slice NAL unit, the TemporalId of the APS NAL unit is restricted to be less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1[curPicVpsLayerId][refPicVpsLayerId]), where refPicVpsLayerId is equal to the VPS layer ID of the reference layer or APS NAL unit (and curPicVpsLayerId is equal to the VPS layer ID of the picture associated with the NAL coded slice). APS NAL units with a specific value of aps_params_type can be ALF_APS, LMCS_APS, or SCALING_APS. This restriction applies to syntax elements used to indicate APS identifiers referenced in PH and SH, such as ph_alf_aps_id_luma[], ph_alf_aps_id_chroma, ph_cc_alf_cb_aps_id, ph_cc_alf_cr_aps_id, slice_alf_aps_id_luma[], slice_alf_aps_id_chroma, slice_cc_alf_cb_aps_id, slice_cc_alf_cr_aps_id, ph_lmcs_aps_id, and ph_scaling_list_aps_id.

[0088] For example, the syntax element ph_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice associated with the PH refer. The value of alf_chroma_filter_signal_flag of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be equal to 1. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be less than or equal to the TemporalId of the picture associated with the PH. If the value of nuh_layer_id of the APS NAL unit is less than the value of nuh_layer_id of the picture associated with the PH, the TemporalId of the referenced APS NAL unit shall be less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1V), where refPicVpsLayerId is equal to the VPS layer ID of the reference layer or APS NAL unit (curPicVpsLayerId is equal to the VPS layer ID of the picture associated with the PH).

[0089] As another example, the syntax element slice_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice refer. The TemporalId of APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be less than or equal to the TemporalId of the coded slice NAL unit. If the nuh_layer_id value of the APS NAL unit is less than the nuh_layer_id value of the coded slice NAL unit, the TemporalId of the referenced APS NAL unit shall be less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1[curPicVpsLayerId][refPicVpsLayerId]) where refPicVpsLayerId is equal to the VPS layer ID of the reference layer or APS NAL unit (and curPicVpsLayerId is equal to the VPS layer ID of the coded slice NAL unit).

[0090] When slice_alf_enabled_flag is equal to 1 and slice_alf_aps_id_chroma is not present, the value of slice_alf_aps_id_chroma is inferred to be equal to the value of ph_alf_aps_id_chroma.The value of alf_chroma_filter_signal_flag shall be equal to 1 for APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma.

[0091] In some embodiments, the restriction on the maximum sublayer and the restriction on the referenced APS NAL units can be combined. An APS NAL unit with a specific value of aps_params_type can be ALF_APS, LMCS_APS, or SCALING_APS. The combined restriction applies to syntax elements used to indicate reference to APS identifiers in PH and SH, such as ph_alf_aps_id_luma[], ph_alf_aps_id_chroma, ph_cc_alf_cb_aps_id, ph_cc_alf_cr_aps_id, slice_alf_aps_id_luma[], slice_alf_aps_id_chroma, slice_cc_alf_cb_aps_id, slice_cc_alf_cr_aps_id, ph_lmcs_aps_id, and ph_scaling_list_aps_id.

[0092] For example, the syntax element ph_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice associated with the PH refer. In APS NAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma, the value of alf_chroma_filter_signal_flag shall be equal to 1. If the value of nuh_layer_id of the APS NAL unit is equal to the value of the picture associated with the PH_layer_id, the TemporalId of the APS NAL unit with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to ph_alf_aps_id_chroma shall be less than or equal to the TemporalId of the picture associated with the PH. Otherwise (the value of nuh_layer_id of the APS NAL unit is less than the value of nuh_layer_id of the picture associated with the PH), the TemporalId of the referenced APS NAL unit shall be less than max_tid_il_ref_pics_plus1[refPicVpsLayerId] (or max_tid_il_ref_pics_plus1[curPicVpsLayerId][refPicVpsLayerId]), where refPicVpsLayerId is equal to the VPS layer ID of the reference layer or APS NAL unit (and curPicVpsLayerId is equal to the layer ID of the picture associated with the PH).

[0093] As another example, the syntax element slice_alf_aps_id_chroma specifies the adaptation_parameter_set_id of the ALF APS to which the chroma components of the slice refer. If the nuh_layer_id value of the APS NAL unit is equal to the nuh_layer_id value of the coded slice NAL unit, the TemporalId of the APS NAL unit with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be less than or equal to the TemporalId of the coded slice NAL unit. Otherwise (the value of nuh_layer_id of the APS NAL unit is less than the value of nuh_layer_id of the coded slice NAL unit), the TemporalId of the referenced APS NAL unit shall be less than max_tid_il_ref_pics_plus1[ refPicVpsLayerId ]( or max_tid_il_ref_picsLayerId ]( or max_tid_il_ref_picLayerIds[VPSLayer]VPSLayerV), where refPicVpsLayerId is equal to the reference layer or APS The VPS layer ID of the NAL unit (and curPicVpsLayerId is equal to the VPS layer ID of the coded slice NAL unit). When slice_alf_enabled_flag is equal to 1 and slice_alf_aps_id_chroma is not present, the value of slice_alf_aps_id_chroma is inferred to be equal to the value of ph_alf_aps_id_chroma. The value of alf_chroma_filter_signal_flag shall be equal to 1 for APSNAL units with aps_params_type equal to ALF_APS and adaptation_parameter_set_id equal to slice_alf_aps_id_chroma.

[0094] V. Sub-bitstream extraction process

[0095] In some embodiments, certain NAL units in the bitstream can be extracted from the bitstream as a sub-bitstream. The sub-bitstream extraction process is a specific process that removes NAL units in the bitstream that do not belong to a target set (determined by the target OLS index and the target highest TemporalId) from the bitstream and outputs a sub-bitstream (called OutBitstream) consisting of the NAL units in the bitstream that belong to the target set.

[0096] In some embodiments, the output sub-bitstream OutBitstream is derived as follows:

[0097] – The bitstream outBitstream is set to be the same as the bitstream inBitstream.

[0098] – Remove from outBitstream all NAL units whose TemporalId is greater than tIdTarget.

[0099] – Remove from outBitstream all NAL units whose nal_unit_type is not equal to any of VPS_NUT, DCI_NUT and EOB_NUT and whose nuh_layer_id is not contained in the list LayerIdInOls[targetOlsIdx].

[0100] – Remove from outBitstream all NAL units that satisfy all of the following conditions:

[0101] –nal_unit_type is not equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0102] –nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j], where the value of j is in the range of 0 to NumLayersInOls[targetOlsIdx]-1 (inclusive).

[0103] – TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][j].

[0104] In some embodiments, NAL units can be removed or extracted from the outBitstream if certain conditions are met. For example, if the TemporalId of a NAL unit is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][j], or if all NAL units have a nal_unit_type in the range of 0 to 13 and are not equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT, the NAL unit is deleted from the outBitstream. In some embodiments, the restriction on NAL unit removal applies only to VCL NAL units. In other words, the restriction does not apply to non-VCL units.

[0105] VI. Example Video Encoder

[0106] Figure 5 An example video encoder 0500 that can perform inter-layer prediction is illustrated. As shown, video encoder 0500 receives an input video signal from a video source 0505 and encodes the signal into a bitstream 0595. Video encoder 0500 includes several components or modules for encoding the signal from video source 0505, including at least some components selected from a transform module 0510, a quantization module 0511, an inverse quantization module 0514, an inverse transform module 0515, an intra-picture estimation module 0520, an intra-prediction module 0525, a motion compensation module 0530, a motion estimation module 0535, a loop filter 0545, a reconstructed picture buffer 0550, an MV buffer 0565, an MV prediction module 0575, and an entropy encoder 0590. Motion compensation module 0530 and motion estimation module 0535 are part of inter-prediction module 0540.

[0107] In some embodiments, modules 0510-0590 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic apparatus. In some embodiments, modules 0510-0590 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 0510-0590 are shown as separate modules, some of these modules may be combined into a single module.

[0108] Video source 0505 provides a raw video signal representing uncompressed pixel data for each video frame. Subtractor 0508 calculates the difference between the raw video pixel data from video source 0505 and the predicted pixel data 0513 from motion compensation module 0530 or intra-frame prediction module 0525. Transform module 0510 converts the difference (or residual pixel data or residual signal) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). Quantization module 0511 quantizes the transform coefficients 516 into quantized data (or quantized coefficients) 0512, which are encoded into a bitstream 0595 by entropy encoder 0590.

[0109] Inverse quantization module 0514 inverse quantizes quantized data (or quantized coefficients) 0512 to obtain transform coefficients, and inverse transform module 0515 performs an inverse transform on the transform coefficients to generate a reconstructed residual 0519. Reconstructed residual 0519 is added to predicted pixel data 0513 to generate reconstructed pixel data 0517. In some embodiments, reconstructed pixel data 0517 is temporarily stored in a line buffer (not shown) for use in intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by loop filter 0545 and stored in reconstructed picture buffer 0550. In some embodiments, reconstructed picture buffer 0550 is a memory external to video encoder 0500. In some embodiments, reconstructed picture buffer 0550 is a memory internal to video encoder 0500.

[0110] The intra picture estimation module 0520 performs intra prediction based on the reconstructed pixel data 0517 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 0590 to be encoded into a bitstream 0595. The intra prediction module 0525 also uses the intra prediction data to generate predicted pixel data 0513.

[0111] The motion estimation module 0535 performs inter-frame prediction by generating MVs to refer to pixel data of a previously decoded frame stored in the reconstructed picture buffer 0550. These MVs are provided to the motion compensation module 0530 to generate predicted pixel data.

[0112] Instead of encoding the complete actual MV in the bitstream, the video encoder 0500 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 0595.

[0113] MV prediction module 0575 generates a predicted MV based on a reference MV generated for encoding the previous video frame, i.e., a motion-compensated MV used for motion compensation. MV prediction module 0575 retrieves the reference MV from the previous video frame from MV buffer 0565. Video encoder 0500 stores the MV generated for the current video frame in MV buffer 0565 as a reference MV for generating the predicted MV.

[0114] The MV prediction module 0575 uses the reference MV to create a predicted MV. The predicted MV can be calculated using spatial MV prediction or temporal MV prediction. The difference between the predicted MV and the motion-compensated MV (MC MV) for the current frame (residual motion data) is encoded into the bitstream 0595 by the entropy encoder 0590.

[0115] The entropy encoder 0590 encodes various parameters and data into a bitstream 0595 using an entropy coding technique such as adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 0590 encodes various header elements, flags, and quantized transform coefficients 0512 and residual motion data as syntax elements into the bitstream 0595. The bitstream 0595 is in turn stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0116] The loop filter 0545 performs a filtering or smoothing operation on the reconstructed pixel data 0517 to reduce encoding artifacts, particularly at pixel block boundaries. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

[0117] Figure 6 The portion of video encoder 0500 that implements inter-layer prediction restrictions is illustrated. As shown, when the video encoder is performing inter-layer prediction to encode a video picture, motion estimation module 0535 provides reference picture selection by identifying the layer and sub-layer of the reference picture to motion compensation module 0530. The motion compensation module, in turn, retrieves the selected video content from reconstructed picture buffer 0550, which stores video content for pictures in different layers and temporal sub-layers. The reference picture selection is also provided to entropy encoder 0590 for encoding into bitstream 0595.

[0118] The video encoder 0500 determines a set of inter-layer prediction restriction parameters 0600. The inter-layer prediction restriction parameter set 0600 can specify maximum sub-layer or temporal identifier limits for different combinations of coded layers and referenced layers (e.g., max_tid_il_ref_pics_plus1[i][j] for different i and j pairs). The entropy encoder 0590 encodes the set of inter-layer prediction restriction parameters 0600 into the bitstream 0595 (e.g., VPS).

[0119] The inter-layer prediction restriction parameter 0600 is used to configure the reconstructed picture buffer 0550 and the motion estimation module 0535. The reconstructed picture buffer 0550 can be optimized for inter-layer prediction (e.g., only storing data for layers and sub-layers suitable for ILRP). The inter-layer prediction restriction parameter 0600 is also used to configure the motion estimation module 0535 to limit its search for reference pictures in the reference layer to those that satisfy the restriction parameter 0600.

[0120] Figure 7The process 0700 of encoding a video picture using inter-layer prediction is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 0500 perform the process 0700 by executing instructions stored in a computer-readable medium. In some embodiments, the electronic device implementing the encoder 0500 performs the process 0700.

[0121] In step 710, an encoder receives video content to be encoded into a bitstream as video pictures in multiple different layers. At least one of the multiple layers includes multiple temporal sub-layers corresponding to levels in a hierarchical temporal prediction structure. Each temporal sub-layer is associated with a temporal identifier. The data for the video pictures is carried in network abstraction layer (NAL) units.

[0122] In step 720, the encoder signals an inter-layer prediction restriction parameter (e.g., a restriction on the maximum temporal sub-layer or max_tid_il_ref_pics_plus1) in the bitstream to limit the maximum temporal sub-layer used in inter-layer prediction. In some embodiments, the inter-layer prediction restriction parameter is signaled in the VPS applicable to the first picture when a specific flag in the VPS (e.g., max_tid_ref_present_flag) indicates that the inter-layer prediction restriction parameter is present.

[0123] In step 730, the encoder encodes the first picture in the first layer by referencing data of the second picture in the second layer. The temporal identifier of the reference data satisfies the received inter-layer prediction restriction parameter. The temporal identifier of the reference data is encoded in a header of a NAL unit that carries video content of the second picture in the second layer. In some embodiments, the temporal identifier of the reference data satisfies the restrictions of the received inter-layer prediction restriction parameter when the temporal identifier is less than or equal to a value indicated by the inter-layer prediction restriction parameter. In some embodiments, when the inter-layer prediction restriction parameter is set to zero, the video encoder may disable inter-layer prediction for the first picture.

[0124] When the third picture is associated with a temporal identifier greater than a value indicated by the inter-layer prediction restriction parameter, the video encoder may exclude the third picture from being used as a reference for inter-layer prediction of the first picture.

[0125] In some embodiments, the inter-layer prediction restriction parameter applies only when the second layer is used as a reference layer for inter-layer prediction of the first layer, and does not apply when another layer is used as a reference layer for inter-layer prediction of the first layer. When the inter-layer prediction restriction parameter is set to zero, the second layer is not used as a reference layer for encoding the first layer using inter-layer prediction. In some embodiments, each entry in an inter-layer reference prediction (ILRP) list of a (slice) of a first picture is an IRAP picture or a reference picture in a reference layer with a temporal identifier less than an inter-frame identifier, wherein the inter-layer prediction restriction parameter applies to the first layer and the reference layer.

[0126] VII. Example Video Decoding

[0127] In some embodiments, the encoder may send (or generate) one or more syntax elements in a bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0128] Figure 8 An example video decoder 0800 capable of performing inter-layer prediction is illustrated. As shown, video decoder 0800 is a picture decoding or video decoding circuit that receives a bitstream 0895 and decodes the contents of the bitstream into pixel data for a video frame for display. Video decoder 0800 includes several components or modules for decoding bitstream 0895, including a selection from an inverse quantization module 0811, an inverse transform module 0810, an intra-frame prediction module 0825, a motion compensation module 0830, a loop filter 0845, a decoded picture buffer 0850, an MV buffer 0865, an MV prediction module 0875, and a parser 0890. Motion compensation module 0830 is part of inter-frame prediction module 0840.

[0129] In some embodiments, modules 0810-0890 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 0810-0890 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 0810-0890 are shown as separate modules, some of these modules may be combined into a single module.

[0130] The parser 0890 (or entropy decoder) receives the bitstream 0895 and performs initial parsing according to the syntax defined by the video coding or picture coding standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 0812. The parser 0890 parses the various syntax elements using entropy coding techniques such as adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0131] Inverse quantization module 0811 inversely quantizes quantized data (or quantized coefficients) 0812 to obtain transform coefficients 0816. Inverse transform module 0810 inversely transforms transform coefficients 0816 to generate reconstructed residual signal 0819. Reconstructed residual signal 0819 is added to predicted pixel data 0813 from intra prediction module 0825 or motion compensation module 0830 to generate decoded pixel data 0817. The decoded pixel data is filtered by loop filter 0845 and stored in decoded picture buffer 0850. In some embodiments, decoded picture buffer 0850 is a memory external to video decoder 0800. In some embodiments, decoded picture buffer 0850 is a memory internal to video decoder 0800.

[0132] The intra prediction module 0825 receives intra prediction data from the bitstream 0895 and generates predicted pixel data 0813 based on the data from the decoded pixel data 0817 stored in the decoded picture buffer 0850. In some embodiments, the decoded pixel data 0817 is also stored in a line buffer (not shown) for intra-picture prediction and spatial MV prediction.

[0133] In some embodiments, the contents of the decoded picture buffer 0850 are used for display. The display device 0855 either directly accesses the contents of the decoded picture buffer 0850 for display, or accesses the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 0850 via pixel transfer.

[0134] The motion compensation module 0830 generates predicted pixel data 0813 from the decoded pixel data 0817 stored in the decoded picture buffer 0850 according to motion compensated MVs (MC MVs). These motion compensated MVs are decoded by adding residual motion data received from the bitstream 0895 to predicted MVs received from the MV prediction module 0875.

[0135] MV prediction module 0875 generates a predicted MV based on a reference MV generated for decoding a previous video frame, such as a motion-compensated MV used for performing motion compensation. MV prediction module 0875 retrieves the reference MV for the previous video frame from MV buffer 0865. Video decoder 0800 stores the motion-compensated MV generated for decoding the current video frame in MV buffer 0865 as a reference MV for generating the predicted MV.

[0136] The loop filter 0845 performs a filtering or smoothing operation on the decoded pixel data 0817 to reduce encoding artifacts, particularly at pixel block boundaries. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

[0137] Figure 9 The portion of video decoder 0800 that implements inter-layer prediction restrictions is illustrated. As shown, when the video decoder is performing inter-layer prediction to decode a video picture, entropy decoder 0890 provides reference picture selection by identifying the reference picture's layer and sub-layer to motion compensation module 0830. The motion compensation module, in turn, retrieves the selected video content from decoded picture buffer 0850, which stores video content from different layers and sub-layers.

[0138] The entropy decoder 0890 receives a set of inter-layer prediction restriction parameters 0900 from the bitstream 0895. The inter-layer prediction restriction parameter set 0900 may specify maximum sub-layer or temporal identifier limits for different combinations of coded layers and referenced layers (e.g., max_tid_il_ref_pics_plus1[i][j] for different i and j pairs).

[0139] The inter-layer prediction restriction parameters 0900 are used to configure the decoded picture buffer 0850. In some embodiments, the decoded picture buffer 0850 may be optimized for inter-layer prediction (eg, only storing data for layers and sub-layers that qualify as ILRP).

[0140] Figure 10 The process 1000 for decoding a video picture using inter-layer prediction is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement decoder 0800 to perform process 1000 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 0800 performs process 1000.

[0141] In step 1010, a decoder receives data from a bitstream that carries data for a video picture in multiple different layers. At least one of the multiple layers has multiple temporal sub-layers corresponding to levels in a hierarchical temporal prediction structure. Each temporal sub-layer is associated with a temporal identifier. The video picture data is carried in a Network Abstraction Layer (NAL) unit.

[0142] In step 1020, the decoder receives an inter-layer prediction restriction parameter (e.g., max_tid_il_ref_pics_plus1) that limits the maximum temporal sub-layer used in inter-layer prediction. In some embodiments, the inter-layer prediction restriction parameter is signaled in the VPS applicable to the first picture when a specific flag in the VPS (e.g., max_tid_ref_present_flag) indicates that the inter-layer prediction restriction parameter is present.

[0143] In step 1030, the decoder reconstructs the first picture in the first layer by referencing data of the second picture in the second layer. The temporal identifier of the reference data satisfies the constraints of the received inter-layer prediction restriction parameter. The temporal identifier of the reference data is encoded in a header of a NAL unit that carries video content of the second picture in the second layer. In some embodiments, the temporal identifier of the reference data satisfies the constraints of the received inter-layer prediction restriction parameter when the temporal identifier is less than or equal to a value indicated by the inter-layer prediction restriction parameter. In some embodiments, when the inter-layer prediction restriction parameter is set to zero, the video decoder may disable inter-layer prediction for the first picture.

[0144] When the third picture is associated with a temporal identifier greater than a value indicated by the inter-layer prediction restriction parameter, the video decoder may exclude the third picture from being used as a reference for inter-layer prediction of the first picture.

[0145] In some embodiments, the inter-layer prediction restriction parameter applies only when the second layer is used as a reference layer for inter-layer prediction of the first layer, and does not apply when another layer is used as a reference layer for inter-layer prediction of the first layer. When the inter-layer prediction restriction parameter is set to zero, the second layer is not used as a reference layer for decoding the first layer using inter-layer prediction. In some embodiments, each entry in an inter-layer reference prediction (ILRP) list of a (slice) of a first picture is an IRAP picture or a reference picture in a reference layer with a temporal identifier less than the inter-frame prediction restriction parameter, wherein the inter-layer prediction restriction parameter applies to both the first layer and the reference layer.

[0146] VIII. Example Electronic System

[0147] Many of the above features and applications are implemented as software processes specified as a set of instructions recorded on a computer-readable medium (also referred to as computer-readable media). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media does not include carrier waves and electronic signals transmitted over wireless or wired connections.

[0148] In this specification, the term "software" is intended to include firmware residing in read-only memory or application programs stored in magnetic memory that can be read into memory for processing by a processor. Furthermore, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement the software inventions described herein is within the scope of the present disclosure. In some embodiments, when the software program is installed to run on one or more electronic systems, the software program defines one or more specific machine implementations that execute and perform the operations of the software program.

[0149] Figure 11 An electronic system 1100 is conceptually illustrated, and some embodiments of the present disclosure may be implemented therein. Electronic system 1100 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system 1100 includes a bus 1105, a processing unit 1110, a graphics processing unit (GPU) 1115, system memory 1120, a network 1125, read-only memory 1130, a permanent memory storage device 1135, input devices 1140, and output devices 1145.

[0150] Bus 1105 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of electronic system 1100. For example, bus 1105 communicatively connects processing unit 1110 with GPU 1115, read-only memory 1130, system memory 1120, and permanent storage device 1135.

[0151] From these various memory units, processing unit 1110 retrieves instructions to execute and data to process in order to perform the processes of the present disclosure. In various embodiments, the processing unit may be a single-core processor or a multi-core processor. Some instructions are passed to and executed by GPU 1115. GPU 1115 can offload various computations or supplement the image processing provided by processing unit 1110.

[0152] Read-only memory (ROM) 1130 stores static data and instructions used by processing unit 1110 and other modules of the electronic system. Persistent storage device 1135, on the other hand, is a read-write storage device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 1100 is turned off. Some embodiments of the present disclosure use a mass storage device (e.g., a magnetic or optical disk and its corresponding disk drive) as persistent storage device 1135.

[0153] Other embodiments use removable storage devices (such as floppy disks, pen drives, etc., and their corresponding disk drives) as permanent storage devices. Like the permanent storage device 1135, the system memory 1120 is a readable and writable storage device. However, unlike the storage device 1135, the system memory 1120 is a volatile read-write memory, such as a random access memory. The system memory 1120 stores some instructions and data used by the processor at runtime. In some embodiments, the processes according to the present disclosure are stored in the system memory 1120, the permanent storage device 1135 and / or the read-only memory 1130. For example, according to some embodiments, various memory units include instructions for processing multimedia clips. From these different memory units, the processing unit 1110 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

[0154] Bus 1105 is also connected to input and output devices 1140 and 1145. Input device 1140 enables a user to convey information and select commands to the electronic system. Input device 1140 includes an alphanumeric keyboard and cursor device (also known as a "cursor control device"), a camera (such as a webcam), a microphone or similar device for receiving voice commands, etc. Output device 1145 displays images generated by the electronic system or outputs data in other ways. Output device 1145 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), as well as a speaker or similar audio output device. Some embodiments include devices used as input and output devices, such as a touch screen.

[0155] Finally, if Figure 11 As shown, bus 1105 also connects electronic system 1100 to a network 1125 via a network adapter (not shown). In this manner, a computer network (e.g., a local area network (LAN), a wide area network (WAN), or an intranet), or a network of networks, such as any or all components of electronic system 110, may be used in conjunction with the present disclosure.

[0156] Some embodiments include storing computer program instructions on a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disc read-only memory (CD-ROM), compact disc recordable memory (CD-R), compact disc rewritable memory (CD-RW), digital versatile disc read-only memory (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable CDs, ultra-high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0157] While the above discussion primarily involves microprocessors or multi-core processors executing software, many of the aforementioned features and applications are implemented by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. Additionally, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0158] As used in this specification and any claims of this application, the terms "computer," "server," "processor," and "memory" refer to electronic or other technological devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms display or showing mean displaying on an electronic device. The terms "computer-readable medium," "computer-readable media," and "machine-readable medium" as used in this specification and any claims of this application are strictly limited to tangible, physical objects that store information in a form that can be read by a computer. These terms exclude any wireless signals, wired download signals, and any other temporary signals.

[0159] Although the present disclosure has been described with reference to many specific details, one of ordinary skill in the art will recognize that the present disclosure can be embodied in other specific forms without departing from the spirit of the present disclosure. Figure 7 and Figure 10) conceptually illustrate the processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processes may be implemented using several sub-processes or as part of a larger macro-process. Therefore, it will be understood by those skilled in the art that the present disclosure is not limited by the foregoing illustrative details, but is defined by the appended claims.

[0160] The subject matter described herein sometimes illustrates that different components are contained within or connected to different other components. It should be understood that the architectures described in this manner are merely examples, and that many other architectures that can actually implement the same functionality are conceptually "associated" to achieve the desired functionality. Therefore, any two components combined herein to achieve a particular functionality can be considered to be "associated" to each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two components that are so associated can also be considered to be "operably connected" or "operably coupled" to achieve the desired functionality, and any two components that can be so associated can also be considered to be "operably connected" or "operably coupled" to achieve the desired functionality. Specific examples of operable coupling include, but are not limited to, physically matable and / or physically interactive components and / or wirelessly interactive and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0161] In addition, with respect to the use of substantially any plural and / or singular terms herein, those skilled in the art can translate from the plural to the singular and / or from the singular to the plural and / or application depending on the context. For clarity, various singular / plural arrangements may be explicitly set forth herein.

[0162] Furthermore, those skilled in the art will understand that, in general, the terms used herein, and particularly in the appended claims, such as the bodies of the appended claims, are generally intended to be “open” terms, e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” and so forth. Those skilled in the art will further understand that if a specific number of claim references is intended to be introduced, such intent will be explicitly recited in the claim, and that no such intent is present in the absence of such a recitation. For example, to aid understanding, the following appended claims may contain statements that use the introductory phrases “at least one” and “one or more” to introduce claim references. However, the use of such phrases should not be construed as introducing claim references to limit any particular claim containing such introduced claim references to embodiments containing only one such reference, even when the same claim includes the introductory phrases “one or more” or “at least one” and an indefinite article, such as “a”, should be interpreted as “at least one” or “one or more”; the same applies to the use of definite articles to introduce claim references. Furthermore, even when explicit reference is made to a specific number of an incorporated claim reference, those skilled in the art will recognize that such reference should be interpreted as referring to at least the referenced number, e.g., a bare reference without other modifiers such as "two references" refers to at least two or more references. Furthermore, this also applies in common contexts such as "at least one of A, B, and C, etc." Generally speaking, such specificity is intended to be understood by those skilled in the art in a conventional sense, e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, A, B, or C, etc. Those skilled in the art will further understand that, in practice, any disjunctive word and / or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to encompass the possibility of including one, either, or both of the terms. For example, the phrase "A or B" would be understood to include the possibility of "A" or "B" or "A and B."

[0163] From the foregoing, it should be understood that various embodiments of the present disclosure have been described herein for illustrative purposes and that various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. A video decoding method, comprising: receiving data from a bitstream carrying video pictures in a plurality of different layers, wherein at least one of the plurality of different layers comprises a plurality of temporal sub-layers corresponding to a plurality of levels in a hierarchical temporal prediction structure, each of the temporal sub-layers being associated with a temporal identifier; receiving an inter-layer prediction restriction parameter indicating a maximum value of a temporal identifier; and Inter-layer prediction is performed by referring to reference data of a second picture in a second layer to reconstruct a first picture in a first layer, wherein the second layer is at least one layer of the multiple different layers, wherein the maximum value of the temporal identifier is for the first layer and the second layer, and the temporal identifier corresponding to the second picture corresponding to the second layer satisfies the restrictions of the received inter-layer prediction restriction parameters.

2. The video decoding method according to claim 1, wherein: The video picture data is carried by a Network Abstraction Layer (NAL) unit, and a time identifier of the reference data is encoded in a header of the NAL unit carrying the second picture in the second layer.

3. The video decoding method according to claim 1, wherein: When a specific flag in a video parameter set (VPS) indicates that the inter-layer prediction restriction parameter exists, the inter-layer prediction restriction parameter is signaled in the video parameter set applicable to the first picture.

4. The video decoding method according to claim 1, wherein: When the temporal identifier is less than or equal to the value indicated by the inter-layer prediction restriction parameter, the temporal identifier of the reference data satisfies the restriction of the received inter-layer prediction restriction parameter.

5. The video decoding method according to claim 1, wherein: The method further includes excluding the third picture from being used as a reference for inter-layer prediction of the first picture when the temporal identifier of the third picture is greater than a value indicated by the inter-layer prediction restriction parameter.

6. The video decoding method according to claim 1, wherein: Also includes: When the inter-layer prediction restriction parameter is set to zero, inter-layer prediction of the first picture is not allowed.

7. The video decoding method according to claim 1, wherein: When the second layer is used as a reference layer for inter-layer prediction of the first layer, the inter-layer prediction restriction parameter is applicable, whereas when another layer is used as the reference layer for inter-layer prediction of the first layer, the inter-layer prediction restriction parameter is not applicable.

8. The video decoding method according to claim 7, wherein: When the inter-layer prediction restriction parameter is set to zero, the second layer is not used as a reference layer for the inter-layer prediction decoding of the first layer.

9. The video decoding method according to claim 1, wherein: An entry in the inter-layer reference prediction (ILRP) list of the first picture is a reference picture in a reference layer with a temporal signature that satisfies the inter-layer prediction restriction parameter restrictions applicable to the first layer and the reference layer.

10. The video decoding method according to claim 1, wherein: Each entry in the inter-layer reference prediction (ILRP) list for the first picture is an intra random access point picture (IRAP) or a reference picture in a reference layer having a temporal identifier that satisfies inter-layer prediction restriction parameters applicable to the first layer and the reference layer.

11. A video encoding method, comprising: Receiving video content encoded into a bitstream of video pictures in a plurality of different layers, wherein at least one of the plurality of different layers comprises a plurality of temporal sub-layers corresponding to a plurality of levels in a hierarchical temporal prediction structure, each temporal sub-layer being associated with a temporal identifier; signaling, in the bitstream, an inter-layer prediction restriction parameter indicating a maximum value of a temporal identifier; as well as The first picture in the first layer is encoded by referring to the reference data of the second picture in the second layer to perform inter-layer prediction, wherein the second layer is at least one layer described in the multiple different layers, wherein the maximum value of the temporal identifier is for the first layer and the second layer, and the temporal identifier corresponding to the second picture corresponding to the second layer satisfies the restrictions of the inter-layer prediction restriction parameters.

12. An electronic device comprising: A video decoder circuit is configured to perform operations including: receiving data from a bitstream carrying video pictures in a plurality of different layers, wherein at least one of the plurality of different layers comprises a plurality of temporal sub-layers corresponding to a plurality of levels in a hierarchical temporal prediction structure, each of the temporal sub-layers being associated with a temporal identifier; receiving an inter-layer prediction restriction parameter indicating a maximum value of a temporal identifier; and Inter-layer prediction is performed by referring to reference data of a second picture in a second layer to reconstruct a first picture in a first layer, wherein the second layer is at least one layer of the multiple different layers, wherein the maximum value of the temporal identifier is for the first layer and the second layer, and the temporal identifier of the second layer corresponding to the second picture satisfies the restrictions of the received inter-layer prediction restriction parameters.

Images