OLS for multi-view scalability

Abstract

A video coding mechanism is disclosed. [Solution] The mechanism includes receiving a bitstream containing an output layer set (OLS) and a video parameter set (VPS). The OLS contains one or more layers of coded pictures, and the VPS contains an OLS mode identification code (ols_mode_idc) for each OLS that specifies that all layers within each OLS are output layers. The output layers are determined based on the ols_mode_idc in the VPS. The coded pictures from the output layers are decoded to produce decoded pictures. The decoded pictures are transferred for display as part of a decoded video sequence.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This patent application claims priority to and incorporates by reference U.S. Provisional Patent Application No. 62 / 905,132, entitled "Signalling Of Output Layer Sets For Multiview Scalability", filed on September 24, 2019 by Ye - Kui Wang.

[0002] [Technical Field] The present disclosure generally relates to video coding, and more specifically relates to constructing output layer sets (OLS) in a multi - layer bitstream to support spatial and signal - to - noise (SNR) scalability for multi - view video.

Background Art

[0003] The amount of video data required to depict even relatively short videos can be quite large, which can cause difficulties when the data is streamed or communicated in other ways across a communication network having limited bandwidth capacity. Thus, video data is generally compressed before being communicated across today's telecommunications networks. Since memory resources can be limited, the size of the video can also be a problem when the video is stored on a storage device. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Due to limited network resources and the ever - increasing demand for higher video quality, improved compression and decompression techniques that improve the compression ratio without sacrificing much or any of the image quality are desired. [Overview of the Initiative]

[0004] In one embodiment, the disclosure includes a method implemented by a decoder, the method comprising: receiving a bitstream including an output layer set (OLS) and a video parameter set (VPS) by the decoder's receiver, wherein the OLS includes one or more layers of coded pictures, and the VPS includes an OLS mode identification code (ols_mode_idc) for each OLS that specifies that all layers within each OLS are output layers; determining the output layers based on the ols_mode_idc in the VPS by the decoder's processor; and decoding the coded pictures from the output layers by the decoder's processor and generating the decoded pictures.

[0005] Some video coding systems are configured to decode and output only the highest coding layer, as indicated by the layer ID, along with one or more designated lower layers. This can raise scalability issues because the decoder may not want to decode the highest layer. Specifically, the decoder generally requests the highest layer it can support, but it is generally impossible for the decoder to decode a layer higher than the requested layer. As a concrete example, the decoder may want to receive and decode the third layer out of a total of 15 coding layers. Since layers 4-15 are not required to decode the third layer, the third layer can be sent to the decoder without such layers. However, since the highest layer (layer 15) does not exist, and the video system is always instructed to decode and display the highest layer, the decoder may be unable to properly decode and display the third layer. This leads to errors when video scalability is attempted in such systems. This can be a significant problem because requiring the decoder to always support the highest layer results in a system that cannot scale to intermediate layers based on different hardware and network requirements. This issue becomes more complex when multiview is used. In multiview, more than one layer is output for display. For example, a user might use a headset, and different layers might be displayed to each eye to create the impression of three-dimensional (3D) video. Systems that do not support scalability also cannot support multiview scalability.

[0006] This example includes a mechanism for supporting multiview scalability. Layers are contained within an OLS. An encoder can transmit an OLS containing layers to scale for specific characteristics such as size or SNR. Furthermore, the encoder may transmit an ols_mode_idc syntax element, for example in a VPS. The ols_mode_idc syntax element can be set to 1 to indicate the use of multiview scalability. For example, ols_mode_idc could indicate that the total number of OLSs is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers between 0 and i (inclusive), and that for each OLS, all layers can be considered output layers. This supports scalability because the decoder can receive and decode all layers within a given OLS. Since all layers are output layers, the decoder can select and render the desired output layer. Thus, the total number of encoded layers may not affect the decoding process, and errors can be avoided while still providing scalable multiview video. Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and therefore reduce processor, memory, and / or network resource utilization in both the encoder and decoder. In certain embodiments, using ols_mode_idc provides bit saving in encoded bitstreams containing multiple OLSs where a lot of data is shared, and therefore provides savings in streaming servers and bandwidth savings for transmitting such bitstreams. For example, the advantage of setting ols_mode_idc to 1 is to support use cases such as multi-view applications where two or more views, each represented by one layer, are output and displayed simultaneously.

[0007] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS.

[0008] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc includes a layer having a layer index of 0 or greater and i or less for the i-th OLS.

[0009] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc is equal to 1.

[0010] Optionally, in any of the above embodiments, another implementation of that embodiment provides that the VPS includes VPS_max_layers_minus1, which specifies the number of layers designated by the VPS, which is the maximum allowed number of layers in each coded video sequence (CVS) that references the VPS.

[0011] Optionally, in any of the above embodiments, another implementation of that embodiment provides that the total number of OLS (TotalNumOlss) is equal to vps_max_layers_minus1 plus 1 when ols_mode_idc is equal to 0, or when ols_mode_idc is equal to 1.

[0012] Optionally, in any of the above embodiments, another implementation of that embodiment is such that the number of layers in the i-th OLS (NumLayersInOls[i]) and the network abstraction layer (NAL) unit header layer identifier (nuh_layer_id) value (LayerIdInOLS[i][j]) of the j-th layer in the i-th OLS are as follows, i.e., NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] This is derived as follows, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

[0013] In one embodiment, the disclosure includes a method implemented by an encoder, the method comprising: the step of encoding a bitstream including one or more OLSs, each containing one or more layers of a coded picture, using the encoder's processor; the step of encoding a VPS into a bitstream, the VPS including an ols_mode_idc specifying that for each OLS, all layers within each OLS are output layers; and the step of storing the bitstream in a memory coupled to the processor for communication to a decoder.

[0014] Some video coding systems are configured to decode and output only the highest coding layer, as indicated by the layer ID, along with one or more designated lower layers. This can raise scalability issues because the decoder may not want to decode the highest layer. Specifically, the decoder generally requests the highest layer it can support, but it is generally impossible for the decoder to decode a layer higher than the requested layer. As a concrete example, the decoder may want to receive and decode the third layer out of a total of 15 coding layers. Since layers 4-15 are not required to decode the third layer, the third layer can be sent to the decoder without such layers. However, since the highest layer (layer 15) does not exist, and the video system is always instructed to decode and display the highest layer, the decoder may be unable to properly decode and display the third layer. This leads to errors when video scalability is attempted in such systems. This can be a significant problem because requiring the decoder to always support the highest layer results in a system that cannot scale to intermediate layers based on different hardware and network requirements. This issue becomes more complex when multiview is used. In multiview, more than one layer is output for display. For example, a user might use a headset, and different layers might be displayed to each eye to create the impression of three-dimensional (3D) video. Systems that do not support scalability also cannot support multiview scalability.

[0015] This example includes a mechanism for supporting multiview scalability. Layers are contained within an OLS. An encoder can transmit an OLS containing layers to scale for specific characteristics such as size or SNR. Furthermore, the encoder may transmit an ols_mode_idc syntax element, for example in a VPS. The ols_mode_idc syntax element can be set to 1 to indicate the use of multiview scalability. For example, ols_mode_idc could indicate that the total number of OLSs is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers between 0 and i (inclusive), and that for each OLS, all layers can be considered output layers. This supports scalability because the decoder can receive and decode all layers within a given OLS. Since all layers are output layers, the decoder can select and render the desired output layer. Thus, the total number of encoded layers may not affect the decoding process, and errors can be avoided while still providing scalable multiview video. Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and therefore reduce processor, memory, and / or network resource utilization in both the encoder and decoder. In certain embodiments, using ols_mode_idc provides bit saving in encoded bitstreams containing multiple OLSs where a lot of data is shared, and therefore provides savings in streaming servers and bandwidth savings for transmitting such bitstreams. For example, the advantage of setting ols_mode_idc to 1 is to support use cases such as multi-view applications where two or more views, each represented by one layer, are output and displayed simultaneously.

[0016] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS.

[0017] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc includes a layer having a layer index of 0 or greater and i or less for the i-th OLS.

[0018] Optionally, in any of the above embodiments, another implementation of that embodiment provides that ols_mode_idc is equal to 1.

[0019] Optionally, in any of the above embodiments, another implementation of that embodiment provides that the VPS includes vps_max_layers_minus1 which specifies the number of layers designated by the VPS, which is the maximum allowed number of layers in each CVS that references the VPS.

[0020] Optionally, in any of the above embodiments, another implementation of that embodiment provides that TotalNumOlss is equal to vps_max_layers_minus1 plus 1 when ols_mode_idc is equal to 0, or when ols_mode_idc is equal to 1.

[0021] Optionally, in any of the above embodiments, another implementation of that embodiment is such that NumLayersInOls[i] and LayerIdInOLS[i][j] are as follows, i.e., NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0] = vps_layer_id[i] } else if (ols_mode_idc == 0 || ols_mode_idc == 1) { NumLayersInOls[i] = i + 1 for (j = 0; j < NumLayersInOls[i]; j++) LayerIdInOls[i][j] = vps_layer_id[j] is derived as such, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLSs specified by the VPS, and each_layer_is_an_ols_flag is a flag indicating that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

[0022] In one embodiment, the present disclosure includes a video coding device including a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, and the processor, receiver, memory, and transmitter are configured to execute any of the methods of the above aspects.

[0023] In one embodiment, the present disclosure includes a non - transient computer - readable medium including a computer program product used by a video coding device, and the computer program product includes computer - executable instructions stored in the non - transient computer - readable medium that, when executed by a processor, cause the video coding device to execute any of the methods of the above aspects.

[0024] In one embodiment, the present disclosure provides a receiving means for receiving a bitstream including an OLS and a VPS, wherein the OLS includes one or more layers of coded pictures, and the VPS includes an ols_mode_idc specifying that for each OLS all layers within each OLS are output layers; and a decoder including a determining means for determining the output layers based on the ols_mode_idc in the VPS; a decoding means for decoding the coded pictures from the output layers and generating a decoded picture; and a transferring means for transferring the decoded picture for display as part of a decoded video sequence.

[0025] Some video coding systems are configured to decode and output only the highest coding layer, as indicated by the layer ID, along with one or more designated lower layers. This can raise scalability issues because the decoder may not want to decode the highest layer. Specifically, the decoder generally requests the highest layer it can support, but it is generally impossible for the decoder to decode a layer higher than the requested layer. As a concrete example, the decoder may want to receive and decode the third layer out of a total of 15 coding layers. Since layers 4-15 are not required to decode the third layer, the third layer can be sent to the decoder without such layers. However, since the highest layer (layer 15) does not exist, and the video system is always instructed to decode and display the highest layer, the decoder may be unable to properly decode and display the third layer. This leads to errors when video scalability is attempted in such systems. This can be a significant problem because requiring the decoder to always support the highest layer results in a system that cannot scale to intermediate layers based on different hardware and network requirements. This issue becomes more complex when multiview is used. In multiview, more than one layer is output for display. For example, a user might use a headset, and different layers might be displayed to each eye to create the impression of three-dimensional (3D) video. Systems that do not support scalability also cannot support multiview scalability.

[0026] This example includes a mechanism for supporting multiview scalability. Layers are contained within an OLS. An encoder can transmit an OLS containing layers to scale for specific characteristics such as size or SNR. Furthermore, the encoder may transmit an ols_mode_idc syntax element, for example in a VPS. The ols_mode_idc syntax element can be set to 1 to indicate the use of multiview scalability. For example, ols_mode_idc could indicate that the total number of OLSs is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers between 0 and i (inclusive), and that for each OLS, all layers can be considered output layers. This supports scalability because the decoder can receive and decode all layers within a given OLS. Since all layers are output layers, the decoder can select and render the desired output layer. Thus, the total number of encoded layers may not affect the decoding process, and errors can be avoided while still providing scalable multiview video. Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and therefore reduce processor, memory, and / or network resource utilization in both the encoder and decoder. In certain embodiments, using ols_mode_idc provides bit saving in encoded bitstreams containing multiple OLSs where a lot of data is shared, and therefore provides savings in streaming servers and bandwidth savings for transmitting such bitstreams. For example, the advantage of setting ols_mode_idc to 1 is to support use cases such as multi-view applications where two or more views, each represented by one layer, are output and displayed simultaneously.

[0027] Optionally, in any of the above embodiments, another implementation of that embodiment provides that the decoder is further configured to perform any of the above embodiments.

[0028] In one embodiment, the disclosure provides an encoding means for encoding a bitstream comprising one or more OLSs, each comprising one or more layers of a coded picture, and encoding a VPS into a bitstream, wherein the VPS includes an encoding means comprising an ols_mode_idc specifying that for each OLS, all layers within each OLS are output layers, and an encoder comprising a storage means for storing the bitstream for communication to a decoder.

[0029] Some video coding systems are configured to decode and output only the highest coding layer, as indicated by the layer ID, along with one or more designated lower layers. This can raise scalability issues because the decoder may not want to decode the highest layer. Specifically, the decoder generally requests the highest layer it can support, but it is generally impossible for the decoder to decode a layer higher than the requested layer. As a concrete example, the decoder may want to receive and decode the third layer out of a total of 15 coding layers. Since layers 4-15 are not required to decode the third layer, the third layer can be sent to the decoder without such layers. However, since the highest layer (layer 15) does not exist, and the video system is always instructed to decode and display the highest layer, the decoder may be unable to properly decode and display the third layer. This leads to errors when video scalability is attempted in such systems. This can be a significant problem because requiring the decoder to always support the highest layer results in a system that cannot scale to intermediate layers based on different hardware and network requirements. This issue becomes more complex when multiview is used. In multiview, more than one layer is output for display. For example, a user might use a headset, and different layers might be displayed to each eye to create the impression of three-dimensional (3D) video. Systems that do not support scalability also cannot support multiview scalability.

[0030] This example includes a mechanism for supporting multiview scalability. Layers are contained within an OLS. An encoder can transmit an OLS containing layers to scale for specific characteristics such as size or SNR. Furthermore, the encoder may transmit an ols_mode_idc syntax element, for example in a VPS. The ols_mode_idc syntax element can be set to 1 to indicate the use of multiview scalability. For example, ols_mode_idc could indicate that the total number of OLSs is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers between 0 and i (inclusive), and that for each OLS, all layers can be considered output layers. This supports scalability because the decoder can receive and decode all layers within a given OLS. Since all layers are output layers, the decoder can select and render the desired output layer. Thus, the total number of encoded layers may not affect the decoding process, and errors can be avoided while still providing scalable multiview video. Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and therefore reduce processor, memory, and / or network resource utilization in both the encoder and decoder. In certain embodiments, using ols_mode_idc provides bit saving in encoded bitstreams containing multiple OLSs where a lot of data is shared, and therefore provides savings in streaming servers and bandwidth savings for transmitting such bitstreams. For example, the advantage of setting ols_mode_idc to 1 is to support use cases such as multi-view applications where two or more views, each represented by one layer, are output and displayed simultaneously.

[0031] Optionally, in any of the above embodiments, another implementation of that embodiment provides that the encoder is further configured to perform any of the methods of the above embodiments.

[0032] For the purpose of clarity, any one of the embodiments described above may be combined with one or more of the other embodiments described above to create a new embodiment within the scope of this disclosure.

[0033] These and other features will be better understood from the following detailed description made in relation to the attached drawings and claims. [Brief explanation of the drawing]

[0034] For a further and complete understanding of this disclosure, references are made herein to the attached drawings and the following brief description relating to the detailed description, where similar reference numerals represent similar parts. [Figure 1] This is a flowchart illustrating an exemplary method for coding a video signal. [Figure 2] This is a schematic diagram illustrating an exemplary coding and decoding (codec) system for video coding. [Figure 3] This is a schematic diagram illustrating an example video encoder. [Figure 4] This is a schematic diagram illustrating an exemplary video decoder. [Figure 5] This is a schematic diagram showing an exemplary multilayer video sequence configured for interlayer prediction. [Figure 6] This is a schematic diagram showing an exemplary video sequence with an OLS configured for multi-view scalability. [Figure 7] This is a schematic diagram showing an exemplary bitstream, including an OLS configured for multi-view scalability. [Figure 8] This is a schematic diagram of an exemplary video coding device. [Figure 9] This is a flowchart illustrating an exemplary method for encoding a video sequence with an OLS configured for multi-view scalability. [Figure 10]This is a flowchart illustrating an exemplary method for decoding a video sequence containing an OLS configured for multi-view scalability. [Figure 11] This is a schematic diagram of an exemplary system for coding video sequences with an OLS configured for multi-view scalability. [Modes for carrying out the invention]

[0035] Firstly, exemplary implementations of one or more embodiments are provided below, but it should be understood that the systems and / or methods of the disclosure can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary implementations, drawings and techniques shown below, including the exemplary designs and implementations herein illustrated and described herein, and may be modified within the scope of the appended claims, along with the scope of their complete equivalents.

[0036] The following terms are defined as defined herein, unless used in the opposite context. Specifically, the following definitions are intended to provide further clarification to this disclosure. However, terms may be described differently in different contexts. Therefore, the following definitions should be considered supplementary and should not be considered to limit any other definitions of such terms provided herein.

[0037] A bitstream is a sequence of bits containing compressed video data for transmission between an encoder and a decoder. An encoder is a device configured to use an encoding process to compress video data into a bitstream. A decoder is a device configured to use a decoding process to reconstruct video data from the bitstream for display. A picture is an array of luma samples and / or chroma samples that make up a frame or its fields. A picture being encoded or decoded may now be referred to as a picture for clarity in the discussion.

[0038] A network abstraction layer (NAL) unit is a syntax structure containing data in the form of a raw byte sequence payload (RBSP), an indication of the data type, and interspersed with anti-emulation bytes as desired. A video coding layer (VCL) NAL unit is a NAL unit coded to contain video data, such as coded slices of a picture. A non-VCL NAL unit is a NAL unit containing non-video data, such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations. A layer is a set of VCL NAL units and associated non-VCL NAL units that share a specified characteristic (e.g., common resolution, frame rate, image size, etc.). VCL NAL units in a layer may share a specific value for the NAL unit header layer identifier (nuh_layer_id). A coded picture is a coded representation of a picture that includes a VCL NAL unit with a specific value for the NAL unit header layer identifier (nuh_layer_id) within an access unit (AU), and also includes all of the picture's coding tree units (CTUs). A decoded picture is a picture produced by applying the decoding process to a coded picture. A coded video sequence (CVS) is a sequence of AUs in decoding order that includes one or more coded video sequence start (CVSS) AUs and optionally one or more AUs that are not CVSS AUs.A CVSS AU is an AU that contains prediction units (PUs) for each layer specified by the video parameter set (VPS), and the coded picture within each PU is the start picture of the CVS / coded layer video sequence (CLVS).

[0039] An output layer set (OLS) is a set of layers in which one or more layers are designated as output layers. An output layer is a layer designated for output (e.g., to a display). The highest layer is the layer in the OLS that has the highest layer identifier (ID) among all layers in the OLS. In some exemplary OLS modes, the highest layer may always be the output layer. In other modes, the designated layer and / or all layers are output layers. A video parameter set (VPS) is a data unit containing parameters related to the entire video. Cross-layer prediction is a mechanism for coding the current picture in the current layer by referencing a reference picture in the reference layer, where the current picture and reference picture are in the same AU, and the reference layer has a lower nuh_layer_id than the current layer.

[0040] The OLS mode identification code (ols_mode_idc) is a syntax element that indicates information about the number of OLSs, the layers of the OLS, and the output layers within the OLS. The VPS maximum layers minus 1 (vps_max_layers_minus1) is a syntax element that conveys the number of layers specified by the VPS, and therefore the maximum number of layers allowed in the corresponding CVS. The flag that each layer is an OLS (each_layer_is_an_ols_flag) is a syntax element that conveys whether each OLS in the bitstream contains a single layer or not. The total number of OLS (TotalNumOLss) is a variable that specifies the total number of OLSs specified by the VPS. The number of layers in the i-th OLS (NumLayersInOLS[i]) is a variable that specifies the number of layers in a particular OLS indicated by the OLS index value of i. The Layer ID within the OLS (LayerIdInOLS[i][j]) is a variable that specifies the nuh_layer_id value of the j-th layer in the i-th OLS, indicated by the layer index j and OLS index i. vps_layer_id[i] is a syntax element that indicates the layer ID of the i-th layer.

[0041] The following abbreviations are used here: Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Coded Video Sequence (CVS), Joint Video Experts Team (JVET), Motion Constrained Tile Set (MCTS), Maximum Transfer Unit (MTU), Network Abstraction Layer (NAL), Output Layer Set (OLS), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), Video Parameter Set (VPS), and Versatile Video Coding (VVC).

[0042] Many video compression techniques can be used to reduce the size of video files with minimal data loss. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or eliminate data redundancy in a video sequence. In block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be called tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction for reference samples in adjacent blocks within the same picture. Video blocks in an intercoded (P) or bidirectional (B) slice of a picture may be coded using spatial prediction for reference samples in adjacent blocks within the same picture, or temporal prediction for reference samples in other reference pictures. A picture may also be called a frame and / or image, and a reference picture may be called a reference frame and / or reference image. Spatial or temporal predictions produce predicted blocks representing image blocks. Residual data represents the pixel difference between the original image blocks and the predicted blocks. Thus, an intercoded block is encoded according to a motion vector pointing to a block of reference samples forming the predicted block, and residual data indicating the difference between the coded block and the predicted block. An intracoded block is encoded according to the intracoded mode and residual data. For further compression, the residual data may be transformed from the pixel domain to the transformation domain. These produce residual transformation coefficients, which may be quantized. The quantized transformation coefficients may first be arranged in a two-dimensional array. The quantized transformation coefficients may be scanned to generate a one-dimensional vector of transformation coefficients.To achieve even greater compression, entropy coding may be applied. Such video compression techniques are described in more detail below.

[0043] To ensure that encoded video can be accurately decoded, video is encoded and decoded according to the corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC).The ITU-T and ISO / IEC Joint Video Experts Team (JVET) has begun developing a video coding standard called Versatile Video Coding (VVC). VVC is included in the Working Class (WD), which includes JVET-O2001-v14.

[0044] To support scalability, picture layers can be used. For example, video can be coded into multiple layers. Layers may be coded without referencing other layers. Such layers are called simulcast layers. Thus, simulcast layers can be decoded without referencing other layers. As another example, layers can be coded using inter-layer prediction. This allows the current layer to be coded by including only the difference between the current layer and the reference layer. For example, the current layer and the reference layer may contain the same video sequence coded by varying characteristics such as signal-to-noise ratio (SNR), picture size, frame rate, etc.

[0045] Some video coding systems are configured to decode and output only the highest coding layer, as indicated by a layer identifier (ID), along with one or more specified lower layers. This can raise scalability issues because the decoder may not want to decode the highest layer. Specifically, the decoder generally requests the highest layer it can support, but it is generally impossible for the decoder to decode a layer higher than the requested layer. As a concrete example, the decoder may want to receive and decode the third layer out of a total of 15 coding layers. Since layers 4-15 are not required to decode the third layer, the third layer can be sent to the decoder without such layers. However, since the highest layer (layer 15) does not exist and the video system is always instructed to decode and display the highest layer, the decoder may be unable to properly decode and display the third layer. This leads to errors when video scalability is attempted in such systems. This can be a significant problem because requiring the decoder to always support the highest layer results in a system that cannot scale to intermediate layers based on different hardware and network requirements. This issue becomes more complex when multiview is used. In multiview, more than one layer is output for display. For example, a user might use a headset, and different layers might be displayed to each eye to create the impression of three-dimensional (3D) video. Systems that do not support scalability also cannot support multiview scalability.

[0046] Disclosed herein are mechanisms for supporting multi-view scalability. Layers are contained in an output layer set (OLS). An encoder can transmit an OLS containing layers to scale for a specific characteristic such as size or SNR. Spatial scalability allows video sequences to be coded into layers such that each OLS contains enough data to decode the video sequence to a corresponding output screen size. Therefore, spatial scalability may include a set of layers for decoding video for a smartphone screen, a set of layers for decoding video for a large television screen, and a set of layers for intermediate screen sizes. SNR scalability allows video sequences to be coded into layers such that each OLS contains enough data to decode the video sequence at a different SNR. Therefore, SNR scalability may include a set of layers that can be decoded for low-quality video, high-quality video, and various intermediate video qualities based on network conditions. Furthermore, the encoder may transmit an OLS mode identification code (ols_mode_idc) syntax element, for example, in the video parameter set (VPS). The ols_mode_idc syntax element can be set to 1 to indicate the use of multiview scalability. For example, ols_mode_idc could indicate that the total number of OLSs is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers from 0 to i (inclusive), and that for each OLS, all layers can be considered output layers. This supports scalability because the decoder can receive and decode all layers within a given OLS. Since all layers are output layers, the decoder can select and render the desired output layer. In this way, the total number of encoded layers may not affect the decoding process, and errors can be avoided while still providing scalable multiview video.Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and thus reduce the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0047] Figure 1 is a flowchart of an exemplary operation method 100 for coding a video signal. Specifically, the video signal is encoded by an encoder. The encoding process compresses the video signal by using various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted to the user while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally reflects the encoding process, allowing the decoder to consistently reconstruct the video signal.

[0048] In step 101, a video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. In another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. The video file may contain both audio and video components. The video component contains a series of image frames that give a visual impression of motion when viewed in sequence. Each frame contains pixels, which are expressed here in terms of brightness, called the lumens component (or lumens sample), and color, called the chromens component (or color sample). In some examples, the frames may also include depth values ​​to support three-dimensional display.

[0049] In step 103, the video is partitioned into blocks. Partitioning involves subdividing the pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a predetermined size (e.g., 64 pixels × 64 pixels). A CTU contains both luminous and chroma samples. The coding tree may be used to divide the CTUs into blocks and then recursively subdivide the blocks until a configuration supporting further coding is achieved. For example, the luminous component of a frame may be subdivided until the individual blocks contain relatively uniform brightness values. Furthermore, the chroma component of a frame may be subdivided until the individual blocks contain relatively uniform color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

[0050] In step 105, various compression mechanisms are used to compress the image blocks partitioned in step 103. For example, interpretation and / or intrapretation may be used. Interpretation is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Therefore, blocks that draw objects in a reference frame do not need to be described repeatedly in adjacent frames. Specifically, objects such as tables may remain in the same position across multiple frames. Therefore, a table is described once, and adjacent frames can refer back to the reference frame. Pattern matching mechanisms may be used to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to the movement of the object or the movement of the camera. As a particular example, a video may show a car moving between screens across multiple frames. Motion vectors can be used to describe such motion. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in a given frame to the coordinates of that object in a reference frame. Therefore, interpretation can encode image blocks in the current frame as a set of motion vectors pointing to offsets from the corresponding blocks in the reference frame.

[0051] Intra-prediction encodes blocks within a common frame. It leverages the fact that luma and chroma components tend to cluster within a frame. For example, some green patches in a tree tend to be adjacent to similar green patches. Intra-prediction uses multiple directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional modes indicate that a current block is similar / identical to samples of adjacent blocks in the corresponding direction. Planar mode indicates that a series of blocks along a row / column (e.g., a plane) can be interpolated based on adjacent blocks at the edges of the row. Planar mode effectively shows smooth brightness / color transitions across rows / columns by using a relatively constant slope when changing values. DC mode is used for boundary smoothing and indicates that a block is similar / identical to the mean value related to samples of all adjacent blocks in the angular direction of the directional prediction mode. Thus, intra-predicted blocks can represent image blocks as various relational prediction mode values ​​instead of their actual values. Furthermore, the interpretation block can represent the image block as a motion vector value instead of its actual value. In either case, the prediction block may not accurately represent the image block in some cases. Any difference is stored in the residual block. A transformation may be applied to the residual block to further compress the file.

[0052] In step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above may result in the creation of block-noised images in the decoder. Furthermore, the block-based prediction scheme may encode blocks and then reconstruct the encoded blocks for later use as reference blocks. The in-loop filtering scheme repeatedly applies noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to blocks / frames. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. In addition, these filters mitigate artifacts in the reconstructed reference block so that artifacts are less likely to create further artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0053] Once the video signal has been partitioned, compressed, and filtered, in step 109 the resulting data is encoded into a bitstream. The bitstream includes the above data and any signaling data desired to support proper video signal reconstruction in the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder on request. The bitstream may also be broadcast and / or multicast to multiple decoders. The creation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 may occur sequentially and / or simultaneously over many frames and blocks. The order shown in Figure 1 is presented for clarity and ease of explanation and is not intended to limit the video coding process to a specific order.

[0054] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. In step 111, the decoder uses the syntax data from the bitstream to determine the partitioning for the frame. The partitioning should match the result of the block partitioning in step 103. The entropy coding / decoding used in step 111 is described here. During the compression process, the encoder makes many choices, such as selecting a block partitioning scheme from several possible options based on the spatial location of values ​​in the input image. Communicating a strict choice may involve using a number of bins. When used here, bins are binary values ​​treated as variables (e.g., bit values ​​that can change depending on the context). Entropy coding allows the encoder to discard any option that is obviously not feasible in a particular case, leaving only the acceptable options. A codeword is then assigned to each acceptable option. The length of the codeword is based on the number of acceptable options (e.g., one bin for two options, two bins for three or four options, etc.). The encoder then encodes the codeword for the selected options. This method reduces the size of the codeword so that it is only the size desired to uniquely indicate a selection from a small subset of acceptable options, rather than a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of acceptable options, in a similar manner to the encoder. By determining the set of acceptable options, the decoder can read the codeword and determine the selection made by the encoder.

[0055] In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate residual blocks. The decoder then uses the residual blocks and corresponding prediction blocks to reconstruct the image blocks according to the partitioning. The prediction blocks may include both intra-prediction blocks and inter-prediction blocks generated by the encoder in step 105. The reconstructed image blocks are then positioned in the frames of the reconstructed video signal according to the partitioning data determined in step 111. The syntax for step 113 may also be transmitted in the bitstream via entropy coding as described above.

[0056] In step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to that in step 107 of the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters may be applied to the frames to remove blocking artifacts. Once the frames are filtered, in step 117, the video signal can be output to a display for viewing by the end user.

[0057] Figure 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functions to support the implementation of operation method 100. The codec system 200 is generalized to show components used in both the encoder and the decoder. The codec system 200 receives and partitions a video signal, as described in relation to steps 101 and 103 of operation method 100, which produces a partitioned video signal 201. Then, when acting as an encoder, the codec system 200 compresses the partitioned video signal 201 into a coded bitstream, as described in relation to steps 105, 107 and 109 of method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream, as described in relation to steps 111, 113, 115 and 117 of operation method 100. The codec system 200 includes an overall coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. Such components are combined as shown in the figure. In Figure 2, black lines indicate the movement of data being encoded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All components of the codec system 200 may be present in the encoder. The decoder may include a subset of the components of the codec system 200.For example, the decoder may include an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. These components are described below.

[0058] A partitioned video signal 201 is a captured video sequence partitioned into blocks of pixels by a coding tree. The coding tree uses various partitioning modes to subdivide blocks of pixels into smaller blocks of pixels. These blocks can then be further subdivided into even smaller blocks. Blocks may also be called nodes on the coding tree. Larger parent nodes are subdivided into smaller child nodes. The number of times a node is subdivided is called the node / coding tree depth. Divided blocks can be contained in a coding unit (CU) in some cases. For example, a CU may be part of a CTU that includes a lumen block, a red difference chroma (Cr) block, and a blue difference chroma (Cb) block, along with the corresponding syntax instructions for that CU. The partitioning mode may include binary trees (BT), triple trees (TT), and quad trees (QT) of various shapes, depending on the partitioning mode used, which are used to partition a node into two, three, or four child nodes, respectively. The partitioned video signal 201 is then transferred for compression to the overall coder control component 201, the transformation scaling and quantization component 213, the intrapicture estimation component 215, the filter control analysis component 227, and the motion estimation component 221.

[0059] The overall coder control component 211 is configured to make decisions related to coding the video sequence images into a bitstream, according to application constraints. For example, the overall coder control component 211 manages the optimization of bitrate / bitstream size for reconstruction quality. Such decisions may be based on memory space / bandwidth availability and image resolution requirements. The overall coder control component 211 also manages buffer utilization in relation to the transmission rate to mitigate buffer underrun and overrun problems. To manage these problems, the overall coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the overall coder control component 211 may dynamically increase the complexity of compression to increase resolution and bandwidth usage, or decrease the complexity of compression to decrease resolution and bandwidth usage. Thus, the overall coder control component 211 controls other components of the codec system 200 to balance video signal reconstruction quality with bitrate concerns. The overall coder control component 211 creates control data that controls the operation of other components. The control data is also transferred to the header format and CABAC component 231 so that it is encoded into a bitstream to convey parameters for decoding at the decoder.

[0060] The partitioned video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for interpretation. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform interpretation coding of the received video blocks for one or more blocks in one or more reference frames in order to provide time prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0061] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors that estimate the motion for a video block. The motion vectors may, for example, represent the displacement of the object to be coded relative to a prediction block. A prediction block is a block that has been found to closely match the block to be coded with respect to the pixel difference. A prediction block may also be called a reference block. Such a pixel difference may be determined by the sum of absolute difference (SAD), the sum of square difference (SSD), or other difference metrics. HEVC uses several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be split into CTBs, which can then be split into CBs to be included in a CU. A CU can be coded as a prediction unit (PU) containing prediction data and / or a transform unit (TU) containing transformed residual data for the CU. The motion estimation component 221 generates motion vectors, PUs, and TUs by using rate distortion analysis as part of the rate distortion optimization process. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc., for the current block / frame, or it may select the reference blocks, motion vectors, etc., that have the best rate distortion characteristics. The best rate distortion characteristics balance both the quality of video reconstruction (e.g., the amount of data loss due to compression) and coding efficiency (e.g., the size of the final encoding).

[0062] In some examples, the codec system 200 may calculate the values ​​of the sub-integer pixel positions of the reference picture stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate the values ​​of the reference picture at a quarter-pixel position, an eighth-pixel position, or other fractional pixel position. Thus, the motion estimation component 221 may perform motion searches on full-pixel and fractional-pixel positions and output motion vectors with fractional-pixel precision. The motion estimation component 221 calculates the motion vector for the PU of a video block in the intercoded slice by comparing the position of the PU with the position of a predicted block in the reference picture. The motion estimation component 221 outputs the calculated motion vector to the header format and CABAC component 231 for encoding as motion data, and also to the motion compensation component 219 as motion.

[0063] Motion compensation performed by the motion compensation component 219 may include fetching or generating a predicted block based on a motion vector determined by the motion estimation component 221. Similarly, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Upon receiving a motion vector for the PU of the current video block, the motion compensation component 219 may locate the predicted block pointed to by the motion vector. A residual video block is then formed by subtracting the pixel values ​​of the predicted block from the coded pixel values ​​of the current video block, thereby forming the pixel difference value. Generally, the motion estimation component 221 performs motion estimation on the lumens component, and the motion compensation component 219 uses a motion vector calculated based on the lumens component for both the chromens and lumens components. The predicted and residual blocks are then transferred to the transformation scaling and quantization component 213.

[0064] The partitioned video signal 201 is also sent to the intra-picture estimation component 215 and the intra-picture prediction component 217. Similar to the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated, but are illustrated separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block for the block in the current frame, instead of inter-predicting performed by the motion estimation component 221 and the motion compensation component 219 between frames, as described above. In particular, the intra-picture estimation component 215 determines the intra-prediction mode to be used to encode the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode from several tested intra-prediction modes for encoding the current block. The selected intra-prediction mode is then forwarded to the header format and CABAC component 231 for encoding.

[0065] For example, the intra-picture estimation component 215 calculates rate distortion values ​​for various tested intra-prediction modes using rate distortion analysis and selects the intra-prediction mode with the best rate distortion characteristics among the tested modes. Rate distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block coded to produce that coded block, and the bit rate (e.g., number of bits) used to produce that coded block. The intra-picture estimation component 215 calculates a ratio from the distortion and rate for various coded blocks and determines which intra-prediction mode exhibits the best rate distortion value for that block. Furthermore, the intra-picture estimation component 215 may be configured to code depth blocks of a depth map using a depth modeling mode (DMM) based on rate distortion optimization (RDO).

[0066] When implemented in an encoder, the intra-picture prediction component 217 may generate residual blocks from prediction blocks based on a selected intra-prediction mode determined by the intra-picture estimation component 215, or, when implemented in a decoder, may read residual blocks from the bitstream. The residual blocks contain the difference in values ​​between the prediction blocks and the original blocks, represented as a matrix. The residual blocks are then transferred to the transformation scaling and quantization component 213. The intra-picture estimation component 215 and the intra-picture prediction component 217 may operate on both the luminous and chroma components.

[0067] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transform to the residual block to generate a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform may transform the residual information from a pixel value domain to a transform domain such as a frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying scaling coefficients to the residual information so that different frequency information is quantized at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bitrate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be changed by adjusting the quantization parameters. In some examples, the transformation scaling and quantization component 213 may then perform a scan of a matrix containing the quantized transformation coefficients. The quantized transformation coefficients are then transferred to the header format and CABAC component 231 for encoding within the bitstream.

[0068] The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transform, and / or quantization to reconstruct the residual block in the pixel domain for later use as a reference block that may become a predicted block for other current blocks, for example. The motion estimation component 221 and / or motion compensation component 219 may compute the reference block by adding the residual block back to the corresponding predicted block for use in motion estimation of subsequent blocks / frames. A filter is applied to the reconstructed reference block to mitigate artifacts created during scaling, quantization, and transform. Such artifacts could otherwise lead to inaccurate predictions (and create further artifacts) when subsequent blocks are predicted.

[0069] The filter-controlled analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, to reconstruct the original image block, the transformed residual block from the scaling and inverse transform component 229 may be combined with the corresponding predicted block from the intra-picture prediction component 217 and / or the motion compensation component 219. The filter may then be applied to the reconstructed image block. In some examples, the filter may be applied to the residual block instead. As with the other components in Figure 2, the filter-controlled analysis component 227 and the in-loop filter component 225 may be highly integrated and implemented together, but are illustrated separately for conceptual purposes. A filter applied to a reconstructed reference block is applied to a specific spatial region and includes several parameters to adjust how such a filter is applied. The filter-controlled analysis component 227 analyzes the reconstructed reference block to determine where such a filter should be applied and sets the corresponding parameters. Such data is transferred to the header format and CABAC component 231 as filter-controlled data for encoding. The in-loop filter component 225 applies such a filter based on the filter-controlled data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Depending on the example, such filters may be applied in the spatial / pixel domain (e.g., reconstructed pixel blocks) or in the frequency domain.

[0070] When operating as an encoder, filtered reconstructed image blocks, residual blocks, and / or predicted blocks are stored in the decoding picture buffer component 223 for later use in motion estimation as described above. When operating as a decoder, the decoding picture buffer component 223 stores the reconstructed and filtered blocks and transfers them to the display as part of the output video signal. The decoding picture buffer component 223 may be any memory device capable of storing predicted blocks, residual blocks, and / or reconstructed image blocks.

[0071] The header format and CABAC component 231 receive data from various components of the codec system 200 and encode such data into a coded bitstream for transmission to the decoder. Specifically, the header format and CABAC component 231 generate various headers to encode control data such as overall control data and filter control data. Furthermore, prediction data, including intra-prediction and motion data, and residual data in the form of quantized conversion coefficient data are all encoded into a bitstream. The final bitstream contains all the information desired by the decoder to reconstruct the original partitioned video signal 201. Such information may also include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, indications of the most likely intra-prediction mode, indications of partition information, etc. Such data may be encoded using entropy coding. For example, information may be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. Following entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0072] Figure 3 is a block diagram illustrating an exemplary video encoder 300. The video encoder 300 may be used to implement the encoding function of the codec system 200 and / or to implement steps 101, 103, 105, 107 and / or 109 of the operation method 100. The encoder 300 partitions the input video signal, resulting in a partitioned video signal 301 substantially similar to the partitioned video signal 201. The partitioned video signal 301 is then compressed by the components of the encoder 300 and encoded into a bitstream.

[0073] Specifically, the partitioned video signal 301 is transferred to the intra-picture prediction component 317 for intra-prediction. The intra-picture prediction component 317 may be substantially the same as the intra-picture estimation component 215 and the intra-picture prediction component 217. The partitioned video signal 301 is also transferred to the motion compensation component 321 for inter-prediction based on a reference block in the decoded picture buffer component 323. The motion compensation component 321 may be substantially the same as the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-picture prediction component 317 and the motion compensation component 321 are transferred to the transformation and quantization component 313 for transformation and quantization of the residual blocks. The transformation and quantization component 313 may be substantially the same as the transformation scaling and quantization component 213. The transformed and quantized residual blocks and the corresponding prediction blocks (along with the associated control data) are transferred to the entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially the same as the header format and the CABAC component 231.

[0074] The transformed and quantized residual blocks and / or corresponding prediction blocks are also transferred from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into reference blocks for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially the same as the scaling and inverse transform component 229. Depending on the example, the in-loop filter in the in-loop filter component 325 is also applied to the residual blocks and / or the reconstructed reference blocks. The in-loop filter component 325 may be substantially the same as the filter-controlled analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may contain multiple filters as described with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded picture buffer component 323 for use as reference blocks by the motion compensation component 321. The decoded picture buffer component 323 may be substantially the same as the decoded picture buffer component 223.

[0075] Figure 4 is a block diagram showing an exemplary video decoder 400. The video decoder 400 may be used to implement the decoding function of the codec system 200 and / or to implement steps 111, 113, 115 and / or 117 of the operation method 100. The decoder 400, for example, receives a bitstream from the encoder 300 and generates an output video signal reconstructed based on the bitstream for display to the end user.

[0076] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may use header information to provide context for interpreting further data encoded as codewords within the bitstream. The decoded information includes any desired information for decoding the video signal, such as overall control data, filter control data, partition information, motion data, prediction data, and quantized transformation coefficients from residual blocks. The quantized transformation coefficients are transferred to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0077] The reconstructed residual blocks and / or predicted blocks are transferred to the intra-picture prediction component 417 for reconstruction into image blocks based on intra-prediction operations. The intra-picture prediction component 417 may be similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 uses a prediction mode to position a reference block within the frame, applies the residual blocks to the result, and reconstructs the intra-predicted image block. The reconstructed intra-predicted image block and / or residual block and the corresponding intra-prediction data are transferred to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 and the in-loop filter component 425 may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or predicted block, and this information is stored in the decoded picture buffer component 423. The reconstructed image blocks from the decoded picture buffer component 423 are transferred to the motion compensation component 421 for interpretation. The motion compensation component 421 may be substantially the same as the motion estimation component 221 and / or motion compensation component 219. Specifically, the motion compensation component 421 uses motion vectors from a reference block to generate a prediction block, applies a residual block to the result, and reconstructs the image block. The resulting reconstructed block may also be transferred to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 continues to store further reconstructed image blocks, which can be reconstructed into frames via partition information. Such frames may also be arranged in a sequence, which is output to a display as a reconstructed output video signal.

[0078] Figure 5 is a schematic diagram showing an exemplary multilayer video sequence 500 configured for interlayer prediction 521. The multilayer video sequence 500 may be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. The multilayer video sequence 500 is included to illustrate exemplary application of layers in the coded video sequence. The multilayer video sequence 500 is any video sequence that uses multiple layers, such as layer N 531 and layer N+1 532.

[0079] In one example, a multilayer video sequence 500 may use interlayer prediction 521. Interlayer prediction 521 is applied between pictures 511, 512, 513, and 514 and pictures 515, 516, 517, and 518 in different layers. In the illustrated example, pictures 511, 512, 513, and 514 are part of layer N+1 532, and pictures 515, 516, 517, and 518 are part of layer N 531. Layers like layer N 531 and / or layer N+1 532 are groups of pictures all relating to similar values ​​of characteristics such as similar size, quality, resolution, signal-to-noise ratio, capability, etc. A layer may be formally defined as a set of VCL NAL units and associated non-VCL NAL units that share the same nuh_layer_id. A VCL NAL unit is a NAL unit coded to contain video data such as coded slices of a picture. A non-VCL NAL unit is a NAL unit that contains non-video data such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations.

[0080] In the illustrated example, layer N+1 532 is associated with a larger image size than layer N 531. Therefore, in this example, pictures 511, 512, 513, and 514 in layer N+1 532 have a larger picture size (e.g., larger height and width, and therefore more samples) than pictures 515, 516, 517, and 518 in layer N 531. However, such pictures can be separated between layer N+1 532 and layer N 531 based on other characteristics. Although only two layers, namely layer N+1 532 and layer N 531, are shown, a set of pictures can be separated into any number of layers based on their associated characteristics. Layers N+1 532 and N 531 may also be indicated by layer IDs. A layer ID is a data item associated with a picture, indicating that the picture is part of the layer in which it is shown. Therefore, each picture 511-518 may be associated with a corresponding layer ID to indicate which layer N+1 532 or layer N 531 contains the corresponding picture. For example, the layer ID may include a NAL unit header layer identifier (nuh_layer_id), which is a syntax element that specifies the identifier of the layer containing the NAL unit (e.g., the slices and / or parameters of the pictures within the layer). Layers associated with lower quality / bitstream size, such as layer N 531, are generally assigned lower layer IDs and are called lower layers. Furthermore, layers associated with higher quality / bitstream size, such as layer N+1 532, are generally assigned higher layer IDs and are called upper layers.

[0081] Pictures 511-518 in different layers 531-532 are configured to be displayed by choice. For example, if a smaller picture is desired, the decoder may decode and display picture 515 at the current display time, or if a larger picture is desired, the decoder may decode and display picture 511 at the current display time. Thus, pictures 511-514 in the upper layer N+1 532 contain substantially the same image data as the corresponding pictures 515-518 in the lower layer N 531 (despite the difference in picture size). Specifically, picture 511 contains substantially the same image data as picture 515, picture 512 contains substantially the same picture data as picture 516, and so on.

[0082] Pictures 511-518 can be coded by referencing other pictures 511-518 in the same layer N 531 or N+1 532. Coding a picture by referencing other pictures in the same layer results in interpretation 523. Interpretation 523 is indicated by a solid arrow. For example, picture 513 may be coded by using interpretation 523 with references to one or two of pictures 511, 512 and / or 514 in layer N+1 532, where one picture is referenced for one-way interpretation and / or two pictures are referenced for bidirectional interpretation. Furthermore, picture 517 may be coded by employing interpretation 523 with references to one or two of pictures 515, 516 and / or 518 in layer N 531, where one picture is referenced for one-way interpretation and / or two pictures are referenced for bidirectional interpretation. When performing interprediction 523, if a picture is used as a reference for other pictures in the same layer, the picture may be called a reference picture. For example, picture 512 may be a reference picture used to code picture 513 according to interprediction 523. Interprediction 523 can also be called intralayer prediction in a multilayer context. Thus, interprediction 523 is a mechanism for coding a sample in the current picture by referencing an indicated sample in a reference picture that is different from the current picture, and the reference picture and the current picture are in the same layer.

[0083] Pictures 511-518 can also be coded by referencing other pictures 511-518 in different layers. This process is known as inter-layer prediction 521 and is indicated by a dashed arrow. Inter-layer prediction 521 is a mechanism for coding a sample of the current picture by referencing an indicated sample in a reference picture, where the current picture and reference picture are in different layers and therefore have different layer IDs. For example, a picture in a lower layer N 531 can be used as a reference picture to code the corresponding picture in a higher layer N+1 532. As a concrete example, picture 511 can be coded by referencing picture 515 according to inter-layer prediction 521. In such a case, picture 515 is used as the inter-layer reference picture. The inter-layer reference picture is the reference picture used in inter-layer prediction 521. In most cases, inter-layer prediction 521 is constrained so that a current picture, such as picture 511, can only use inter-layer reference pictures, such as picture 515, which are contained in the same AU and are in a lower layer. An AU is a set of pictures associated with a specific output time in a video sequence, and therefore an AU can contain only one picture per layer. When multiple layers (e.g., more than two) are available, the inter-layer prediction 521 can encode / decode the current picture based on multiple inter-layer reference pictures at a lower level than the current picture.

[0084] The video encoder can use the multilayer video sequence 500 to encode pictures 511-518 via many different combinations and / or permutations of inter-prediction 523 and inter-layer prediction 521. For example, picture 515 may be coded according to intra-prediction. Then, pictures 516-518 can be coded according to inter-prediction 523 by using picture 515 as a reference picture. Furthermore, picture 511 may be coded according to inter-layer prediction 521 by using picture 515 as an inter-layer reference picture. Then, pictures 512-514 can be coded according to inter-prediction 523 by using picture 511 as a reference picture. Thus, a reference picture can function as both a single-layer reference picture and an inter-layer reference picture for different coding mechanisms. By coding the picture of the upper layer N+1 532 based on the picture of the lower layer N 531, the upper layer N+1 532 can avoid using intra-prediction, which has much lower coding efficiency than inter-prediction 523 and inter-layer prediction 521. Therefore, the inferior coding efficiency of intra-prediction can be limited to the smallest / lowest quality picture, and thus limited to coding the smallest amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures can be indicated in the reference picture list entries contained in the reference picture list structure.

[0085] To perform such operations, layers such as layer N 531 and layer N+1 532 may be included in OLS525. OLS525 is a set of layers in which one or more layers are designated as output layers. An output layer is a layer designated for output (e.g., to a display). For example, layer N 531 may be included only to support inter-layer prediction 521 and may never be output. In such a case, layer N+1 532 is decoded based on layer N 531 and output. In such a case, OLS525 includes layer N+1 532 as an output layer. OLS525 may include many layers in different combinations. For example, an output layer in OLS525 can be coded according to inter-layer prediction 521 based on one, two or many lower layers. Furthermore, OLS525 may include more than one output layer. Thus, OLS525 may include one or more output layers and any of the support layers necessary to reconstruct the output layer. A multi-layer video sequence 500 can be coded by using many different OLS525s, each using a different combination of layers.

[0086] As a concrete example, inter-layer prediction 521 may be used to support scalability. For example, video can be coded with a base layer such as layer N 531 and several enhancement layers such as layer N+1 532, layer N+2, layer N+3, etc., which are coded according to inter-layer prediction 521. The video sequence can be coded for several scalable characteristics such as signal-to-noise ration (SNR), frame rate, picture size, etc. Then, OLS 525 can be created for each acceptable characteristic. For example, OLS 525 for a first resolution may include only layer N 531, OLS 525 for a second resolution may include layer N 531 and layer N+1 532, and OLS for a third resolution may include layer N 531, layer N+1 532, layer N+2, etc. Thus, the OLS525 can transmit data in a way that allows the decoder to decode whichever version of the multilayer video sequence 500 is desired, based on network conditions, hardware constraints, etc.

[0087] Figure 6 is a schematic diagram showing an exemplary video sequence 600 having an OLS configured for multi-view scalability. Video sequence 600 is a concrete example of a multi-layer video sequence 500. Thus, video sequence 600 can be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Video sequence 600 is useful for scalability.

[0088] An exemplary video sequence 600 includes OLS620, OLS621, and OLS622, which may be substantially similar to OLS525. Although three OLS are shown, any number of OLS may be used. Each OLS620, 621, and 622 is referenced by an OLS index and contains one or more layers. Specifically, OLS620, 621, and 622 each contain layers 630, 630, and 631, and layers 630, 631, and 632, respectively. Layers 630, 631, and 632 may be substantially similar to layers N 531 and N+1 532. Layers 630, 631, and 632 are referenced by layer indices. Video sequence 600 contains the same number of layers as the number of OLS. Specifically, OLS620, which has the lowest OLS index, contains layer 630, which has the lowest layer index. Each other OLS contains all the layers of the previous OLS with a lower OLS index plus 1. For example, OLS621 has a higher OLS index than OLS620 and contains layers 630 and 631, which are all the layers of OLS620 plus 1. Similarly, OLS622 has a higher OLS index than OLS621 and contains layers 630, 631 and 632, which are all the layers of OLS621 plus 1. This pattern may continue until a layer with the highest layer index and an OLS with the highest OLS index are reached.

[0089] Furthermore, layer 630 is the base layer. All other layers 631 and 632 are enhancement layers coded according to inter-layer predictions based on all layers with lower layer indices. Specifically, layer 630 is the base layer and is not coded according to inter-layer predictions. Layer 631 is an enhancement layer coded according to inter-layer predictions based on layer 630. Furthermore, layer 632 is an enhancement layer coded according to inter-layer predictions based on layers 630 and 631. As a result, OLS620 will contain layer 630, which has the lowest quality SNR and / or smallest image size. Since OLS620 does not use inter-layer predictions at all, OLS620 can be fully decoded without referencing any layers except layer 630. OLS621 contains layer 631, which has a higher quality SNR and / or image size than layer 630, and since OLS621 also contains layer 630, layer 631 can be fully decoded according to inter-layer predictions. Similarly, OLS622 includes layer 632, which has a higher quality SNR and / or image size than layers 630 and 631, and since OLS622 also includes layers 630 and 631, layer 632 can be fully decoded according to the inter-layer prediction. Thus, video sequence 600 is coded to scale to any pre-selected SNR and / or image size by sending the corresponding OLS622, 621, or 620 to the decoder. When more OLS622, 621, and 620 are used, video sequence 600 can be scaled to a greater SNR image quality and / or image size.

[0090] Therefore, the video sequence 600 can support spatial scalability. Spatial scalability allows the video sequence 600 to be coded into layers 630, 631, and 632 such that layers 630, 631, and 632 are placed in OLS 620, 621, and 622, each containing enough data to decode the video sequence 600 to the corresponding output screen size. Thus, spatial scalability may include a set of layers for decoding video for smartphone screens (e.g., layer 630), a set of layers for decoding video for large television screens (e.g., layers 630, 631, and 632), and a set of layers for intermediate screen sizes (e.g., layers 630, 631). SNR scalability allows video sequence 600 to be coded into layers 630, 631, and 632 such that layers 630, 631, and 632 are placed in OLS 620, 621, and 622, each containing enough data to decode video sequence 600 at different SNRs. Therefore, SNR scalability may include a set of layers (e.g., layer 630) that can be decoded for low-quality video, high-quality video (e.g., layers 630, 631, and 632), and various intermediate video qualities (e.g., layers 630 and 631) to support different network conditions.

[0091] This disclosure provides efficient signaling to enable the accurate and efficient use of a video sequence 600 having multiview layers. As an example, layers 630, 631, and 632 may all be designated as output layers. The decoder can then select and render layers 630, 631, and 632 as desired to realize the multiview. To support this implementation, the coding of the video sequence 600 may be indicated according to the ols_mode_idc syntax element. For example, the ols_mode_idc syntax element may identify the video sequence 600 as OLS mode 1. Thus, the ols_mode_idc syntax element can be set to 1 and transmitted in the bitstream to indicate that the video sequence 600 is to be used. Therefore, the decoder can receive any OLS and, based on ols_mode_idc, determine that the number of OLS620, 621, and 622 is the same as the number of layers 630, 631, and 632, that the current OLS ID i indicates that the current OLS contains a set of layers with IDs from 0 to i, and that all layers in the current OLS are output layers. The decoder can then decode and display layers 630, 631, and / or 632 from OLS620, 621, and / or 622 as desired to achieve multiview.

[0092] Figure 7 is a schematic diagram showing an exemplary bitstream 700 including an OLS configured for multiview scalability. For example, bitstream 700 can be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, bitstream 700 may include coded multilayer video sequences 500 and / or video sequences 600.

[0093] Bitstream 700 includes a VPS 711, one or more Sequence Parameter Sets (SPS) 713, multiple Picture Parameter Sets (PPS) 715, multiple Slice Headers 717, and Image Data 720. The VPS 711 contains data related to the entire bitstream 700. For example, the VPS 711 may include data-related OLS, layers, and / or sublayers used in the bitstream 700. The SPS 713 contains sequence data common to all pictures in the coded video sequences included in the bitstream 700. For example, each layer may contain one or more coded video sequences, and each coded video sequence may reference the SPS 713 for its corresponding parameters. Parameters within the SPS 713 can include picture size, bit depth, coding tool parameters, bitrate limits, etc. While each sequence references the SPS 713, it should be noted that in some examples, a single SPS 713 can contain data for multiple sequences. PPS715 contains parameters that apply to the entire picture. Therefore, each picture in a video sequence may reference PPS715. While each picture references PPS715, it should be noted that in some examples, a single PPS715 can contain data for multiple pictures. For example, several similar pictures may be coded according to similar parameters. In such cases, a single PPS715 may contain data for such similar pictures. PPS715 can indicate the coding tools, quantization parameters, offsets, etc., available for slices within the corresponding picture.

[0094] The slice header 717 contains parameters specific to each slice 727 within the picture 725. Therefore, in a video sequence, there may be one slice header 717 per slice 727. The slice header 717 may include slice type information, POC, reference picture list, prediction weights, tile entry points, deblocking parameters, etc. It should be noted that in some examples, the bitstream 700 may also include a picture header, which is a syntactic structure containing parameters applicable to all slices 727 within a single picture. For this reason, the picture header and slice header 717 may be used interchangeably in several contexts. For example, certain parameters may be moved between the slice header 717 and the picture header depending on whether such parameters are common to all slices 727 within the picture 725.

[0095] Image data 720 includes video data encoded according to interpretation and / or intrapretation, as well as corresponding transformed and quantized residual data. For example, image data 720 may include layer 723 of picture 725. Layer 723 may be organized into OLS721. OLS721 may be substantially similar to OLS525, 620, 621, and / or 622. Specifically, OLS721 is a set of layers of 723, with one or more layers 723 designated as output layers. When layer 723 contains multiview video, all of layer 723 may be designated as output layers. For example, bitstream 700 may be coded to include several OLS721 having video coded at different resolutions, frame rates, picture 725 sizes, etc. Upon request from the decoder, the sub-bitstream extraction process can remove everything from bitstream 700 except the requested OLS721. The encoder can then transmit to the decoder a bitstream 700 containing only the requested OLS721, and therefore only video that meets the requested criteria.

[0096] Layer 723 may be substantially equivalent to Layer N 531, Layer N+1 532, and / or Layers 631, 632, and / or 633. Layer 723 is generally a set of encoded pictures 725. Layer 723 may be formally defined as a set of VCL NAL units that, when decoded, share a specified characteristic (e.g., common resolution, frame rate, image size, etc.). Picture 725 may be coded as a set of VCL NAL units. Layer 723 also includes associated non-VCL NAL units to support the decoding of VCL NAL units. The VCL NAL units of Layer 723 may share a specific value of nuh_layer_id, which is an exemplary layer ID. Layer 723 may be a simulcast layer coded without inter-layer prediction, or a layer coded according to inter-layer prediction based on other layers.

[0097] Picture 725 is an array of luma samples and / or chroma samples that make up a frame or its fields. For example, Picture 725 may be a coded image that may be output for display or used to support the coding of other Picture 725s for output. Picture 725 may contain a set of VCL NAL units. Picture 725 contains one or more slices 727. Slice 727 may be defined as a single NAL unit, specifically an integer number of complete tiles of Picture 725 exclusively contained in a VCL NAL unit, or an integer number of consecutive complete coding tree unit (CTU) rows (e.g., within a tile). Slice 727 is further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a group of samples of a given size that can be partitioned by a coding tree. A CTB is a subset of a CTU and contains the luma or chroma components of a CTU. A CTU / CTB is further divided into coding blocks based on a coding tree. The coding block can then be encoded / decoded according to the prediction mechanism.

[0098] This disclosure includes a mechanism for supporting spatial and / or SNR scalability for multiview video, for example, by using video sequence 600. For example, VPS711 may include ols_mode_idc735. ols_mode_idc735 is a syntax element that indicates information relating to the number of OLS721, layer 723 of OLS721, and output layer 723 within OLS721. Output layer 723 is any layer designated to be output by the decoder rather than simply used for reference-based coding. ols_mode_idc735 may be set to 0 or 2 for coding other types of video. ols_mode_idc735 can be set to 1 for spatial and / or SNR scalability for multiview video. For example, ols_mode_idc735 can be set to 1 to indicate that the total number of OLS721s in a video sequence is equal to the total number of layers 723 specified in VPS711, that the i-th OLS721 contains layers between 0 and i (inclusive), and that for each OLS721, all layers contained within the OLS721 are output layers. This set of conditions may describe any video sequence 600 having any number of OLS721s. The advantage of using ols_mode_idc735 is that it provides bit saving. Decoders in application systems generally receive only a single OLS. However, ols_mode_idc735 also provides bit saving in encoded bitstreams containing multiple OLSs where much data is shared, thus providing savings on streaming servers and bandwidth savings for transmitting such bitstreams. Specifically, the advantage of setting ols_mode_idc735 to 1 is that it supports use cases such as multi-view applications, where two or more views, each represented by a single layer, are output and displayed simultaneously.

[0099] In some examples, VPS711 also includes the VPS Max Layers Minus 1 (vps_max_layers_minus1)737 syntax element. vps_max_layers_minus1737 is a syntax element that conveys the number of layers 723 specified by VPS700, and therefore the maximum number of layers 723 allowed in the corresponding coded video sequence within bitstream 700. ols_mode_idc735 may refer to the vps_max_layers_minus1737 syntax element. For example, ols_mode_idc735 may indicate that the total number of OLS721 is equal to the number of layers 723 specified by vps_max_layers_minus1737.

[0100] Furthermore, VPS711 may include each_layer_is_an_ols_flag733. each_layer_is_an_ols_flag733 is a syntax element that communicates whether each OLS721 in bitstream 700 contains a single layer 723. For example, when scalability is not used, each OLS721 may contain a single simulcast layer. Therefore, each_layer_is_an_ols_flag733 can be set (e.g., to 0) to indicate that one or more OLS721s contain more than one layer 723 to support scalability. Thus, each_layer_is_an_ols_flag733 can be used to support scalability. For example, a decoder can check each_layer_is_an_ols_flag733 to determine that some of the OLS721s contain more than one layer 723. When each_layer_is_an_ols_flag733 is set to 0, and ols_mode_idc735 is set to 1 (or 0 when used in a different mode), the total number of OLS (TotalNumOlss) can be set to equal to vps_max_layers_minus1737. TotalNumOlss is a variable used by both the decoder and the hypothetical reference decoder (HRD) in the encoder. TotalNumOlss is a variable used to store the number of OLS721 based on the data in bitstream 700. TotalNumOlss can then be used for decoding in the decoder or for checking bitstream 700 errors in the HRD in the encoder.

[0101] VPS711 may also include the VPS layer identifier (vps_layer_id[i])731 syntax element. vps_layer_id[i]731 is an array that stores the layer ID (e.g., nuh_layer_id) of each layer. Thus, vps_layer_id[i]731 indicates the layer ID of the i-th layer.

[0102] The decoder or HRD may also use the data in VPS711 to determine the configuration of OLS721 and layer 723. In a specific example, the number of layers in the i-th OLS (numLayersInOls[i]) and the layer ID in the OLS (LayerIdInOLS[i][j]) specifying the nuh_layer_id value of the j-th layer in the i-th OLS are as follows, i.e., NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] This is derived as follows, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

[0103] The data in VPS711 can be used to support SNR and / or spatially scalable layer 723, including multiview video. Layer 723 can be encoded and contained in OLS721. The encoder can transmit a bitstream 700 containing the requested OLS721 and VPS711 to the decoder. The decoder can then use the information in VPS711 to accurately decode layer 723 in OLS721. This technique supports coding efficiency while supporting scalability. Specifically, the decoder can quickly determine the number of layers 723 in OLS721, determine that all layers in OLS721 are output layers, and decode the output layers according to inter-layer predictions. The decoder can then select the output layers to be rendered to realize multiview. Thus, the decoder may receive the layers 723 necessary to decode the views for multiview, and the decoder can decode and display the pictures 725 from layer 723 as desired. Thus, the total number of encoded layers 723 may not affect the decoding process, and one or more of the above errors may be avoided. Therefore, the disclosure mechanism improves the functionality of the encoder and / or decoder. Furthermore, the disclosure mechanism can reduce the bitstream size and thus reduce the utilization of processor, memory, and / or network resources in both the encoder and decoder.

[0104] The above information is explained in more detail below. Hierarchical video coding is also called scalable video coding or video coding with scalability. Scalability in video coding may be supported by using multi-layer coding techniques. A multi-layer bitstream includes a base layer (BL) and one or more enhancement layers (EL). Examples of scalability include spatial scalability, signal-to-noise ratio (SNR) scalability, multi-view scalability, and frame rate scalability. When multi-layer coding techniques are used, a picture or part thereof may be coded without using a reference picture (intra-prediction), by referencing a reference picture in the same layer (inter-prediction), and / or by referencing a reference picture in another layer (inter-layer prediction). A reference picture currently used for inter-layer prediction of a picture is called an inter-layer reference picture (ILRP). Figure 5 shows an example of multi-layer coding for spatial scalability, where pictures in different layers have different resolutions.

[0105] Some video coding families provide support for scalability in a profile separate from the profile for single-layer coding. Scalable video coding (SVC) is a scalable extension of advanced video coding (AVC) that provides support for spatial, temporal, and quality scalability. For SVC, a flag is propagated in each MB within an EL picture to indicate whether the EL macroblock (MB) is predicted using a block of the same location from a lower layer. Predictions from blocks of the same location may include textures, motion vectors, and / or coding modes. SVC implementations do not necessarily have to directly reuse the unmodified AVC implementation in these designs. The SVC EL macroblock syntax and decoding process is different from the AVC syntax and decoding process.

[0106] Scalable HEVC (SHVC) is an extension of HEVC that provides support for spatial and quality scalability. Multiview HEVC (MV-HEVC) is an extension of HEVC that provides support for multiview scalability. 3D HEVC (3D-HEVC) is an extension of HEVC that provides support for more advanced and efficient 3D video coding than MV-HEVC. Temporal scalability may be included as an integral part of the single-layer HEVC codec. In the multi-layer extension of HEVC, decoded pictures used for inter-layer prediction originate from only the same AU and are treated as long-term reference pictures (LTRPs). Such pictures are assigned a reference index in the reference picture list, along with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the reference index value to refer to the inter-layer reference picture in the reference picture list. Spatial scalability involves resampling a reference picture or a portion thereof when the ILRP has a different spatial resolution than the current picture being encoded or decoded. Reference picture resampling can be achieved at either the picture level or the coding block level.

[0107] VVC may also support hierarchical video coding. A VVC bitstream can contain multiple layers. All layers can be independent of each other. For example, each layer can be coded without using inter-layer prediction. In this case, the layer is also called a simulcast layer. In some cases, some of the layers are coded using ILP. Flags in the VPS can indicate whether a layer is a simulcast layer or whether some layers use ILP. When some layers use ILP, layer dependencies between layers are also propagated in the VPS. Unlike SHVC and MV-HEVC, VVC does not require an OLS to be specified. The OLS contains a specified set of layers, one or more of which are specified as output layers. Output layers are the layers of the output OLS. In some implementations of VVC, when a layer is a simulcast layer, only one layer may be selected for decoding and output. In some implementations of VVC, the entire bitstream containing all layers is specified to be decoded when any of the layers use ILP. Furthermore, certain layers within the layers are specified as output layers. The output layers may be indicated as only the highest layer, all layers, or the highest layer plus a set of lower layers indicated by that layer.

[0108] The above configuration involves specific problems. In some video coding systems, when inter-layer prediction is used, the entire bitstream and all layers are specified to be decoded, and a specific layer within the layers is specified as the output layer. The output layer may be specified as only the highest layer, all layers, or the highest layer plus a set of lower layers specified. For simplicity in describing the problem, two layers may be used by an upper layer that uses lower layers for inter-layer prediction references. For multi-view scalability, the system should specify the use of only lower layers (decoding and output of only lower layers). The system should also specify the use of both layers (decoding and output of both layers). Unfortunately, this is not possible in some video coding systems.

[0109] Generally, this disclosure describes a method for simple and efficient signaling of output layer sets (OLS) for multiview scalability. The description of the technique is based on VVC by ITU-T and ISO / IEC JVET. However, the technique is also applicable to hierarchical video coding based on other video codec specifications.

[0110] One or more of the above problems may be solved as follows. Specifically, this disclosure includes a simple and efficient method for signaling OLS for spatial and SNR scalability. A video coding system may use VPS to indicate that some layers use ILP, the total number of OLS specified by VPS is equal to the number of layers, the i-th OLS includes layers with layer indices between 0 and i, and for each OLS, only the best layer within the OLS is output.

[0111] An exemplary implementation of the above mechanism is as follows. The syntax of the exemplary video parameter set is as follows. [Table 1] TIFF2026077726000003.tif181170

[0112] The semantics for an exemplary video parameter set are as follows: The VPS RBSP should be available to the decoding process before being referenced and should be contained in at least one access unit with a TemporalId equal to 0, or provided through an external mechanism. The VPS NAL unit containing the VPS RBSP should have a nuh_layer_id equal to vps_layer_id[0]. All VPS NAL units in a CVS with a specific value of vps_video_parameter_set_id should have the same content. vps_video_parameter_set_id provides an identifier for the VPS to be referenced by other syntax elements. vps_max_layers_minus1 plus 1 specifies the maximum allowed number of layers in each CVS referencing the VPS. vps_max_sub_layers_minus1 plus 1 specifies the maximum number of time sublayers that may exist in each CVS referencing the VPS. The value of vps_max_sub_layers_minus1 should be in the range of 0 to 6.

[0113] The vps_all_independent_layers_flag may be set to equal to 1 to specify that all layers in CVS are coded independently without using inter-layer prediction. The vps_all_independent_layers_flag may be set to equal to 0 to specify that one or more layers in CVS may use inter-layer prediction. If it does not exist, the value of vps_all_independent_layers_flag is assumed to be equal to 1. When vps_all_independent_layers_flag is equal to 1, the value of vps_independent_layer_flag[i] is assumed to be equal to 1. When vps_all_independent_layers_flag is equal to 0, the value of vps_independent_layer_flag[0] is assumed to be equal to 1. vps_layer_id[i] specifies the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​m and n, when m is less than n, the value of vps_layer_id[m] should be less than vps_layer_id[n]. vps_independent_layer_flag[i] may be set to equal to 1 to specify that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] may be set to equal to 0 to specify that the layer with index i may use inter-layer prediction and that vps_layer_dependency_flag[i] exists in the VPS. If it does not exist, the value of vps_independent_layer_flag[i] is assumed to be equal to 1.

[0114] The vps_direct_dependency_flag[i][j] may be set to 0 to specify that the layer with index j is not a direct reference layer for the layer with index i. The vps_direct_dependency_flag[i][j] may be set to 1 to specify that the layer with index j is a direct reference layer for the layer with index i. When vps_direct_dependency_flag[i][j] does not exist for i and j in the range of 0 to vps_max_layers_minus1, vps_direct_dependency_flag[i][j] is presumed to be equal to 0. The variable DirectDependentLayerIdx[i][j], which specifies the j-th direct dependent layer of the i-th layer, is defined as follows: for(i=1;i <vps_max_layers_minus1;i++) if(!vps_independent_layer_flag[i]) for(j=i,k=0;j>=0;j--) if(vps_direct_dependency_flag[i][j]) DirectDependentLayerIdx[i][k++]=j It is derived as follows.

[0115] The variable GeneralLayerIdx[i], which specifies the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i], is as follows: for(i=0;i<=vps_max_layers_minus1;i++) GeneralLayerIdx[vps_layer_id[i]]=i It is derived as follows.

[0116] `each_layer_is_an_ols_flag` may be set to equal to 1 to specify that each output layer set contains only one layer, and each layer in the bitstream is itself an output layer set, by having only one contained layer as the output layer. `each_layer_is_an_ols_flag` may be set to equal to 0 to specify that an output layer set may contain more than one layer. If `vps_max_layers_minus1` is equal to 0, the value of `each_layer_is_an_ols_flag` is assumed to be equal to 1. Otherwise, if `vps_all_independent_layers_flag` is equal to 0, the value of `each_layer_is_an_ols_flag` is assumed to be equal to 0.

[0117] ols_mode_idc may be set to 0 to specify that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices between 0 and i (inclusive), and for each OLS, only the highest layer within the OLS is output. ols_mode_idc may be set to 1 to specify that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices between 0 and i (inclusive), and for each OLS, all layers within the OLS are output. ols_mode_idc may be set to 2 to specify that the total number of OLS specified by the VPS is explicitly communicated, and for each OLS, the highest layer within the OLS and a set of explicitly communicated lower layers are output. The value of ols_mode_idc should be in the range of 0 to 2. The value of ols_mode_idc 3 is reserved. When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of ols_mode_idc is estimated to be equal to 2.

[0118] Adding 1 to num_output_layer_sets_minus1 specifies the total number of OLS specified by the VPS when ols_mode_idc is equal to 2. The variable TotalNumOlss, which specifies the total number of OLS specified by the VPS, is as follows, i.e., if(vps_max_layers_minus1==0) TotalNumOlss=1 else if(each_layer_is_an_ols_flag||ols_mode_idc==0||ols_mode_idc==1) TotalNumOlss=vps_max_layers_minus1+1 else if(ols_mode_idc==2) TotalNumOlss=num_output_layer_sets_minus1+1 It is derived as follows.

[0119] `layer_included_flag[i][j]` specifies whether the j-th layer (for example, a layer with a nuh_layer_id equal to vps_layer_id[j]) is included in the i-th OLS when ols_mode_idc is equal to 2. `layer_included_flag[i][j]` may be set to equal to 1 to specify that the j-th layer is included in the i-th OLS. `layer_included_flag[i][j]` may be set to equal to 0 to specify that the j-th layer is not included in the i-th OLS.

[0120] The variable NumLayersInOls[i], which specifies the number of layers in the i-th OLS, and the variable LayerIdInOls[i][j], which specifies the nuh_layer_id value of the j-th layer in the i-th OLS, are defined as follows: NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] } else if(ols_mode_idc==2){ for(k=0,j=0;k<=vps_max_layers_minus1;k++) if(layer_included_flag[i][k]) LayerIdInOls[i][j++]=vps_layer_id[k] NumLayersInOls[i]=j } } It can also be derived as follows.

[0121] The variable OlsLayeIdx[i][j], which specifies the OLS layer index of the layer having a nuh_layer_id equal to LayerIdInOls[i][j], is defined as follows: for(i=0,i <TotalNumOlss;i++) for j=0;j <NumLayersInOls[i];j++) OlsLayeIdx[i][LayerIdInOls[i][j]]=j It can also be derived as follows.

[0122] The lowest layer within each OLS should be an independent layer. In other words, for each i in the range of 0 to TotalNumOlss-1, the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] should be equal to 1. Each layer may be contained in at least one OLS specified by the VPS. In other words, for each layer having a particular value of nuh_layer_id (for example, nuhLayerId is equal to one of vps_layer_id[k] for k in the range of 0 to vps_max_layers_minus1), there should be at least one pair of i and j values, where i is in the range of 0 to TotalNumOlss-1 and j is in the range including NumLayersInOls[i]-1, so that the value of LayerIdInOls[i][j] is equal to nuhLayerId. Any layer within the OLS shall be either an OLS output layer or a (direct or indirect) reference layer of an OLS output layer.

[0123] The vps_output_layer_flag[i][j] specifies whether the j-th layer in the i-th OLS is output when ols_mode_idc is equal to 2. The vps_output_layer_flag[i] may be set to equal to 1 to specify that the j-th layer in the i-th OLS is output. The vps_output_layer_flag[i] may be set to equal to 0 to specify that the j-th layer in the i-th OLS is not output. When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of vps_output_layer_flag[i] can be presumed to be equal to 1.

[0124] The variable OutputLayerFlag[i][j], where a value of 1 specifies that the j-th layer in the i-th OLS should be output, and a value of 0 specifies that the j-th layer in the i-th OLS should not be output, is defined as follows: for(i=0,i <TotalNumOlss;i++){ OutputLayerFlag[i][NumLayersInOls[i]-1]=1 for(j=0;j <NumLayersInOls[i]-1;j++) if(ols_mode_idc[i]==0) OutputLayerFlag[i][j]=0 else if(ols_mode_idc[i]==1) OutputLayerFlag[i][j]=1 else if (ols_mode_idc[i]==2) OutputLayerFlag[i][j]=vps_output_layer_flag[i][j] } It can also be derived as follows.

[0125] The 0th OLS will only include the lowest layer (e.g., the layer with a nuh_layer_id equal to vps_layer_id[0]), and for the 0th OLS, only the included layers will be output. vps_constraint_info_present_flag may be set to equal to 1 to indicate that the general_constraint_info() syntax structure exists in the VPS. vps_constraint_info_present_flag may be set to equal to 0 to indicate that the general_constraint_info() syntax structure does not exist in the VPS. vps_reserved_zero_7bits should be equal to 0 in a compatible bitstream. Other values ​​for vps_reserved_zero_7bits are reserved. The decoder should ignore the value of vps_reserved_zero_7bits.

[0126] The general_hrd_params_present_flag may be set to equal to 1 to specify that the syntax elements num_units_in_tick and time_scale and the syntax structure general_hrd_parameters() exist in the SPS RBSP syntax structure. The general_hrd_params_present_flag may be set to equal to 0 to specify that the syntax elements num_units_in_tick and time_scale and the syntax structure general_hrd_parameters() do not exist in the SPS RBSP syntax structure. num_units_in_tick is the number of time units of a clock operating at a frequency of time_scale Hertz (Hz) corresponding to one increment of the clock tick counter (called a clock tick). num_units_in_tick should be greater than 0. The clock tick in seconds is equal to the quotient obtained by dividing num_units_in_tick by time_scale. For example, when the picture rate of the video signal is 25Hz, time_scale may be equal to 27,000,000, num_units_in_tick may be equal to 1,080,000, and as a result, the clock tick may be equal to 0.04 seconds.

[0127] `time_scale` is the number of time units that elapse in one second. For example, a time coordinate system that measures time using a 27 megahertz (MHz) clock has a `time_scale` of 27,000,000. The value of `time_scale` should be greater than 0. `vps_extension_flag` may be set to 0 to specify that the `vps_extension_data_flag` syntax element does not exist in the VPS RBSP syntax structure. `vps_extension_flag` may be set to 1 to specify that the `vps_extension_data_flag` syntax element exists in the VPS RBSP syntax structure. `vps_extension_data_flag` may have any value. The presence and value of `vps_extension_data_flag` do not affect the decoder's suitability to the profile. A suitable decoder should ignore all `vps_extension_data_flag` syntax elements.

[0128] Figure 8 is a schematic diagram of an exemplary video coding device 800. The video coding device 800 is suitable for implementing examples / embodiments of the disclosure described herein. The video coding device 800 includes a downstream port 820, an upstream port 850, and / or a transceiver unit (Tx / Rx) 810 including a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding device 800 also includes a processor 830 including a logic unit and / or a central processing unit (CPU) for processing data, and a memory 832 for storing data. The video coding device 800 may also include an electrical-to-electrical (OE) component, an electrical-to-optical (EO) component, and / or a wireless communication component coupled to the upstream port 850 and / or downstream port 820 for communicating data over an electrical, optical, or wireless communication network. The video coding device 800 may also include an input and / or output (I / O) device 860 for communicating data to and from the user. The I / O device 860 may include output devices such as a display for displaying video data and a speaker for outputting audio data. The I / O device 860 may also include input devices such as a keyboard, mouse, trackball, and / or corresponding interfaces for interacting with such output devices.

[0129] The processor 830 is implemented by hardware and software. The processor 830 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 830 communicates with downstream port 820, Tx / Rx 810, upstream port 850, and memory 832. The processor 830 includes a coding module 814. The coding module 814 may implement embodiments of the disclosures described herein, such as methods 100, 900, and 1000, using a multi-layer video sequence 500, a video sequence 600, and / or a bitstream 700. The coding module 814 may also implement any other methods / mechanisms described herein. Furthermore, the coding module 814 may implement a codec system 200, an encoder 300, and / or a decoder 400. For example, coding module 814 may be used to code a video sequence into layers and / or OLS to support multiview scalability. For example, coding module 814 may encode the ols_mode_idc syntax element into a VPS in the bitstream and / or decode it from the VPS in the bitstream. The ols_mode_idc syntax element can indicate that the total number of OLS in the video sequence is equal to the total number of layers specified in the VPS, that the i-th OLS contains layers from 0 to i (inclusive), and for each OLS, all layers within the OLS are output. Thus, coding module 814 may use the ols_mode_idc syntax element to indicate / determine that all layers received from the scalable video can be decoded and displayed as desired to achieve multiview video.Therefore, the coding module 814 provides the video coding device 800 with additional functionality and / or coding efficiency when coding video data. Thus, the coding module 814 improves the functionality of the video coding device 800 and addresses problems specific to the field of video coding. Furthermore, the coding module 814 enables the video coding device 800 to switch to different states. Alternatively, the coding module 814 can be implemented as instructions stored in memory 832 and executed by processor 830 (for example, as a computer program product stored on a non-temporary medium).

[0130] Memory 832 includes one or more memory types such as disks, tape drives, solid-state drives, read-only memory (ROM), random access memory (RAM), flash memory, ternary content-addressable memory (TCAM), and static random-access memory (SRAM). Memory 832 may also be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution.

[0131] Figure 9 is a flowchart of an exemplary method 900 for encoding a video sequence having an OLS configured for multiview scalability, such as a multilayer video sequence 500 and / or a video sequence 600 within a bitstream 700. Method 900 may be used with an encoder such as a codec system 200, an encoder 300, and / or a video coding device 800 when performing Method 100.

[0132] Method 900 may begin when an encoder receives a video sequence and decides, for example, based on user input, to encode the video sequence as a scalable multiview video sequence in a set of layers and an OLS. The video sequence may be configured to support multiview and may be coded to support SNR scalability, spatial scalability, scalability by other characteristics described herein, or a combination thereof. In step 901, the encoder can encode a bitstream containing one or more OLSs, each containing one or more layers of the coded picture. For example, the layers may include a base layer having the lowest layer ID and various enhancement layers having increasing layer IDs. Each enhancement layer having a layer ID of j may be coded according to inter-layer predictions based on the base layer and any enhancement layer having a layer ID less than j. The OLS may include an OLS ID, which may be denoted by i to distinguish it from the layer ID of j. For example, there may be one OLS per layer. Thus, an OLS having an OLS ID of i may include an output layer having a layer ID of j, where i is equal to i. An OLS with OLS ID i may also include all layers with layer IDs between 0 and j-1 (inclusive). In this example, all layers may be set as output layers. For example, an OLS with OLS ID 5 may include layers 0 through 5, each of which is shown as an output layer.

[0133] In step 903, the encoder encodes the VPS into a bitstream. The configuration of the OLS and layers may be indicated by the VPS. The VPS includes the ols_mode_idc syntax element. ols_mode_idc can be set to specify that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS. Furthermore, ols_mode_idc can be set to specify that the i-th OLS contains layers with layer indices greater than or equal to 0 and / or j (for example, in this case i is equal to j). ols_mode_idc may also be set to specify that for each OLS, all layers within each OLS are output layers. For example, ols_mode_idc may be set to one of several modes. When ols_mode_idc is set to 1, the above modes may be propagated. In some examples, the VPS may also include vps_max_layers_minus1, which specifies the number of layers specified by the VPS. This is also the maximum allowed number of layers within each CVS referencing the VPS. ols_mode_idc may refer to vps_max_layers_minus1.

[0134] As an example, a video sequence can be decoded in a hypothetical reference decoder (HRD) in a decoder and / or encoder for standard verification purposes. When decoding a video sequence, the variable TotalNumOlss for the video sequence can be set to equal to vps_max_layers_minus1 plus 1 when each_layer_is_an_ols_flag in the VPS is set to 0, when ols_mode_idc is set to 0, or when ols_mode_idc is set to 1. As a specific example, the number of layers in the i-th OLS (NumLayersInOls[i]) and the layer ID in the OLS that specifies the nuh_layer_id value of the j-th layer in the i-th OLS (LayerIdInOLS[i][j]) are as follows, i.e., NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] This can be derived as follows, where vps_layer_id[i] is the identifier of the i-th VPS layer, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer. Once the IDs of the OLS and the IDs of the output layers are known, the HRD in the encoder can begin decoding the coded pictures in the output layers by using cross-layer predictions to perform conformance checks to ensure the video conforms to the standard.

[0135] In step 905, the encoder can store the bitstream for communication to the decoder. For example, the decoder may be aware of available OLS (Olst-Live Streams) (e.g., via communication and / or other protocols such as dynamic adaptive streaming over hypertext transfer protocol, DASH). The decoder can select and request the OLS with the best ID that can be properly decoded / displayed by the decoder. For example, in the case of spatial scalability, the decoder can request an OLS with multiview video and picture sizes relevant to the screen connected to the decoder. In the case of SNR scalability, the decoder can request the best ID OLS with multiview video that can be decoded in light of current network conditions (e.g., in light of available communication bandwidth). The encoder and / or intermediate cache or content server can then transmit the OLS and associated layers to the decoder for decoding. Thus, the encoder can create multiview video sequences that can be scaled up or down based on the decoder's needs.

[0136] Figure 10 is a flowchart of an exemplary method 1000 for decoding a video sequence containing an OLS configured for multi-view scalability, such as a multi-layer video sequence 500 and / or a video sequence 600 within a bitstream 700. Method 1000 may be used with a decoder such as a codec system 200, a decoder 400, and / or a video coding device 800 when performing Method 100.

[0137] Method 1000 may begin when the decoder begins receiving a bitstream containing an OLS having a set of layers of a scalable multiview video sequence, for example, as a result of Method 900. The video sequence may be coded to support SNR scalability, spatial scalability, scalability by other characteristics described herein, or a combination thereof. In step 1001, the decoder can receive a bitstream containing an OLS and a VPS. For example, an OLS may contain one or more layers of coded pictures. The layers may include a base layer having the lowest layer ID and various enhancement layers having increasing layer IDs. Each enhancement layer having a layer ID of j may be coded according to inter-layer predictions based on the base layer and any enhancement layer having a layer ID less than j. The OLS may include an OLS ID, which may be denoted by i to distinguish it from the layer ID of j. For example, there may be one OLS per layer coded bitstream. Thus, an OLS having an OLS ID of i may include an output layer having a layer ID of j, where i is equal to i. The OLS received with OLS ID i may also include all layers having layer IDs between 0 and j-1. In this example, all layers may be set as output layers. For example, the received OLS with OLS ID 5 may include layers 0 to 5, each of which is shown as an output layer. The configuration of the OLS and layers may be shown by VPS.

[0138] For example, VPS includes the ols_mode_idc syntax element. ols_mode_idc can be set to specify that the total number of OLS specified by VPS is equal to the number of layers specified by VPS. Furthermore, ols_mode_idc can be set to specify that the i-th OLS contains layers with layer indices greater than or equal to i and / or j (for example, in this case i is equal to j). ols_mode_idc may also be set to specify that for each OLS, all layers within each OLS are output layers. For example, ols_mode_idc may be set to one of several modes. When ols_mode_idc is set to 1, the above modes may be propagated. In some examples, VPS may also include vps_max_layers_minus1, which specifies the number of layers specified by VPS. This is also the maximum allowed number of layers within each CVS referencing VPS. ols_mode_idc may refer to vps_max_layers_minus1.

[0139] In step 1003, the decoder can determine the output layer based on ols_mode_idc in the VPS. As a specific example, when determining the configuration of a video sequence, the variable TotalNumOlss for the video sequence can be set to equal vps_max_layers_minus1 plus 1 when each_layer_is_an_ols_flag in the VPS is set to 0, when ols_mode_idc is set to 0, or when ols_mode_idc is set to 1. As a specific example, the number of layers in the i-th OLS (NumLayersInOls[i]) and the layer ID in the OLS (LayerIdInOLS[i][j]) specifying the nuh_layer_id value of the j-th layer in the i-th OLS are as follows, i.e., NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] This can be derived as follows, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

[0140] In step 1005, using the ID of the output layer, the decoder can decode the coded picture from the output layer and generate a decoded picture. For example, the decoder can decode all of the output layer using inter-layer prediction and decode the upper layers based on the lower layers as desired. The decoder can also select layers to achieve multiview. In step 1007, the decoder can transfer the decoded picture for display as part of the decoded video sequence. For example, the decoder may transfer the decoded picture from the first layer for display on a first screen (or part thereof) and transfer the picture from the second set of layers for display on a second screen (or part thereof).

[0141] As a specific example, the decoder may be aware of available OLS (for example, via other protocols such as dynamic adaptive streaming over hypertext transfer protocol, DASH) over communication and / or hypertext transfer protocols. The decoder can select and request the OLS with the best ID that can be properly decoded / displayed by the decoder. For example, in the case of spatial scalability, the decoder can request an OLS with a picture size relevant to the screen connected to the decoder. In the case of SNR scalability, the decoder can request the best ID OLS that can be decoded in light of current network conditions (for example, in light of available communication bandwidth). The encoder and / or intermediate cache or content server can then transmit the OLS and associated layers to the decoder for decoding to support multiview. Thus, the encoder can create a multiview video sequence that can be scaled up or down based on the decoder's needs. The decoder can then decode the requested video sequence upon reception by using method 1000.

[0142] Figure 11 is a schematic diagram of an exemplary system 1100 for coding a video sequence having an OLS configured for multiview scalability, such as a multilayer video sequence 500 and / or a video sequence 600 within a bitstream 700. System 1100 may be implemented by an encoder and decoder such as a codec system 200, an encoder 300, a decoder 400, and / or a video coding device 800. Furthermore, system 1100 may be used when implementing methods 100, 900, and / or 1000.

[0143] System 1100 includes a video encoder 1102. The video encoder 1102 includes an encoding module 1105 for encoding a bitstream containing one or more OLSs, each containing one or more layers of a coded picture. Furthermore, the encoding module 1105 is for encoding a VPS into the bitstream, the VPS containing an ols_mode_idc for each OLS, specifying that all layers within each OLS are output layers. The video encoder 1102 further includes a storage module 1106 for storing the bitstream for communication toward the decoder. The video encoder 1102 further includes a transmission module 1107 for transmitting the bitstream toward the video decoder 1100. The video encoder 1102 may be further configured to perform any of the steps of method 900.

[0144] System 1100 also includes a video decoder 1110. The video decoder 1110 includes a receive module 1111 for receiving a bitstream including an OLS and a VPS, where the OLS includes one or more layers of coded pictures, and the VPS includes an ols_mode_idc for each OLS that specifies that all layers within each OLS are output layers. The video decoder 1100 further includes a determination module 1113 for determining the output layers based on the ols_mode_idc in the VPS. The video decoder 1110 further includes a decode module 1115 for decoding the coded pictures from the output layers and generating the decoded pictures. The video decoder 1110 further includes a transfer module 1115 for transferring the decoded pictures for display as part of a decoded video sequence. The video decoder 1110 may be further configured to perform any of the steps of method 1000.

[0145] The first component is directly coupled to the second component when there are no intervening components other than lines, traces, or other media between the first component and the second component. The first component is indirectly coupled to the second component when there are intervening components other than lines, traces, or other media between the first component and the second component. The term "coupled" and its variations include both direct and indirect coupling. The use of the term "about" means a range including ±10% of the following number unless otherwise specified.

[0146] It should be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of the steps of such methods should be understood to be merely illustrative. Similarly, in methods consistent with various embodiments of the present disclosure, further steps may be included in such methods, and certain steps may be omitted or combined.

[0147] While several embodiments are provided in this disclosure, it should be understood that the systems and methods of the disclosure may be embodied in many other specific forms without departing from the intent or scope of this disclosure. These examples should be considered illustrative, not restrictive, and their intent is not limited to the details given herein. For example, various elements or components may be combined or integrated into other systems, or certain features may be omitted or not implemented.

[0148] Furthermore, the technologies, systems, subsystems, and methods described and illustrated individually or separately in various embodiments may be combined or integrated with other systems, components, technologies, or methods without departing from the scope of this disclosure. Other examples of modifications, substitutions, and alterations are evident to those skilled in the art and may be made without departing from the true intent and scope of the disclosures herein.

Claims

1. A method implemented by a decoder, the method is A step of receiving a bitstream containing a video parameter set (VPS), wherein the VPS contains an output layer set (OLS) mode identification code (ols_mode_idc), and an ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS contains layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers. The steps include determining the output layer based on the ols_mode_idc within the VPS, The steps include decoding the output layer and generating the decoded picture. A method that includes this.

2. The method according to claim 1, wherein the VPS includes a VPS maximum layer minus 1 (vps_max_layers_minus1) which specifies the number of layers designated by the VPS, and the value of vps_max_layers_minus1 plus 1 is the maximum allowed number of layers in each coded video sequence (CVS) that references the VPS.

3. The method according to claim 1, wherein the VPS includes each_layer_is_an_ols_flag, where each_layer_is_an_ols_flag equal to 1 specifies that each output layer set contains only one layer, and each_layer_is_an_ols_flag equal to 0 specifies that at least one OLS contains more than one layer.

4. The method according to claim 2, wherein when vps_max_layers_minus1 is greater than 0, the VPS further includes vps_all_independent_layers_flag, where vps_all_independent_layers_flag equal to 1 specifies that all layers in the CVS are coded independently without using inter-layer prediction, and vps_all_independent_layers_flag equal to 0 specifies that one or more of the layers in the CVS use inter-layer prediction.

5. The method according to claim 1, wherein the total number of OLS (TotalNumOlss) is equal to vps_max_layers_minus1 plus 1 when ols_mode_idc is equal to 0 or 1.

6. Number of layers in the i-th OLS (NumLayersInOls [i]) and the network abstraction layer (NAL) unit header layer identifier (nuh_layer_id) value (LayerIdInOLS) of the jth layer in the OLS of i. [i][j]) is as follows, that is, NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] The method according to claim 1, wherein the following is derived, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

7. A method implemented by an encoder, the method is A method comprising the step of encoding a video parameter set (VPS) into a bitstream, wherein the VPS includes an output layer set (OLS) mode identifier code (ols_mode_idc), where ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers.

8. The method according to claim 7, wherein the VPS includes a VPS maximum layer count minus 1 (vps_max_layers_minus1) that specifies the number of layers specified by the VPS, and the value of vps_max_layers_minus1 plus 1 is the maximum allowed number of layers in each coded video sequence (CVS) that references the VPS.

9. The method according to claim 7, wherein the VPS includes each_layer_is_an_ols_flag, where each_layer_is_an_ols_flag equal to 1 specifies that each output layer set contains only one layer, and each_layer_is_an_ols_flag equal to 0 specifies that at least one OLS contains more than one layer.

10. The method according to claim 8, wherein when vps_max_layers_minus1 is greater than 0, the VPS further includes vps_all_independent_layers_flag, where vps_all_independent_layers_flag equal to 1 specifies that all layers in the CVS are coded independently without using inter-layer prediction, and vps_all_independent_layers_flag equal to 0 specifies that one or more of the layers in the CVS use inter-layer prediction.

11. The method according to claim 7, wherein the total number of OLS (TotalNumOlss) is equal to vps_max_layers_minus1 plus 1 when ols_mode_idc is equal to 0, or when ols_mode_idc is equal to 1.

12. Number of layers in the i-th OLS (NumLayersInOls [i]) and the network abstraction layer (NAL) unit header layer identifier (nuh_layer_id) value (LayerIdInOLS) of the jth layer in the OLS of i. [i][j]) is as follows, that is, NumLayersInOls[0]=1 LayerIdInOls[0][0]=vps_layer_id[0] for(i=1,i <TotalNumOlss;i++){ if(each_layer_is_an_ols_flag){ NumLayersInOls[i]=1 LayerIdInOls[i][0]=vps_layer_id[i] }else if(ols_mode_idc==0||ols_mode_idc==1){ NumLayersInOls[i]=i+1 for(j=0;j <NumLayersInOls[i];j++) LayerIdInOls[i][j]=vps_layer_id[j] The method according to claim 7, wherein the following is derived, where vps_layer_id[i] is the i-th VPS layer identifier, TotalNumOlss is the total number of OLS specified by the VPS, and each_layer_is_an_ols_flag is a flag that each layer is an OLS, specifying whether at least one OLS contains more than one layer.

13. A decoder including a processor and a memory coupled to the processor, The decoder is configured to perform the method described in any one of claims 1 to 6, wherein the processor is configured to perform the method described in any one of claims 1 to 6.

14. An encoder including a processor and a memory coupled to the processor, The processor is configured to perform the method described in any one of claims 7 to 12, wherein the encoder is configured to perform the method described in any one of claims 7 to 12.

15. A non-temporary computer-readable medium containing computer program products used by video coding devices, The computer program product includes a non-temporary computer-readable medium, which, when executed by a processor, includes computer-executable instructions stored on the non-temporary computer-readable medium, causing the video coding device to perform the method according to any one of claims 1 to 12.

16. It is a decoder, A receiving means for receiving a bitstream containing a video parameter set (VPS), wherein the VPS includes an output layer set (OLS) mode identification code (ols_mode_idc), and an ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers. A determination means for determining the output layer based on the ols_mode_idc within the VPS, Decoding means for decoding the output layer and generating a decoded picture, A decoder that includes this.

17. It is an encoder, An encoder comprising encoding means for encoding a video parameter set (VPS) into a bitstream, wherein the VPS includes an output layer set (OLS) mode identification code (ols_mode_idc), and an ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers.

18. A device for storing a bitstream, comprising at least one memory and at least one receiver, The at least one receiver is configured to receive one or more bitstreams, The at least one memory is configured to store the one or more bitstreams, The bitstream includes a video parameter set (VPS), and the VPS includes an output layer set (OLS) mode identification code (ols_mode_idc). A device in which ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers.

19. A method for storing a bitstream, The steps include receiving one or more bitstreams, The steps include storing the bitstream in one or more memory locations. Includes, The bitstream includes a video parameter set (VPS), and the VPS includes an output layer set (OLS) mode identification code (ols_mode_idc). A method in which ols_mode_idc equal to 1 specifies that the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i, and for each OLS, all layers within that OLS are output layers.

20. A non-temporary computer-readable storage medium containing a bitstream, The bitstream includes a video parameter set (VPS), and the VPS includes an output layer set (OLS) mode identification code (ols_mode_idc). The ols_mode_idc, which is equal to 1, is a non-temporary computer-readable storage medium in which the total number of OLS specified by the VPS is equal to the number of layers specified by the VPS, the i-th OLS includes layers having layer indices between 0 and i (inclusive), and for each OLS, all layers within that OLS are output layers.

Images