Simulcast layer for multi-view in video decoding

Abstract

The present invention discloses a video decoding mechanism. The mechanism includes encoding a code stream including one or more layers of a coded image. A video parameter set (VPS) is also encoded into the code stream. When all layers specified by the VPS are independently coded without requiring inter-layer prediction, the VPS includes a flag (each_layer_is_an_ols_flag) indicating that each layer is an output layer set (OLS). The each_layer_is_an_ols_flag indicates whether each OLS includes only one layer. The code stream is stored for transmission to a decoder.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 905,126, filed by Yekui Wang on September 24, 2019, entitled “Support Of Operation Points With More Than One Output Layer For Simulcast Layers,” which is incorporated herein by reference. Technical Field

[0003] The present invention relates generally to video coding, and more particularly to configuring an output layer set (OLS) in a multi-layer bitstream for multi-view applications. Background Art

[0004] Even relatively short videos require a large amount of video data to describe them, which can create difficulties when the data is to be streamed or otherwise transmitted across communications networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted over modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression devices typically use software and / or hardware to encode the video data at the source side before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination side by a video decompression device that decodes the video data. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the Invention

[0005] In one embodiment, the present invention includes a method implemented in a decoder, the method comprising: a receiver of the decoder receiving a code stream including one or more layers of coded images and a video parameter set (VPS), wherein when all layers specified by the VPS are independently coded without inter-layer prediction, a flag (each_layer_is_an_ols_flag) for each layer being an output layer set (OLS) is included in the VPS, the each_layer_is_an_ols_flag indicating whether each OLS includes only one layer; a processor of the decoder decoding the coded image from the output layer of the OLS according to the each_layer_is_an_ols_flag in the VPS to generate a decoded image; and the processor forwarding the decoded image for display as part of a decoded video sequence.

[0006] Image layers can be used to support adaptability. For example, a video can be encoded into multiple layers. A layer can be encoded without reference to other layers. Such a layer is called a simulcast layer. Accordingly, a simulcast layer can be decoded without reference to other layers. For another example, a layer can be encoded using inter-layer prediction. This allows encoding the current layer by including only the difference between the current layer and a reference layer. Layers can be organized into open layers (OLSs). An OLS is a set of layers consisting of at least one output layer and any layers that support decoding of the output layer. In a specific example, the first OLS may include a base layer, while the second OLS may include the base layer and enhancement layers with added features. In one example, the first OLS can be sent to the decoder to enable decoding of the video at a base resolution, or the second OLS can be transmitted to enable decoding of the video at a higher enhanced resolution. This allows the video to be scaled according to user requests. In some cases, adaptability is not used and each layer is encoded as a simulcast layer. Some systems infer that if all layers are simulcast, each OLS should include a single layer (because no reference layer is used). This inference improves decoding efficiency because the indication can be omitted from the encoded bitstream. However, this inference does not support multi-view. Multi-view is also known as stereo video. In multi-view, two video sequences of the same scene are recorded by spatially offset cameras. These two video sequences are displayed to the user on different lenses of a head-mounted device. Displaying a different spatially offset sequence for each eye in this way can create a three-dimensional (3D) video and / or the impression of visual depth. Therefore, an OLS that implements multi-view includes two layers (e.g., one layer for each eye). However, when these layers are all multicast, a video decoder can use this inference to infer that each OLS includes only one layer. This may lead to errors because the decoder may only display one layer of the multi-view or may not be able to continue to display either of the two layers. Therefore, when all layers are simulcast, the inference that each OLS includes a single layer may prevent multi-view applications from being rendered correctly at the decoder.

[0007] This example includes a mechanism that enables a video coding system to correctly decode multi-view video when all layers in the video are simulcast and inter-layer prediction is not used. When none of the layers use inter-layer prediction (e.g., all layers are simulcast), a VPS all independent layer flag (vps_all_independent_layers_flag) can be included in the VPS in the codestream and set to 1. When this flag is set to 1, each_layer_is_an_ols_flag is indicated in the VPS. each_layer_is_an_ols_flag can be set to indicate whether each OLS includes a single layer or whether at least one OLS includes more than one layer (e.g., to support multi-view). Therefore, vps_all_independent_layers_flag and each_layer_is_an_ols_flag can be used to support multi-view applications. In addition, when this occurs, the OLS mode identification code (ols_mode_idc) can be set to 2 in the VPS. This will result in the number of OLSs and the layers associated with the OLS being explicitly indicated. The decoder can then use this information to correctly decode the OLS, which includes multi-view video. This approach supports error correction while maintaining decoding efficiency. Thus, the disclosed mechanism increases the functionality of the encoder and / or decoder. Furthermore, the disclosed mechanism can reduce the bitstream size and, therefore, reduce the use of processor, memory, and / or network resources at the encoder and decoder.

[0008] Optionally, according to any of the above aspects, in another implementation manner of the aspect, when it is indicated that each OLS includes only one layer and each layer is the only output layer in each OLS, the each_layer_is_an_ols_flag is set to 1.

[0009] Optionally, according to any of the above aspects, in another implementation manner of the aspect, when indicating that at least one OLS includes more than one layer, the each_layer_is_an_ols_flag is set to 0.

[0010] Optionally, according to any of the above aspects, in another implementation of the aspect, when the OLS mode identification code (ols_mode_idc) is equal to 2, the total number of OLSs is explicitly indicated, the layers associated with the OLSs are explicitly indicated, and when the VPS all independent layer flag (vps_all_independent_layers_flag) is set to 1, and when the each_layer_is_an_ols_flag is set to 0, the ols_mode_idc is inferred to be equal to 2.

[0011] Optionally, according to any of the above aspects, in another implementation of the aspect, the VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently encoded without inter-layer prediction.

[0012] Optionally, according to any of the above aspects, in another implementation of the aspect, the VPS includes a VPS maximum number of layers minus 1 (vps_max_layers_minus1) syntax element, the vps_max_layers_minus1 syntax element indicates the number of layers specified by the VPS, and when vps_max_layers_minus1 is greater than 0, the vps_all_independent_layers_flag is indicated.

[0013] Optionally, according to any of the above aspects, in another implementation of the aspect, when ols_mode_idc is equal to 2, the VPS includes the number of output layer sets minus 1 (num_output_layer_sets_minus1), and the num_output_layer_sets_minus1 represents the total number of OLSs specified by the VPS.

[0014] In one embodiment, the present invention includes a method implemented in an encoder, the method comprising: a processor of the encoder encoding a codestream including one or more layers of a coded image; the processor encoding a VPS into the codestream, wherein, when all layers specified by the VPS are independently coded without requiring inter-layer prediction, the VPS includes each_layer_is_an_ols_flag, the each_layer_is_an_ols_flag indicating whether each OLS includes only one layer; a memory coupled to the processor storing the codestream for sending to a decoder.

[0015] Image layers can be used to support adaptability. For example, a video can be encoded into multiple layers. A layer can be encoded without reference to other layers. Such a layer is called a simulcast layer. Accordingly, a simulcast layer can be decoded without reference to other layers. For another example, a layer can be encoded using inter-layer prediction. This allows encoding the current layer by including only the difference between the current layer and a reference layer. Layers can be organized into open layers (OLSs). An OLS is a set of layers consisting of at least one output layer and any layers that support decoding of the output layer. In a specific example, the first OLS may include a base layer, while the second OLS may include the base layer and enhancement layers with added features. In one example, the first OLS can be sent to the decoder to enable decoding of the video at a base resolution, or the second OLS can be transmitted to enable decoding of the video at a higher enhanced resolution. This allows the video to be scaled according to user requests. In some cases, adaptability is not used and each layer is encoded as a simulcast layer. Some systems infer that if all layers are simulcast, each OLS should include a single layer (because no reference layer is used). This inference improves decoding efficiency because the indication can be omitted from the encoded bitstream. However, this inference does not support multi-view. Multi-view is also known as stereo video. In multi-view, two video sequences of the same scene are recorded by spatially offset cameras. These two video sequences are displayed to the user on different lenses of a head-mounted device. Displaying a different spatially offset sequence for each eye in this way can create the impression of 3D video and / or visual depth. Therefore, an OLS that implements multi-view includes two layers (e.g., one layer for each eye). However, when these layers are all broadcast simultaneously, a video decoder can use this inference to infer that each OLS includes only one layer. This may lead to errors because the decoder may only display one layer of the multi-view or may not be able to continue to display either of the two layers. Therefore, when all layers are broadcast simultaneously, the inference that each OLS includes a single layer may prevent multi-view applications from being rendered correctly at the decoder.

[0016] This example includes a mechanism that enables the video coding system to correctly decode multi-view video when all layers in the video are simulcast and inter-layer prediction is not used. When none of the layers use inter-layer prediction (e.g., all layers are simulcast), the vps_all_independent_layers_flag can be included in the VPS in the codestream and set to 1. When this flag is set to 1, each_layer_is_an_ols_flag is indicated in the VPS. each_layer_is_an_ols_flag can be set to indicate whether each OLS includes a single layer or whether at least one OLS includes more than one layer (e.g., to support multi-view). Therefore, vps_all_independent_layers_flag and each_layer_is_an_ols_flag can be used to support multi-view applications. In addition, when this occurs, ols_mode_idc can be set to 2 in the VPS. This will result in the number of OLSs and the layers associated with the OLS being explicitly indicated. The decoder can then use this information to correctly decode the OLSs that include multi-view video. This approach supports error correction while maintaining decoding efficiency. Thus, the disclosed mechanism increases the functionality of the encoder and / or decoder. Furthermore, the disclosed mechanism can reduce the size of the bitstream and, therefore, reduce the use of processor, memory, and / or network resources at the encoder and decoder.

[0017] Optionally, according to any of the above aspects, in another implementation manner of the aspect, when it is indicated that each OLS includes only one layer and each layer is the only output layer in each OLS, the each_layer_is an_ols_flag is set to 1.

[0018] Optionally, according to any of the above aspects, in another implementation manner of the aspect, when indicating that at least one OLS includes more than one layer, the each_layer_is_an_ols_flag is set to 0.

[0019] Optionally, according to any of the above aspects, in another implementation of the aspect, when ols_mode_idc is equal to 2, the total number of OLSs is explicitly indicated, the layers associated with the OLSs are explicitly indicated, and when vps_all_independent_layers_flag is set to 1, and when each_layer_is_an_ols_flag is set to 0, the ols_mode_idc is inferred to be equal to 2.

[0020] Optionally, according to any of the above aspects, in another implementation of the aspect, the VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently encoded without inter-layer prediction.

[0021] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the VPS includes a vps_max_layers_minus1 syntax element, the vps_max_layers_minus1 syntax element indicates the number of layers specified by the VPS, and when vps_max_layers_minus1 is greater than 0, the vps_all_independent_layers_flag is indicated.

[0022] Optionally, according to any of the above aspects, in another implementation manner of the aspect, when ols_mode_idc is equal to 2, the VPS includes num_output_layer_sets_minus1, where num_output_layer_sets_minus1 represents the total number of OLSs specified by the VPS.

[0023] In one embodiment, the present invention includes a video decoding device, which includes: a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are used to execute the method according to any one of the above aspects.

[0024] In one embodiment, the present invention includes a non-transitory computer-readable medium, the non-transitory computer-readable medium including a computer program product for use by a video decoding device, wherein the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium, and when the computer-executable instructions are executed by a processor, the video decoding device performs the method according to any of the above aspects.

[0025] In one embodiment, the present invention includes a decoder comprising: a receiving module for receiving a code stream including one or more layers of a coded image and a VPS, wherein when all layers specified by the VPS are independently coded without inter-layer prediction, each_layer_is_an_ols_flag is included in the VPS, and the each_layer_is_an_ols_flag indicates whether each OLS includes only one layer; a decoding module for decoding the coded image of the output layer from the OLS according to the each_layer_is_an_ols_flag in the VPS to generate a decoded image; and a forwarding module for forwarding the decoded image for display as part of a decoded video sequence.

[0026] Image layers can be used to support adaptability. For example, a video can be encoded into multiple layers. A layer can be encoded without reference to other layers. Such a layer is called a simulcast layer. Accordingly, a simulcast layer can be decoded without reference to other layers. For another example, a layer can be encoded using inter-layer prediction. This allows encoding the current layer by including only the difference between the current layer and a reference layer. Layers can be organized into open layers (OLSs). An OLS is a set of layers consisting of at least one output layer and any layers that support decoding of the output layer. In a specific example, the first OLS may include a base layer, while the second OLS may include the base layer and enhancement layers with added features. In one example, the first OLS can be sent to the decoder to enable decoding of the video at a base resolution, or the second OLS can be transmitted to enable decoding of the video at a higher enhanced resolution. This allows the video to be scaled according to user requests. In some cases, adaptability is not used and each layer is encoded as a simulcast layer. Some systems infer that if all layers are simulcast, each OLS should include a single layer (because no reference layer is used). This inference improves decoding efficiency because the indication can be omitted from the encoded bitstream. However, this inference does not support multi-view. Multi-view is also known as stereo video. In multi-view, two video sequences of the same scene are recorded by spatially offset cameras. These two video sequences are displayed to the user on different lenses of a head-mounted device. Displaying a different spatially offset sequence for each eye in this way can create the impression of 3D video and / or visual depth. Therefore, an OLS that implements multi-view includes two layers (e.g., one layer for each eye). However, when these layers are all broadcast simultaneously, a video decoder can use this inference to infer that each OLS includes only one layer. This may lead to errors because the decoder may only display one layer of the multi-view or may not be able to continue to display either of the two layers. Therefore, when all layers are broadcast simultaneously, the inference that each OLS includes a single layer may prevent multi-view applications from being rendered correctly at the decoder.

[0027] This example includes a mechanism that enables the video coding system to correctly decode multi-view video when all layers in the video are simulcast and inter-layer prediction is not used. When none of the layers use inter-layer prediction (e.g., all layers are simulcast), the vps_all_independent_layers_flag can be included in the VPS in the codestream and set to 1. When this flag is set to 1, each_layer_is_an_ols_flag is indicated in the VPS. each_layer_is_an_ols_flag can be set to indicate whether each OLS includes a single layer or whether at least one OLS includes more than one layer (e.g., to support multi-view). Therefore, vps_all_independent_layers_flag and each_layer_is_an_ols_flag can be used to support multi-view applications. In addition, when this occurs, ols_mode_idc can be set to 2 in the VPS. This will result in the number of OLSs and the layers associated with the OLS being explicitly indicated. The decoder can then use this information to correctly decode the OLSs that include multi-view video. This approach supports error correction while maintaining decoding efficiency. Thus, the disclosed mechanism increases the functionality of the encoder and / or decoder. Furthermore, the disclosed mechanism can reduce the size of the bitstream and, therefore, reduce the use of processor, memory, and / or network resources at the encoder and decoder.

[0028] Optionally, according to any of the above aspects, in another implementation of the aspect, the decoder is further used to execute the method according to any of the above aspects.

[0029] In one embodiment, the present invention includes an encoder comprising: an encoding module for encoding a code stream comprising one or more layers of a coded image; encoding a VPS into the code stream, wherein, when all layers specified by the VPS are independently encoded without requiring inter-layer prediction, the VPS includes each_layer_is_an_ols_flag, wherein the each_layer_is_an_ols_flag indicates whether each OLS includes only one layer; and a storage module for storing the code stream for sending to a decoder.

[0030] Image layers can be used to support adaptability. For example, a video can be encoded into multiple layers. A layer can be encoded without reference to other layers. Such a layer is called a simulcast layer. Accordingly, a simulcast layer can be decoded without reference to other layers. For another example, a layer can be encoded using inter-layer prediction. This allows encoding the current layer by including only the difference between the current layer and a reference layer. Layers can be organized into open layers (OLSs). An OLS is a set of layers consisting of at least one output layer and any layers that support decoding of the output layer. In a specific example, the first OLS may include a base layer, while the second OLS may include the base layer and enhancement layers with added features. In one example, the first OLS can be sent to the decoder to enable decoding of the video at a base resolution, or the second OLS can be transmitted to enable decoding of the video at a higher enhanced resolution. This allows the video to be scaled according to user requests. In some cases, adaptability is not used and each layer is encoded as a simulcast layer. Some systems infer that if all layers are simulcast, each OLS should include a single layer (because no reference layer is used). This inference improves decoding efficiency because the indication can be omitted from the encoded bitstream. However, this inference does not support multi-view. Multi-view is also known as stereo video. In multi-view, two video sequences of the same scene are recorded by spatially offset cameras. These two video sequences are displayed to the user on different lenses of a head-mounted device. Displaying a different spatially offset sequence for each eye in this way can create the impression of 3D video and / or visual depth. Therefore, an OLS that implements multi-view includes two layers (e.g., one layer for each eye). However, when these layers are all broadcast simultaneously, a video decoder can use this inference to infer that each OLS includes only one layer. This may lead to errors because the decoder may only display one layer of the multi-view or may not be able to continue to display either of the two layers. Therefore, when all layers are broadcast simultaneously, the inference that each OLS includes a single layer may prevent multi-view applications from being rendered correctly at the decoder.

[0031] This example includes a mechanism that enables the video coding system to correctly decode multi-view video when all layers in the video are simulcast and inter-layer prediction is not used. When none of the layers use inter-layer prediction (e.g., all layers are simulcast), the vps_all_independent_layers_flag can be included in the VPS in the codestream and set to 1. When this flag is set to 1, each_layer_is_an_ols_flag is indicated in the VPS. each_layer_is_an_ols_flag can be set to indicate whether each OLS includes a single layer or whether at least one OLS includes more than one layer (e.g., to support multi-view). Therefore, vps_all_independent_layers_flag and each_layer_is_an_ols_flag can be used to support multi-view applications. In addition, when this occurs, ols_mode_idc can be set to 2 in the VPS. This will result in the number of OLSs and the layers associated with the OLS being explicitly indicated. The decoder can then use this information to correctly decode the OLSs that include multi-view video. This approach supports error correction while maintaining decoding efficiency. Thus, the disclosed mechanism increases the functionality of the encoder and / or decoder. Furthermore, the disclosed mechanism can reduce the size of the bitstream and, therefore, reduce the use of processor, memory, and / or network resources at the encoder and decoder.

[0032] Optionally, according to any of the above aspects, in another implementation of the aspect, the encoder is further used to execute the method according to any of the above aspects.

[0033] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0034] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] For a more complete understanding of the present invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0036] Figure 1 is a flow chart of an exemplary method of decoding a video signal.

[0037] Figure 2 is a schematic diagram of an exemplary encoding and decoding (codec) system for video coding.

[0038] Figure 3is a schematic diagram of an exemplary video encoder.

[0039] Figure 4 is a schematic diagram of an exemplary video decoder.

[0040] Figure 5 is a schematic diagram of an exemplary multi-layer video sequence for inter-layer prediction.

[0041] Figure 6 is a diagram of an exemplary multi-view sequence including simulcast layers for multi-view.

[0042] Figure 7 is a diagram illustrating an exemplary code stream including an OLS with a simulcast layer for multi-view.

[0043] Figure 8 is a schematic diagram of an exemplary video decoding apparatus.

[0044] Figure 9 is a flow chart of an exemplary method for encoding a video sequence with OLS for simulcast layers in multi-view.

[0045] Figure 10 is a flow chart of an exemplary method of decoding a video sequence including an OLS for simulcast layers in multi-view.

[0046] Figure 11 is a diagram of an exemplary system for coding a video sequence with OLS for simulcast layers in multi-view. DETAILED DESCRIPTION

[0047] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The present invention should in no way be limited to the illustrative embodiments, drawings, and techniques described below, including the exemplary designs and embodiments illustrated and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0048] The following terms are defined as follows, unless used in a contrary context herein. Specifically, the following definitions are intended to more clearly describe the present invention. However, terms may be defined differently in different contexts. Therefore, the following definitions should be considered supplementary information and should not be construed as limiting any other definitions provided herein for these terms.

[0049] A bitstream is a series of bits comprising video data that is compressed for transmission between an encoder and a decoder. An encoder is a device that compresses video data into a bitstream through the encoding process. A decoder is a device that reconstructs video data from the bitstream for display through the decoding process. An image is an array of luminance samples and / or chrominance samples that create a frame or field. For clarity of discussion, the image being encoded or decoded can be referred to as the current image.

[0050] A network abstraction layer (NAL) unit is a syntactic structure consisting of data in the form of a raw byte sequence payload (RBSP) (indicating the data type) interspersed with aliasing prevention bytes as needed. A video coding layer (VCL) NAL unit is a NAL unit coded to include video data, such as coded slices of a picture. A non-VCL NAL unit is a NAL unit that includes non-video data, such as syntax and / or parameters that support decoding the video data, performing conformance checks, or other operations. A layer is a collection of VCL NAL units and associated non-VCL NAL units that share specified characteristics (e.g., common resolution, frame rate, picture size, etc.). VCL NAL units of a layer can share a specific value for the NAL unit header layer identifier (nuh_layer_id). A coded picture is a coded representation of a picture and, within an access unit (AU), includes VCL NAL units with a specific value for the NAL unit header layer identifier (nuh_layer_id) and all coding tree units (CTUs) of the picture. A decoded picture is a picture produced by applying a decoding process to an encoded picture.

[0051] An output layer set (OLS) is a set of layers in which one or more layers are designated as one or more output layers. An output layer is a layer designated for output (e.g., output to a display). The zeroth (0th) OLS is an OLS that includes only the lowest layer (the layer with the lowest layer identifier), and therefore only includes output layers. A video parameter set (VPS) is a data unit that includes parameters related to the entire video. Inter-layer prediction is a mechanism for encoding a current image in a current layer by referring to a reference image in a reference layer, where the current image and the reference image are included in the same AU and the nuh_layer_id of the reference layer is lower than the nuh_layer_id of the current layer.

[0052] The each_layer_is_an_ols_flag is a syntax element that indicates whether each OLS in the codestream consists of a single layer. The OLS mode identifier (ols_mode_idc) is a syntax element that indicates information related to the number of OLSs, the layers of the OLSs, and the output layers in the OLSs. The VPS all independent layers flag (vps_all_independent_layers_flag) is a syntax element that indicates whether any layer in the codestream is coded using inter-layer prediction. The VPS maximum number of layers minus 1 (vps_max_layers_minus1) is a syntax element that indicates the number of layers specified by the VPS, and therefore the maximum number of layers allowed in the corresponding coded video sequence. The number of output layer sets minus 1 (num_output_layer_sets_minus1) is a syntax element that indicates the total number of OLSs specified by the VPS.

[0053] This document uses the following abbreviations: coding tree block (CTB), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), joint video experts team (JVET), motion-constrained tile set (MCTS), maximum transfer unit (MTU), network abstraction layer (NAL), output layer set (OLS), picture order count (POC), raw byte sequence payload (RBSP), sequence parameter set (SPS), video parameter set (VPS), versatile video coding (VVC), and working draft (WD).

[0054] Many video compression techniques can be used to reduce the size of video files while minimizing data loss. For example, video compression techniques can include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in a video sequence. For block-based video decoding, a video slice (e.g., a video picture or a portion of a video picture) can be partitioned into video blocks, which can also be referred to as treeblocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in intra-coded (I) slices of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in inter-coded unidirectionally predicted (P) or bidirectionally predicted (B) slices of a picture can be decoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture, or can be encoded using temporal prediction with respect to reference samples in other reference pictures. A picture (picture / image) can be referred to as a frame, and a reference picture can be referred to as a reference frame. Spatial prediction or temporal prediction produces a prediction block representing the image block. The residual data represents the pixel differences between the original image block and the prediction block. Thus, an inter-coded block is encoded based on a motion vector pointing to a block of reference samples constituting the prediction block and the residual data representing the difference between the coded block and the prediction block. An intra-coded block is encoded based on an intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain. These produce residual transform coefficients that can be quantized. The quantized transform coefficients can initially be arranged in a two-dimensional array. The quantized transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients. Entropy coding can be used to achieve further compression. These video compression techniques are discussed in more detail below.

[0055] To ensure that the encoded video can be accurately decoded, the video is encoded and decoded according to the corresponding video coding standards. Video coding standards include International Telecommunication Union (ITU) standardization sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes extended versions such as scalable video coding (SVC), multiview video coding (MVC), multiview video coding plus depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extended versions such as scalable HEVC (SHVC), multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The Joint Video Experts Team (JVET) of ITU-T and ISO / IEC has begun developing a video coding standard called universal video coding (VVC). VVC is included in WD, which includes JVET-O2001-v14.

[0056] Image layers can be used to support adaptability. For example, a video can be encoded into multiple layers. A layer can be encoded without reference to other layers. Such a layer is called a simulcast layer. Accordingly, a simulcast layer can be decoded without reference to other layers. For another example, a layer can be encoded using inter-layer prediction. This allows encoding the current layer by including only the difference between the current layer and a reference layer. For example, the current layer and the reference layer can include the same video sequence encoded by varying characteristics such as the signal-to-noise ratio (SNR), image size, and frame rate. Layers can be organized into output layer sets (OLSs). An OLS is a set of layers consisting of at least one output layer and any layers that support decoding of the output layer. In a specific example, a first OLS can include a base layer, while a second OLS can include the base layer and enhancement layers with increased characteristics. In an example where the characteristic is image resolution, the first OLS can be transmitted to the decoder to enable decoding of the video at the base resolution, or the second OLS can be transmitted to enable decoding of the video at a higher enhanced resolution. Thus, the video can be scaled according to user requests.

[0057] In some cases, adaptability is not used and each layer is encoded as a simulcast layer. Some systems infer that if all layers are simulcast, each OLS should include a single layer (since no reference layer is used). This inference improves decoding efficiency because the indication can be omitted from the encoded bitstream. However, this inference does not support multiview. Multiview is also known as stereoscopic video. In multiview, two video sequences of the same scene are recorded by spatially offset cameras. These two video sequences are displayed to the user through different lenses of a head-mounted device. Displaying different spatially offset sequences for each eye in this way can create three-dimensional (3D) video and / or the impression of visual depth. Therefore, an OLS implementing multiview includes two layers (e.g., one layer for each eye). However, when these layers are simulcast, a video decoder can use this inference to infer that each OLS includes only a single layer. This can lead to errors, as the decoder may display only one layer of the multiview or may not be able to continue displaying either layer. Therefore, when all layers are simulcast, the inference that each OLS includes a single layer may prevent multiview applications from rendering correctly at the decoder.

[0058] This document discloses a mechanism that enables a video decoding system to correctly decode multi-view video when all layers in the video are simulcast and inter-layer prediction is not used. When none of the layers use inter-layer prediction (e.g., all layers are simulcast), the VPS all independent layer flag (vps_all_independent_layers_flag) can be included in the VPS in the codestream and set to 1. When this flag is set to 1, the flag (each_layer_is_an_ols_flag) for each layer is an output layer set (OLS) in the VPS. The each_layer_is_an_ols_flag can be set to indicate whether each OLS includes a single layer or whether at least one OLS includes more than one layer (e.g., to support multi-view). Therefore, the vps_all_independent_layers_flag and each_layer_is_an_ols_flag can be used to support multi-view applications. In addition, when this occurs, the OLS mode identification code (ols_mode_idc) can be set to 2 in the VPS. This results in the number of OLSs and the layers associated with each OLS being explicitly indicated. The decoder can then use this information to correctly decode the OLSs, including multi-view video. This approach supports error correction while maintaining decoding efficiency. Thus, the disclosed mechanism increases the functionality of the encoder and / or decoder. Furthermore, the disclosed mechanism can reduce the bitstream size and, therefore, reduce the use of processor, memory, and / or network resources at the encoder and decoder.

[0059] Figure 1 The flowchart of an exemplary method 100 for decoding a video signal is shown. Specifically, the video signal is encoded at the encoder. The encoding process compresses the video signal using various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be sent to users while reducing associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process typically corresponds to the encoding process, allowing the decoder to consistently reconstruct the video signal.

[0060] In step 101, a video signal is input into an encoder. For example, the video signal can be an uncompressed video file stored in a memory. For another example, the video file can be captured by a video capture device (e.g., a camera) and encoded to support live streaming of the video. The video file can include an audio component and a video component. The video component includes a series of image frames. When these image frames are viewed in sequence, they give the visual effect of motion. These frames include pixels represented by light, referred to herein as luminance components (or luminance samples), and pixels represented by color, referred to as chrominance components (or chrominance samples). In some examples, these frames can also include depth values ​​to support three-dimensional viewing.

[0061] In step 103, the video is segmented into blocks. Segmentation involves subdividing the pixels in each frame into square blocks and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC) (also known as H.265 and MPEG-H Part 2), frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64×64 pixels). These CTUs include luma samples and chroma samples. A coding tree can be used to divide the CTU into blocks, and then repeatedly subdivide these blocks until a configuration that supports further encoding is obtained. For example, the luma component of a frame can be subdivided until each block includes relatively uniform luma values. Additionally, the chroma component of a frame can be subdivided until each block includes relatively uniform color values. Thus, the segmentation mechanism varies depending on the content of the video frame.

[0062] In step 105, the image blocks segmented in step 103 are compressed using various compression mechanisms. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction is designed to take advantage of the fact that objects in a typical scene often appear in consecutive frames. Therefore, blocks describing objects in a reference frame do not need to be repeated in adjacent frames. Specifically, an object (e.g., a table) can remain in a fixed position across multiple frames. Therefore, the table is described once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects across multiple frames. In addition, moving objects can be represented across multiple frames due to, for example, object motion or camera movement. In one specific example, a video may show a car moving across the screen across multiple frames. Motion vectors can be used to describe this motion. A motion vector is a two-dimensional vector that provides the offset between the coordinates of an object in one frame and the coordinates of the object in a reference frame. Therefore, inter-frame prediction can encode image blocks in the current frame as a set of motion vectors, representing the offset between the image blocks in the current frame and the corresponding blocks in the reference frame.

[0063] Intra-frame prediction encodes blocks in a common frame. Intra-frame prediction exploits the fact that luma and chroma components tend to cluster within a frame. For example, a patch of green in one part of a tree is often adjacent to similar patches of green. Intra-frame prediction uses a variety of directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. These directional modes indicate that samples in the current block are similar or identical to samples in neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks in a row or column (e.g., a plane) can be interpolated based on neighboring blocks along the edge of that row. Planar mode effectively represents a smooth transition of light or color across rows or columns by using a relatively constant slope of varying values. DC mode is used for boundary smoothing and indicates that a block is similar or identical to the average of samples from all neighboring blocks relative to the angular direction of the directional prediction mode. Therefore, intra-frame prediction blocks can represent image blocks as various relative prediction mode values ​​rather than actual values. Furthermore, inter-frame prediction blocks can represent image blocks as motion vector values ​​rather than actual values. In either case, the prediction block may not accurately represent the image block in some cases. All differences are stored in residual blocks. Transformations can be applied to residual blocks to further compress the file.

[0064] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above may produce blocky images on the decoder side. In addition, a block-based prediction scheme can encode a block and then reconstruct the encoded block for subsequent use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to the block / frame. These filters reduce blocking artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference block, making it less likely that the artifacts will produce other artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0065] Once the video signal has been segmented, compressed, and filtered, the resulting data is encoded into a bitstream in step 109. The bitstream includes the data described above as well as any indicative data needed to support proper video signal reconstruction at the decoder. For example, this data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in memory so that it can be sent to the decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 can be performed continuously and / or simultaneously across multiple frames and blocks. Figure 1The shown order is presented for clarity and ease of description and is not intended to limit the video coding process to a specific order.

[0066] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the bitstream to determine the frame segmentation. This segmentation should match the results of the block segmentation in step 103. The entropy encoding / decoding used in step 111 is described below. The encoder makes many choices during the compression process, for example, selecting a block segmentation scheme from several possible options based on the spatial positioning of values ​​in one or more input images. Indicating the exact choice may use a large number of bits (bins). As used herein, a "bit" is a variable binary value (e.g., a bit value that may vary depending on the content). Entropy encoding causes the encoder to discard any options that are clearly unsuitable for a particular situation, leaving a set of available options. A codeword is then assigned to each available option. The length of the codeword depends on the number of available options (e.g., one binary symbol for two options, two binary symbols for three to four options, and so on). The encoder then encodes the codeword for the selected option. This approach reduces the codeword size because the codeword is as large as expected, uniquely indicating a selection from a small subset of permissible options rather than a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of permissible options in a similar manner to the encoder. By determining the set of permissible options, the decoder can read the codeword and determine the selection made by the encoder.

[0067] In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding prediction block to reconstruct the image block based on the segmentation. The prediction block may include the intra-frame prediction block and the inter-frame prediction block generated by the encoder in step 105. Next, the reconstructed image block is placed in the frame of the reconstructed video signal based on the segmentation data determined in step 111. The syntax used for step 113 can also be indicated in the bitstream through the entropy coding described above.

[0068] In step 115, filtering is performed on the frame of the reconstructed video signal in a manner similar to step 107 at the encoder. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter can be applied to the frame to remove blocking artifacts. Once the frame has been filtered, in step 117, the video signal can be output to a display for viewing by an end user.

[0069] Figure 21 is a schematic diagram of an exemplary encoding and decoding (CODEC) system 200 for video decoding. Specifically, CODEC system 200 provides functionality that supports implementation of method 100. CODEC system 200 is broadly used to describe components used on both the encoder and decoder sides. CODEC system 200 receives a video signal and segments the video signal, as described in conjunction with steps 101 and 103 of method 100, to obtain segmented video signals 201. CODEC system 200, when acting as an encoder, then compresses segmented video signal 201 into an encoded bitstream, as described in conjunction with steps 105, 107, and 109 of method 100. CODEC system 200, when acting as a decoder, generates an output video signal from the bitstream, as described in conjunction with steps 111, 113, 115, and 117 of method 100. The codec system 200 includes a general decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. Figure 2 In FIG, black lines represent the movement of data to be encoded / decoded, while dashed lines represent the movement of control data that controls the operation of other components. Components in codec system 200 may all reside in an encoder. A decoder may include a subset of the components in codec system 200. For example, a decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.

[0070] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide the pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. These blocks can be called nodes on the coding tree. A larger parent node is divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks may be included in a coding unit (CU). For example, a CU may be a sub-part of a CTU, including a luminance block, one or more red difference chrominance (Cr) blocks and one or more blue difference chrominance (Cb) blocks and the corresponding syntax instructions of the CU. The partitioning mode may include a binary tree (BT), a triple tree (TT) and a quad tree (QT), which are used to partition the node into 2, 3 or 4 child nodes of different shapes, respectively, according to the partitioning mode used. The segmented video signal 201 is forwarded to the general decoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.

[0071] The universal decoder control component 211 is used to make decisions related to encoding images in a video sequence into a bitstream based on application constraints. For example, the universal decoder control component 211 manages the optimization of bitrate / bitstream size relative to reconstruction quality. These decisions can be made based on storage space / bandwidth availability and image resolution requirements. The universal decoder control component 211 also manages buffer utilization based on transmission speed to mitigate buffer underruns and overruns. To address these issues, the universal decoder control component 211 manages segmentation, prediction, and filtering performed by other components. For example, the universal decoder control component 211 can dynamically increase compression complexity to improve resolution and increase bandwidth utilization, or decrease compression complexity to reduce resolution and bandwidth utilization. Thus, the universal decoder control component 211 controls other components in the codec system 200 to balance video signal reconstruction quality with bitrate. The universal decoder control component 211 generates control data used to control the operation of other components. The control data is also forwarded to the header format and CABAC component 231 for encoding into the codestream, thereby instructing the decoder on the parameters to be used for decoding.

[0072] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices of the segmented video signal 201 may be divided into a plurality of video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 may perform multiple decoding passes to select an appropriate decoding mode for each block of video data, etc.

[0073] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but for conceptual purposes, they are described separately. Motion estimation, performed by the motion estimation component 221, is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, a motion vector can represent the displacement of a coded object relative to a prediction block. A prediction block is a block that is found to closely match the block to be coded in terms of pixel difference. A prediction block can also be referred to as a reference block. This pixel difference can be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. HEVC uses several coding objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be divided into CTBs, which are then divided into CBs, which are included in a CU. A CU can be encoded as a prediction unit (PU) that includes prediction data and / or a transform unit (TU) that includes the transform residual data of the CU. The motion estimation component 221 uses rate-distortion analysis as part of the rate-distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and may select a reference block, motion vector, etc. having an optimal rate-distortion characteristic. The optimal rate-distortion characteristic balances the quality of video reconstruction (e.g., the amount of data lost due to compression) and decoding efficiency (e.g., the size of the final encoded image).

[0074] In some examples, the codec system 200 can calculate values ​​for sub-integer pixel positions of a reference image stored in the decoded picture buffer component 223. For example, the video codec system 200 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Accordingly, the motion estimation component 221 can perform motion search relative to integer-pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of prediction blocks of the reference image. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header format and CABAC component 231 for encoding and as motion data to the motion compensation component 219.

[0075] The motion compensation performed by the motion compensation component 219 may involve obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. Likewise, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the prediction block to which the motion vector points. The pixel values ​​of the prediction block are then subtracted from the pixel values ​​of the current video block being decoded to obtain the pixel differences, thereby forming a residual video block. Typically, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 applies the motion vector calculated based on the luma component to the chroma and luma components. The prediction block and the residual block are forwarded to the transform, scaling, and quantization component 213.

[0076] The segmented video signal 201 is also sent to an intra-frame estimation component 215 and an intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but for conceptual purposes, they are described separately. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction on the current block relative to blocks in the current frame, replacing the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines an intra-frame prediction mode to encode the current block. In some examples, the intra-frame estimation component 215 selects an appropriate intra-frame prediction mode from a plurality of tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then forwarded to the header format and CABAC component 231 for encoding.

[0077] For example, the intra-frame estimation component 215 performs a rate-distortion analysis on various tested intra-frame prediction modes to calculate rate-distortion values ​​and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original unencoded block that is encoded to generate the encoded block, as well as determines the code rate (e.g., number of bits) used to generate the encoded block. The intra-frame estimation component 215 calculates a ratio based on the distortion and rate of the various encoded blocks to determine the intra-frame prediction mode that exhibits the best rate-distortion value for the block. In addition, the intra-frame estimation component 215 can be used to decode the depth block of the depth image using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0078] The intra prediction component 217, when implemented on an encoder, can generate a residual block from the prediction block based on the selected intra prediction mode determined by the intra estimation component 215, or, when implemented on a decoder, can read the residual block from the bitstream. The residual block contains the difference between the predicted block and the original block, represented as a matrix. The residual block is then forwarded to the transform, scaling, and quantization component 213. The intra estimation component 215 and the intra prediction component 217 can operate on both luma and chroma components.

[0079] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 applies a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform to the residual block, thereby generating a video block comprising residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform converts the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also used to scale the transformed residual information based on frequency, for example. This scaling involves applying a scaling factor to the residual information to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can be modified by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan on the matrix comprising the quantized transform coefficients. The quantized transform coefficients are forwarded to the header format and CABAC component 231 for encoding into the codestream.

[0080] The scaling and inverse transform component 229 performs the opposite operations to the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, inverse transform, and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, to be subsequently used as a reference block. This reference block can become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can calculate a reference block by adding the residual block back to the corresponding prediction block to be used for motion estimation of subsequent blocks / frames. Filters are applied to the reconstructed reference block to reduce artifacts generated during scaling, quantization, and transforming. These artifacts can make the prediction inaccurate (and generate additional artifacts) when predicting subsequent blocks.

[0081] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block from the scaling and inverse transform component 229 can be combined with the corresponding prediction block from the intra prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block instead. Figure 2 Like other components in the , the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and can be implemented together, but are described separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to specific spatial regions, and these filters include multiple parameters to adjust how these filters are used. The filter control analysis component 227 analyzes the reconstructed reference block to determine where these filters need to be used and sets the corresponding parameters. This data is forwarded as filter control data to the header format and CABAC component 231 for encoding. The in-loop filter component 225 applies these filters based on the filter control data. These filters can include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. These filters can be applied in the spatial / pixel domain (e.g., for reconstructed pixel blocks) or in the frequency domain, according to the examples.

[0082] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for subsequent use in motion estimation, as described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and forwards them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0083] The header format and CABAC component 231 receives data from various components of the codec system 200 and encodes this data into an encoded bitstream to be sent to the decoder. Specifically, the header format and CABAC component 231 generates various headers to encode control data (e.g., general control data and filter control data). In addition, prediction data (including intra-frame prediction data and motion data) and residual data in the form of quantized transform coefficient data are encoded into the bitstream. The final bitstream includes all the information required for the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword mapping table), definitions of the coding contexts of various blocks, an indication of the most likely intra-frame prediction mode, an indication of segmentation information, etc. This data can be encoded using entropy coding. For example, this information can be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding, the decoded bitstream can be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0084] Figure 3 is a block diagram of an exemplary video encoder 300. Video encoder 300 can be used to implement the encoding functionality of codec system 200 and / or implement steps 101, 103, 105, 107, and / or 109 of method 100. Encoder 300 segments an input video signal to obtain segmented video signals 301, which are substantially similar to segmented video signal 201. Segmented video signals 301 are then compressed by components within encoder 300 and encoded into a bitstream.

[0085] Specifically, the segmented video signal 301 is forwarded to an intra-frame prediction component 317 for intra-frame prediction. Intra-frame prediction component 317 can be substantially similar to intra-frame estimation component 215 and intra-frame prediction component 217. The segmented video signal 301 is also forwarded to a motion compensation component 321 for inter-frame prediction based on reference blocks in a decoded picture buffer component 323. Motion compensation component 321 can be substantially similar to motion estimation component 221 and motion compensation component 219. The prediction blocks and residual blocks from intra-frame prediction component 317 and motion compensation component 321 are forwarded to a transform and quantization component 313 for transform and quantization of the residual blocks. Transform and quantization component 313 can be substantially similar to transform, scaling, and quantization component 213. The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are forwarded to an entropy coding component 331 for encoding in the bitstream. Entropy coding component 331 can be substantially similar to header format and CABAC component 231.

[0086] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and dequantization component 329 for reconstruction as a reference block for use by the motion compensation component 321. The inverse transform and dequantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to an example, an in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 can include multiple filters, as described in conjunction with the in-loop filter component 225. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block by the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.

[0087] Figure 4 is a block diagram of an exemplary video decoder 400. Video decoder 400 can be used to implement the decoding function of codec system 200 and / or implement steps 111, 113, 115, and / or 117 of operating method 100. For example, decoder 400 receives a bitstream from encoder 300 and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0088] The bitstream is received by entropy decoding component 433. Entropy decoding component 433 is used to perform an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE decoding, or other entropy decoding techniques. For example, entropy decoding component 433 can use header information to provide context for parsing additional data encoded as codewords in the bitstream. The decoded information includes any information required to decode the video signal, such as general control data, filter control data, segmentation information, motion data, prediction data, and quantized transform coefficients in the residual block. The quantized transform coefficients are forwarded to inverse transform and dequantization component 429 for reconstruction into the residual block. Inverse transform and dequantization component 429 can be similar to inverse transform and dequantization component 329.

[0089] The reconstructed residual block and / or prediction block is forwarded to the intra-frame prediction component 417 for reconstruction into an image block based on the intra-frame prediction operation. The intra-frame prediction component 417 can be similar to the intra-frame estimation component 215 and the intra-frame prediction component 217. Specifically, the intra-frame prediction component 417 uses the prediction mode to locate the reference block in the frame and adds the residual block to the result to reconstruct the intra-frame predicted image block. The reconstructed intra-frame predicted image block and / or residual block and the corresponding inter-frame prediction data are forwarded to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or prediction block. This information is stored in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter-frame prediction. Motion compensation component 421 can be substantially similar to motion estimation component 221 and / or motion compensation component 219. Specifically, motion compensation component 421 uses the motion vector of the reference block to generate a prediction block and applies the residual block to the result to reconstruct the image block. The resulting reconstructed block can also be forwarded to decoded image buffer component 423 via in-loop filter component 425. Decoded image buffer component 423 continues to store other reconstructed image blocks. These reconstructed image blocks can be reconstructed into frames using segmentation information. These frames can also be arranged in a sequence. This sequence is output to a display as a reconstructed output video signal.

[0090] Figure 5FIG2 is a diagram illustrating an exemplary multi-layer video sequence 500 for inter-layer prediction 521. For example, the multi-layer video sequence 500 can be encoded by an encoder (e.g., codec system 200 and / or encoder 300) and decoded by a decoder (e.g., codec system 200 and / or decoder 400) according to method 100. The multi-layer video sequence 500 is included to illustrate an exemplary application of layers in an encoded video sequence. The multi-layer video sequence 500 is any video sequence that uses multiple layers (e.g., layer N 531 and layer N+1 532).

[0091] In one example, multi-layer video sequence 500 can use inter-layer prediction 521. Inter-layer prediction 521 is applied between pictures 511, 512, 513, and 514 and pictures 515, 516, 517, and 518 in different layers. In the example shown, pictures 511, 512, 513, and 514 are part of layer N+1 532, while pictures 515, 516, 517, and 518 are part of layer N 531. A layer (e.g., layer N 531 and / or layer N+1 532) is a group of pictures that are all associated with similar characteristic values ​​(e.g., similar size, quality, resolution, signal-to-noise ratio, capabilities, etc.). A layer can be formally defined as a collection of VCL NAL units and associated non-VCL NAL units that share the same nuh_layer_id. A VCL NAL unit is a NAL unit coded to include video data, such as a coded slice of a picture. A non-VCL NAL unit is a NAL unit that includes non-video data, such as syntax and / or parameters to support decoding video data, performing conformance checking, or other operations.

[0092] In the exemplary illustration, layer N+1 532 is associated with a larger image size than layer N 531. Thus, in this example, images 511, 512, 513, and 514 in layer N+1 532 are larger (e.g., larger in height and width, resulting in more samples) than images 515, 516, 517, and 518 in layer N 531. However, these images may be distributed between layer N+1 532 and layer N 531 based on other characteristics. While only two layers are shown: layer N+1 532 and layer N 531, a set of images can be divided into any number of layers based on associated characteristics. Layers N+1 532 and N 531 can also be represented by layer identifiers (IDs). A layer ID is a data item associated with an image and indicates that the image is part of the indicated layer. Therefore, each of images 511 to 518 can be associated with a corresponding layer ID to indicate which layer, layer N+1 532 or layer N 531, contains the corresponding image. For example, the layer ID may include a NAL unit header layer identifier (nuh_layer_id), which is a syntax element that indicates an identifier of a layer that includes a NAL unit (e.g., including slices of pictures and / or parameters in the layer). Layers associated with lower quality / bitstream size (e.g., layer N 531) are typically assigned a lower layer ID and are referred to as lower layers. In addition, layers associated with higher quality / bitstream size (e.g., layer N+1 532) are typically assigned a higher layer ID and are referred to as higher layers.

[0093] Images 511 to 518 in different layers 531 and 532 are displayed alternately. In this specific example, if a smaller image is needed, the decoder can decode and display image 515 at the current display time, or if a larger image is needed, the decoder can decode and display image 511 at the current display time. Therefore, images 511 to 514 in the upper layer N+1 532 and corresponding images 515 to 518 in the lower layer N 531 include substantially the same image data (although the image sizes are different). Specifically, images 511 and 515 include substantially the same image data, images 512 and 516 include substantially the same image data, and so on.

[0094] Pictures 511 to 518 can be decoded with reference to other pictures 511 to 518 in the same layer N 531 or N+1 532. Decoding a picture with reference to another picture in the same layer results in an inter-frame prediction 523. Inter-frame prediction 523 is indicated by a solid arrow. For example, picture 513 can be decoded using inter-frame prediction 523 with reference to one or two pictures 511, 512, and / or 514 in layer N+1 532, where unidirectional inter-frame prediction uses one picture as a reference and / or bidirectional inter-frame prediction uses two pictures as a reference. Furthermore, picture 517 can be decoded using inter-frame prediction 523 with reference to one or two pictures 515, 516, and / or 518 in layer N 531, where unidirectional inter-frame prediction uses one picture as a reference and / or bidirectional inter-frame prediction uses two pictures as a reference. When an image is used as a reference for another image in the same layer when performing inter-frame prediction 523, the image may be referred to as a reference image. For example, image 512 may be a reference image used to decode image 513 according to inter-frame prediction 523. In the context of multiple layers, inter-frame prediction 523 may also be referred to as intra-layer prediction. Thus, inter-frame prediction 523 is a mechanism for decoding samples of a current image by referencing samples in a reference image that is different from the current image, where the reference image and the current image are in the same layer.

[0095] Pictures 511 to 518 can also be decoded by referencing other pictures 511 to 518 in different layers. This process is called inter-layer prediction 521 and is indicated by the dashed arrows. Inter-layer prediction 521 is a mechanism for decoding samples of the current picture by referencing samples in a reference picture, where the current picture and the reference picture are in different layers and therefore have different layer IDs. For example, a picture in a lower layer N 531 can be used as a reference picture for decoding a corresponding picture in a higher layer N+1 532. In a specific example, picture 511 can be decoded using reference picture 515 according to inter-layer prediction 521. In this case, picture 515 is used as an inter-layer reference picture. An inter-layer reference picture is a reference picture used for inter-layer prediction 521. In most cases, inter-layer prediction 521 is constrained so that the current picture (e.g., picture 511) can only use one or more inter-layer reference pictures included in the same AU and located in a lower layer, such as picture 515. An AU is a set of pictures associated with a specific output time in a video sequence, so an AU can include up to one picture per layer. When multiple layers (e.g., more than two layers) are available, inter-layer prediction 521 can encode / decode the current picture based on multiple inter-layer reference pictures at a lower level than the current picture.

[0096] A video encoder can use a multi-layer video sequence 500 to encode pictures 511 through 518 using many different combinations and / or permutations of inter-frame prediction 523 and inter-layer prediction 521. For example, picture 515 can be decoded using intra-frame prediction. Then, using picture 515 as a reference picture, pictures 516 through 518 can be decoded using inter-frame prediction 523. Furthermore, using picture 515 as an inter-layer reference picture, picture 511 can be decoded using inter-layer prediction 521. Then, using picture 511 as a reference picture, pictures 512 through 514 can be decoded using inter-frame prediction 523. Thus, reference pictures can serve as both single-layer reference pictures and inter-layer reference pictures for different decoding mechanisms. By decoding the higher-layer N+1 532 picture based on the lower-layer N 531 picture, the higher-layer N+1 532 picture can avoid using intra-frame prediction, which has significantly lower decoding efficiency than inter-frame prediction 523 and inter-layer prediction 521. Therefore, intra prediction, which is less efficient to decode, is limited to the smallest / lowest quality pictures and, therefore, to decoding the smallest amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures may be indicated in entries of one or more reference picture lists included in the reference picture list structure.

[0097] To perform such operations, layers (e.g., layer N 531 and layer N+1 532) may be included in an OLS 525. An OLS 525 is a set of layers, one or more of which are designated as output layers. An output layer is a layer designated for output (e.g., output to a display). For example, layer N 531 may be included only to support inter-layer prediction 521 and may never be output. In this case, layer N+1 532 is decoded based on layer N 531 and output. In this case, the OLS 525 includes layer N+1 532 as an output layer. The OLS 525 may include many different combinations of layers. For example, an output layer in the OLS 525 may be decoded based on inter-layer prediction 521 based on one, two, or more lower layers. Furthermore, the OLS 525 may include more than one output layer. Thus, the OLS 525 may include one or more output layers and any supporting layers required to reconstruct the output layers. The multi-layer video sequence 500 may be coded using many different OLSs 525, where each OLS 525 uses a different combination of layers.

[0098] In a specific example, inter-layer prediction 521 can be used to support adaptability. For example, a video can be coded into a base layer (e.g., layer N 531) and several enhancement layers (e.g., layer N+1 532, layer N+2, layer N+3, etc.), which are coded based on inter-layer prediction 521. A video sequence can be encoded for several adaptable characteristics (e.g., resolution, frame rate, image size, etc.). An OLS 525 can then be created for each allowed characteristic. For example, the OLS 525 for a first resolution can include only layer N 531, the OLS 525 for a second resolution can include layer N 531 and layer N+1 532, the OLS for a third resolution can include layer N 531, layer N+1 532, layer N+2, and so on. In this way, the OLS 525 can be transmitted to enable a decoder to decode any desired version of the multi-layer video sequence 500 based on network conditions, hardware constraints, etc.

[0099] Figure 6 6 is a diagram of an exemplary multi-view sequence 600 including simulcast layers 631, 632, 633, and 634 for use in multi-view. The multi-view sequence 600 is a type of the multi-layer video sequence 500. Thus, for example, according to method 100, the multi-view sequence 600 can be encoded by an encoder (e.g., the codec system 200 and / or the encoder 300) and decoded by a decoder (e.g., the codec system 200 and / or the decoder 400).

[0100] Multi-view video may also be referred to as stereoscopic video. In multi-view, a video sequence is captured simultaneously from multiple camera angles into a single video stream. For example, a pair of spatially offset cameras may be used to capture the video. Each camera captures the video from a different angle. This forms a pair of views of the same subject. The first view may be presented to the user's right eye, and the second view may be presented to the user's left eye. This can be achieved, for example, using a head mounted display (HMD) that includes a left-eye display and a separate right-eye display. The pair of streams showing the same subject from different angles creates the impression of visual depth, thereby creating a 3D viewing experience.

[0101] To implement multi-view, video can be encoded into multiple OLSs, such as OLS 625 and OLS 626, which are similar to OLS 525. Each view is encoded into a layer, such as layers 631, 632, 633, and 634, which can be similar to layer N531. In a specific example, the right-eye view can be encoded into layer 631, and the left-eye view can be encoded into layer 632. Layers 631 and 632 can then be included in OLS 625. OLS 625 can then be sent to a decoder with layers 631 and 632 marked as output layers. The decoder can then decode and display layers 631 and 632. Thus, OLS 625 provides sufficient data to represent multi-view video. Like other types of video, multi-view video can be encoded into several representations to support different display devices, varying network conditions, and so on. Therefore, OLS 626 is substantially similar to OLS 625, but includes video encoded to implement different features. For example, layer 633 may be substantially similar to layer 631, and layer 634 may be substantially similar to layer 632. However, layers 633 and 634 may have different characteristics than layers 631 and 632. In a specific example, layers 633 and 634 may be encoded at a different resolution, frame rate, screen size, etc. than layers 631 and 632. In a specific example, if a first image resolution is required, OLS 625 may be sent to the decoder, and if a second image resolution is required, OLS 626 may be sent to the decoder.

[0102] In some cases, adaptability is not used. Layers that do not use inter-layer prediction are called simulcast layers. Simulcast layers can be fully decoded without referencing other layers. For example, layers 631 to 634 shown in the figure are all simulcast layers because they do not rely on any reference layer. This configuration may cause errors in some video decoding systems.

[0103] For example, some video decoding systems can be configured to infer that when all layers are simulcast, each OLS consists of a single layer. In some cases, this inference is reasonable. For example, when standard video does not use adaptability, the system can assume that each simulcast layer can be displayed without any other layers, and therefore the OLS should only consist of one layer. This inference may prevent multi-view from functioning properly. As shown, OLS 625 and 626 include two layers 631 and 632, and layers 633 and 634, respectively. In this case, the decoder may not be sure which layer to decode and may not decode and display both layers because only one layer is expected.

[0104] The present invention solves this problem by using the each_layer_is_an_ols_flag in the codestream. Specifically, when all layers 631 to 634 are simulcast, as indicated by vps_all_independent_layers_flag, each_layer_is_an_ols_flag is indicated. each_layer_is_an_ols_flag indicates whether each OLS includes a single layer, or whether any OLS (e.g., OLS 625 and 626) includes more than one layer. This enables the multi-view sequence 600 to be correctly decoded. In addition, ols_mode_idc can be set to indicate that the number of OLSs 625 and 626 and information related to layers 631 to 634 should be explicitly indicated (e.g., indicating which layers in layers 631 to 634 are output layers). These flags provide the decoder with sufficient information to correctly decode and display OLSs 625 and / or 626 using multi-view. It should be noted that each_layer_is_an_ols_flag, vps_all_independent_layers_flag, and ols_mode_idc are named according to the nomenclature used in the VVC standard. For consistency and clarity of discussion, these names are included here. However, these syntax elements may be referred to by other names without departing from the scope of the present invention.

[0105] Figure 7 FIG2 is a schematic diagram of an exemplary codestream 700 including an OLS having a simulcast layer for multi-view. For example, the codestream 700 may be generated by the codec system 200 and / or the encoder 300, and decoded by the codec system 200 and / or the decoder 400 according to the method 100. In addition, the codestream 700 may include a decoded multi-layer video sequence 500 and / or a multi-view sequence 600.

[0106] The bitstream 700 includes a VPS 711, one or more sequence parameter sets (SPSs) 713, multiple picture parameter sets (PPSs) 715, multiple slice headers 717, and image data 720. The VPS 711 includes data related to the entire bitstream 700. For example, the VPS 711 may include the OLS, layers, and / or sublayers related to the data used in the bitstream 700. The SPS 713 includes sequence data common to all images in the coded video sequence included in the bitstream 700. For example, each layer may include one or more coded video sequences, and each coded video sequence may reference the SPS 713 to obtain corresponding parameters. Parameters in the SPS 713 may include image size, bit depth, coding tool parameters, bitrate constraints, and so on. It should be noted that while each sequence references the SPS 713, in some examples, a single SPS 713 may include data for multiple sequences. The PPS 715 includes parameters that apply to the entire image. Therefore, each image in the video sequence may reference the PPS 715. It should be noted that while each picture references a PPS 715, in some examples, a single PPS 715 may include data for multiple pictures. For example, multiple similar pictures may be decoded using similar parameters. In this case, a single PPS 715 may include data for these similar pictures. The PPS 715 may indicate the coding tools, quantization parameters, offsets, and the like that may be used for slices in the corresponding picture.

[0107] The slice header 717 includes parameters specific to each slice 727 in the image 725. Therefore, there can be one slice header 717 for each slice 727 in a video sequence. The slice header 717 may include information such as slice type, POC, reference picture list, prediction weights, partition entry point, deblocking filter parameters, etc. It should be noted that in some examples, the codestream 700 may also include a picture header, which is a syntax structure that includes parameters that apply to all slices 727 in a single picture. Therefore, the terms picture header and slice header 717 may be used interchangeably in certain contexts. For example, the slice header 717 and the picture header may share some of the same parameters, depending on whether these parameters are common to all slices 727 in the picture 725.

[0108] Image data 720 includes video data encoded using inter-frame prediction and / or intra-frame prediction, as well as corresponding transformed and quantized residual data. For example, image data 720 may include layers 723 of an image 725. Layers 723 may be organized into OLSs 721. OLSs 721 may be substantially similar to OLSs 525, 625, and / or 626. Specifically, OLSs 721 are a collection of layers 723, with one or more layers 723 designated as one or more output layers. For example, codestream 700 may be decoded to include multiple OLSs 721, where video is decoded at different resolutions, frame rates, image 725 sizes, and so on. Upon request from the decoder, the sub-codestream extraction process may remove all OLSs 721 from the codestream 700 except for the requested OLS 721. The encoder may then send the codestream 700 to the decoder containing only the requested OLSs 721, thereby transmitting only video that meets the requested criteria.

[0109] Layer 723 may be substantially similar to layer N 531, layer N+1 532, and / or layers 631, 632, 633, and / or 634. Layer 723 is typically a collection of coded pictures 725. Layer 723 may be formally defined as a collection of VCL NAL units that, when decoded, share specified characteristics (e.g., common resolution, frame rate, picture size, etc.). Layer 723 also includes associated non-VCL NAL units to support decoding of VCL NAL units. The VCL NAL units of layer 723 may share a specific value of nuh_layer_id. Layer 723 may be a simulcast layer coded without inter-layer prediction, or layer 723 coded according to inter-layer prediction, as described with reference to each of the respective layers. Figure 6 and Figure 5 Discussed.

[0110] Image 725 is an array of luminance samples and / or an array of chrominance samples that create a frame or frame field. For example, image 725 can be a coded image that is output for display or used to support the encoding of one or more other images 725 for output. Image 725 may include a set of VCL NAL units. Image 725 includes one or more slices 727. A slice 727 can be defined as an integer number of complete partitions or an integer number of consecutive complete coding tree unit (CTU) rows (e.g., within a partition) of image 725 that are exclusively included in a single NAL unit (specifically a VCL NAL unit). Slices 727 are further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a set of samples of a predefined size that can be partitioned by a coding tree. A CTB is a subset of a CTU and includes the luminance component or chrominance component of the CTU. The CTU / CTB is further divided into coding blocks according to the coding tree. The coding blocks can then be encoded / decoded according to a prediction mechanism.

[0111] The present invention includes mechanisms that enable a video coding system to correctly decode multi-view video (e.g., multi-view sequence 600) when all layers 723 in the video are simulcast and inter-layer prediction is not used. For example, VPS 711 may include various data to indicate to the decoder that all layers 723 are simulcast and that OLS 721 includes more than one layer 723. vps_all_independent_layers_flag 731 may be included in VPS 711 in codestream 700. vps_all_independent_layers_flag 731 is a syntax element that indicates whether any layer 723 in codestream 700 is encoded using inter-layer prediction. For example, vps_all_independent_layers_flag 731 may be set to 1 when none of the layers 723 use inter-layer prediction and therefore all layers are simulcast. In other examples, vps_all_independent_layers_flag 731 can be set to 0 to indicate that at least one of the layers 723 uses inter-layer prediction. When vps_all_independent_layers_flag 731 is set to 1 to indicate that all layers 723 are simulcast, each_layer_is_an_ols_flag 733 is indicated in the VPS 711. each_layer_is_an_ols_flag 733 is a syntax element that indicates whether each OLS 721 in the codestream 700 includes a single layer 723. For example, in most cases, each OLS 721 may include a single simulcast layer. However, when multi-view video is encoded into the codestream 700, one or more OLSs 721 may include two simulcast layers. Therefore, each_layer_is_an_ols_flag 733 can be set (e.g., to 1) to indicate that each OLS 721 includes a single layer 723, or set (e.g., to 0) to indicate that at least one OLS 721 includes more than one layer 723 to support multi-view. Therefore, vps_all_independent_layers_flag 731 and each_layer_is_an_ols_flag 733 can be used to support multi-view applications.

[0112] In addition, the VPS 711 may include ols_mode_idc 735. ols_mode_idc 735 is a syntax element that indicates information related to the number of OLSs 721, the layers 723 of the OLSs 721, and the output layers in the OLSs 721. An output layer 723 is any layer designated for output by a decoder, rather than being used solely for reference-based decoding. ols_mode_idc 735 can be set to 0 or 1 for decoding other types of video. ols_mode_idc 735 can be set to 2 to support multi-view. For example, ols_mode_idc 735 can be set to 2 when vps_all_independent_layers_flag 731 is set to 1 (indicating simulcast layers) and when each_layer_is_an_ols_flag 733 is set to 0 (indicating that at least one OLS 721 includes more than one layer 723). When ols_mode_idc 735 is set to 2, the number of OLSs 721 and information related to the number of layers 723 and / or output layers included in each OLS 721 are explicitly indicated.

[0113] The VPS 711 may also include vps_max_layers_minus1 737. vps_max_layers_minus1 737 is a syntax element used to indicate the number of layers 723 specified by the VPS 711, and therefore indicates the maximum number of layers 723 allowed in the corresponding coded video sequence in the codestream 700. The VPS 711 may also include num_output_layer_sets_minus1 739. num_output_layer_sets_minus1 739 is a syntax element used to indicate the total number of OLSs 721 specified by the VPS 711. In one example, when ols_mode_idc 735 is set to 2, vps_max_layers_minus1 737 and num_output_layer_sets_minus1 739 may be indicated in the VPS 711. This results in the number of OLSs 721 and the number of layers 723 being indicated when the video includes multiple views. Specifically, when vps_all_independent_layers_flag_731 is set to 1 (indicating simulcast layers) and when each_layer_is_an_ols_flag 733 is set to 0 (indicating at least one OLS_721 includes more than one layer 723), vps_max_layers_minus1 737 and num_output_layer_sets_minus1 739 may be indicated. The decoder can then use this information to correctly decode OLS 721 including multi-view video. This approach supports error correction while maintaining decoding efficiency. Specifically, multi-view is supported. However, when multi-view is not used, the number of OLSs 721 and / or layers 723 can still be inferred and omitted from the codestream 700. Thus, the disclosed mechanism increases the functionality of encoders and / or decoders by enabling such devices to correctly decode multi-view video. Furthermore, the disclosed mechanism can reduce codestream size and, therefore, reduce processor, memory, and / or network resource usage at the encoder and decoder.

[0114] The above information is described in more detail below. Layered video coding is also called adaptive video coding or video coding with adaptability. The adaptability of video coding is typically supported by using multi-layer coding techniques. The multi-layer code stream includes a base layer (BL) and one or more enhancement layers (EL). Examples of adaptability include spatial adaptability, quality adaptability / signal to noise ratio (SNR) adaptability, multi-view adaptability, frame rate adaptability, etc. When using multi-layer decoding technology, an image or a part thereof can be encoded without using a reference image (intra-frame prediction), can be encoded by referring to a reference image in the same layer (inter-frame prediction), and / or can be encoded by referring to a reference image in one or more other layers (inter-layer prediction). The reference image used for inter-layer prediction of the current image is called an inter-layer reference picture (ILRP). Figure 5 An example of multi-layer coding for spatial adaptability is shown, where images in different layers have different resolutions.

[0115] Some video coding families support adaptability in one or more individual profiles based on one or more profiles for single-layer decoding. Scalable video coding (SVC) is a scalable version of advanced video coding (AVC) that supports spatial adaptability, temporal adaptability, and quality adaptability. For SVC, a flag is indicated in each macroblock (MB) in the EL image to indicate whether the EL MB uses collocated blocks in the lower layer for prediction. The prediction based on collocated blocks can include texture, motion vectors, and / or decoding modes. When implementing SVC, the unmodified AVC implementation cannot be directly reused in the SVC implementation design. The SVC EL macroblock syntax and decoding process are different from the AVC syntax and decoding process.

[0116] Scalable HEVC (SHVC) is an extension of HEVC that provides support for spatial adaptability and quality adaptability. Multiview HEVC (MV-HEVC) is an extension of HEVC that provides support for multi-view adaptability. 3D HEVC (3D-HEVC) is an extension of HEVC that provides support for more advanced and efficient 3D video decoding than MV-HEVC. Temporal adaptability may include components for single-layer HEVC encoding and decoding. In the multi-layer extension of HEVC, the decoded pictures used for inter-layer prediction come only from the same AU and are regarded as long-term reference pictures (LTRP). Such pictures are assigned reference indices in one or more reference picture lists together with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is implemented at the prediction unit (PU) level by setting the value of the reference index to refer to one or more inter-layer reference pictures in one or more reference picture lists. When the ILRP has a different spatial resolution than the current picture being encoded or decoded, spatial adaptability resamples the reference picture or part of it. Reference picture resampling can be done at the picture level or at the coding block level.

[0117] VVC also supports layered video coding. A VVC codestream can include multiple layers. These layers can be independent of each other. For example, each layer can be decoded without using inter-layer prediction. In this case, these layers are also called simulcast layers. In some cases, some of these layers are decoded using ILP. Flags in the VPS can indicate whether these layers are simulcast layers or whether certain layers use ILP. When certain layers use ILP, the layer dependencies between layers are also indicated in the VPS. Unlike SHVC and MV-HEVC, VVC may not specify an Open Layer Score (OLS). An OLS consists of a designated layer set, in which one or more layers in the set are designated as output layers. Output layers are layers designated for output in the OLS. In some VVC implementations, when a layer is a simulcast layer, only one layer can be selected for decoding and output. In some VVC implementations, when any layer uses ILP, the entire codestream, including all layers, is designated for decoding. In addition, some layers are designated as output layers. Output layers can be designated as only the highest layer, all layers, or the highest layer plus a set of designated lower layers.

[0118] There are some issues with the above aspects. For example, when the layers are simulcast layers, only one layer can be selected for decoding and output. However, this approach does not support situations where more than one layer can be decoded and output, such as in multi-view applications.

[0119] In general, this disclosure describes methods for supporting operation points with more than one output layer for a simulcast layer. These techniques are based on VVC from ITU-T and ISO / IEC JVET. However, these techniques are also applicable to layered video coding based on other video codec specifications.

[0120] One or more of the above problems can be solved as follows. Specifically, the present invention includes a simple and efficient method for supporting decoding and output of multiple layers of a code stream including a simulcast layer, as described below. The VPS may include an indication of whether each layer is an OLS. When each layer is an OLS, only one layer can be decoded and output. In this case, the number of OLSs is inferred to be equal to the number of layers. In addition, each OLS includes one layer, and the layer is the output layer. Otherwise, the number of OLSs is explicitly indicated. For each OLS, excluding the 0th OLS, the layers included in the OLS can be explicitly indicated. In addition, each layer in each OLS can be inferred to be an output layer. The 0th OLS only includes the lowest layer, i.e., the output layer.

[0121] An exemplary implementation of the above mechanism is as follows: An exemplary video parameter set syntax is as follows.

[0122]

[0123]

[0124] Example video parameter set semantics are as follows. The VPS RBSP shall be available to the decoding process before being referenced, shall be included in at least one access unit with TemporalId equal to 0, or provided through an external mechanism, and the nuh_layer_id of the VPS NAL unit including the VPS RBSP shall be equal to vps_layer_id[0]. All VPS NAL units in a CVS with a particular value of vps_video_parameter_set_id shall have the same content. vps_video_parameter_set_id provides an identifier for the VPS for reference by other syntax elements. vps_max_layers_minus1+1 indicates the maximum allowed number of layers in each CVS referencing the VPS. vps_max_sub_layers_minus1+1 indicates the maximum number of temporal sub-layers that can exist in each CVS referencing the VPS. The value of vps_max_sub_layers_minus1 shall be in the range of 0 to 6, inclusive.

[0125] vps_all_independent_layers_flag can be set to 1 to indicate that all layers in the CVS are independently coded without using inter-layer prediction. vps_all_independent_layers_flag can be set to 0 to indicate that one or more layers in the CVS can use inter-layer prediction. When vps_all_independent_layers_flag is not present, the value of vps_all_independent_layers_flag is inferred to be equal to 1. When vps_all_independent_layers_flag is equal to 1, the value of vps_independent_layer_flag[i] is inferred to be equal to 1. When vps_all_independent_layers_flag is equal to 0, the value of vps_independent_layer_flag[0] is inferred to be equal to 1. vps_layer_id[i] represents the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​of m and n, when m is less than n, the value of vps_layer_id[m] shall be less than vps_layer_id[n]. vps_independent_layer_flag[i] can be set to 1 to indicate that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] can be set to 0 to indicate that the layer with index i can use inter-layer prediction and vps_layer_dependency_flag[i] is present in the VPS. When vps_independent_layer_flag[i] is not present, the value of vps_independent_layer_flag[i] is inferred to be equal to 1.

[0126] vps_direct_dependency_flag[i][j] may be set to 0 to indicate that the layer with index j is not a direct reference layer for the layer with index i. vps_direct_dependency_flag[i][j] may be set to 1 to indicate that the layer with index j is a direct reference layer for the layer with index i. When vps_direct_dependency_flag[i][j] is not present for i and j in the range of 0 to vps_max_layers_minus1, inclusive, vps_direct_dependency_flag[i][j] is inferred to be equal to 0. The variable DirectDependentLayerIdx[i][j], which represents the jth directly dependent layer of layer i, is derived as follows:

[0127]

[0128] The variable GeneralLayerIdx[i] representing the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i] is derived as follows:

[0129] for(i=0;i<=vps_max_layers_minus1;i++)

[0130] Genera1LayerIdx[vps_layer_id[i]]=i

[0131] each_layer_is_an_ols_flag can be set to 1 to indicate that each output layer set contains only one layer, and that each layer in the codestream is itself an output layer set, in which the single layer contained therein is the only output layer. each_layer_is_an_ols_flag can be set to 0 to indicate that an output layer set can contain more than one layer. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 1. Otherwise, when vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 0.

[0132] ols_mode_idc can be set to 0 to indicate that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS, only the highest layer in the OLS is output. ols_mode_idc can be set to 1 to indicate that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS, all layers in the OLS are output. ols_mode_idc can be set to 2 to indicate that the total number of OLSs specified by the VPS is explicitly indicated, and for each OLS, the highest layer in the OLS and the set of explicitly indicated lower layers are output. ols_mode_idc values ​​should be in the range of 0 to 2, inclusive. The value 3 of ols_mode_idc is reserved. When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of ols_mode_idc is inferred to be equal to 2.

[0133] When ols_mode_idc is equal to 2, num_output_layer_sets_minus1+1 represents the total number of OLSs specified by the VPS. The variable TotalNumOlss representing the total number of OLSs specified by the VPS is derived as follows:

[0134]

[0135]

[0136] When ols_mode_idc is equal to 2, layer_included_flag[i][i] indicates whether the i-th layer (e.g., the layer with nuh_layer_id equal to vps_layer_id[j]) is included in the i-th OLS. layer_included_flag[i][j] can be set to 1 to indicate that the j-th layer is included in the i-th OLS. layer_included_flag[i][j] can be set to 0 to indicate that the j-th layer is not included in the i-th OLS.

[0137] The variable NumLayersInOls[i] representing the number of layers in the i-th OLS and the variable LayerIdInOls[i][j] representing the nuh_layer_id value of the j-th layer in the i-th OLS can be derived as follows:

[0138]

[0139] The variable OlsLayeIdx[i][j] representing the OLS layer index of the layer whose nuh_layer_id is equal to LayerIdInOls[i][j] can be derived as follows:

[0140] for(i=0,i<TotalNumOlss;i++)

[0141] forj=0;j<NumLayersInOls[i];j++)

[0142] OlsLayeIdx[i][LayerIdInOls[i][j]]=j

[0143] The lowest layer in each OLS shall be an independent layer. In other words, for each i in the range of 0 to TotalNumOlss-1 (inclusive), the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] shall be equal to 1. The highest layer (e.g., the layer with nuh_layer_id equal to vps_layer_id[vps_max_layers_minus1]) shall be included in at least one OLS specified by the VPS. In other words, for at least one i in the range of 0 to TotalNumOlss-1 (inclusive), the value of LayerIdInOls[i][NumLayersInOls[i]-1] shall be equal to vps_layer_id[vps_max_layers_minus1].

[0144] When ols_mode_idc is equal to 2, vps_output_layer_flag[i][j] indicates whether the j-th layer in the i-th OLS is output. vps_output_layer_flag[i] can be set to 1 to indicate that the j-th layer in the i-th OLS is output. vps_output_layer_flag[i] can be set to 0 to indicate that the j-th layer in the i-th OLS is not output. When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of vps_output_layer_flag[i] can be inferred to be equal to 1.

[0145] The value of the variable OutputLayerFlag[i][j] is 1, which means the jth layer in the i-th OLS is output, and the value is 0, which means the jth layer in the i-th OLS is not output. This variable can be derived as follows:

[0146]

[0147] Any layer in the OLS shall be an output layer of the OLS or a reference layer (directly or indirectly) of an output layer of the OLS. The 0th OLS includes only the lowest layer (e.g., the layer with nuh_layer_id equal to vps_layer_id[0]), and for the 0th OLS, the only layer included is output. The vps_constraint_info_present_flag can be set to 1 to indicate that the general_constraint_info() syntax structure is present in the VPS. The vps_constraint_info_present_flag can be set to 0 to indicate that the general_constraint_info() syntax structure is not present in the VPS. In conforming codestreams, vps_reserved_zero_7bits shall be equal to 0. Other values ​​of vps_reserved_zero_7bits are reserved. The decoder shall ignore the value of vps_reserved_zero_7bits.

[0148] The general_hrd_params_present_flag can be set to 1 to indicate that the syntax elements num_units_in_tick and time_scale and the syntax structure general_hrd_parameters() are present in the SPS RBSP syntax structure. The general_hrd_params_present_flag can be set to 0 to indicate that the syntax elements num_units_in_tick and time_scale and the syntax structure general_hrd_parameters() are not present in the SPS RBSP syntax structure. num_units_in_tick is the number of time units of a clock running at a frequency of time_scale Hertz (Hz), which corresponds to one increment of the clock tick counter (called a clock tick). num_units_in_tick should be greater than 0. The clock tick (in seconds) is equal to the quotient of num_units_in_tick divided by time_scale. For example, when the image rate of the video signal is 25 Hz, time_scale may be equal to 27000000, num_units_in_tick may be equal to 1080000, and thus the clock tick period may be equal to 0.04 seconds.

[0149] time_scale is the number of time units that pass in one second. For example, a time coordinate system using a 27 megahertz (MHz) clock to measure time has a time_scale of 27000000. The value of time_scale should be greater than 0. vps_extension_flag can be set to 0 to indicate that the vps_extension_data_flag syntax element is not present in the VPS RBSP syntax structure. vps_extension_flag can be set to 1 to indicate that the vps_extension_data_flag syntax element is present in the VPS RBSP syntax structure. vps_extension_data_flag can take any value. The presence and value of vps_extension_data_flag does not affect the decoder's compliance with the profile. Conforming decoders should ignore all vps_extension_data_flag syntax elements.

[0150] Figure 8is a schematic diagram of an exemplary video decoding device 800. The video decoding device 800 is suitable for implementing the disclosed examples / embodiments described herein. The video decoding device 800 includes a downstream port 820, an upstream port 850, and / or a transceiver unit (Tx / Rx) 810, wherein the transceiver unit (Tx / Rx) 810 includes a transmitter and / or a receiver for transmitting data upstream and / or downstream via a network. The video decoding device 800 also includes: a processor 830 including a logic unit and / or a central processing unit (CPU) for processing data; and a memory 832 for storing data. The video decoding device 800 may also include electronic components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to the upstream port 850 and / or the downstream port 820 for transmitting data via an electrical, optical, or wireless communication network. The video decoding device 800 may also include input and / or output (I / O) devices 860 for communicating data with a user. The I / O devices 860 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 860 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0151] The processor 830 is implemented in hardware and software. The processor 830 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 830 communicates with the downlink port 820, Tx / Rx 810, uplink port 850, and memory 832. The processor 830 includes a decoding module 814. The decoding module 814 implements the disclosed embodiments described herein, such as methods 100, 900, and 1000, which can use the multi-layer video sequence 500, the multi-view sequence 600, and / or the code stream 700. The decoding module 814 can also implement any other methods / mechanisms described herein. In addition, the decoding module 814 can implement the codec system 200, the encoder 300, and / or the decoder 400. For example, the decoding module 814 can be configured to decode each_layer_is_an_ols_flag to indicate whether each OLS includes a single layer, or whether at least one OLS includes more than one layer, to support multi-view when using simulcast layers. Thus, the decoding module 814 enables the video decoding device 800 to provide additional functionality and / or decoding efficiency when decoding video data. Thus, the decoding module 814 improves the functionality of the video decoding device 800 and addresses issues specific to the field of video coding. Furthermore, the decoding module 814 can transition the video decoding device 800 to different states. Alternatively, the decoding module 814 can be implemented as instructions stored in the memory 832 and executed by the processor 830 (e.g., as a computer program product stored on a non-transitory medium).

[0152] The memory 832 includes one or more memory types, such as a magnetic disk, a tape drive, a solid-state drive, a read-only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 832 can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the programs.

[0153] Figure 9900 is a flow chart of an exemplary method for encoding a video sequence (e.g., in codestream 700) having an OLS for a simulcast layer in a multi-view (e.g., in multi-view sequence 600). An encoder (e.g., codec system 200, encoder 300, and / or video decoding device 800) may use method 900 when performing method 100.

[0154] Method 900 may begin with an encoder receiving a video sequence and determining, for example based on user input, to encode the video sequence into a set of simulcast layers for use in multi-view. In step 901, the encoder encodes a codestream comprising one or more layers of coded images. For example, these layers may be simulcast layers and may not be encoded based on inter-layer prediction. Furthermore, these layers may be encoded to support multi-view video. Thus, the layers may be organized into OLSs, where one or more OLSs include two layers (e.g., one layer for display to each eye of the end user).

[0155] In step 903, the encoder may encode the VPS into the bitstream. The VPS may include various syntax elements to indicate the layer / OLS configuration to the decoder for correct multi-view decoding and display. For example, the VPS may include vps_all_independent_layers_flag, which may be set to 1 to indicate that all layers specified by the VPS are independently coded without the need for inter-layer prediction. When vps_all_independent_layers_flag is set to 1, therefore when all layers specified by the VPS are independently coded without the need for inter-layer prediction, the VPS may also include each_layer_is_an_ols_flag. each_layer_is_an_ols_flag may indicate whether each OLS includes only one layer, or whether at least one OLS includes more than one layer. For example, when indicating that each OLS includes only one layer and / or each layer is an OLS where the single included layer is the only output layer, each_layer_is_an_ols_flag may be set to 1. Therefore, when multi-view is not in use, each_layer_is_an_ols_flag may be set to 1. For another example, when it is indicated that at least one OLS includes more than one layer and thus the codestream encoded in step 901 includes multi-view video, each_layer_is_an_ols_flag may be set to 0.

[0156] The VPS may also include an ols_mode_idc syntax element. For example, when each_layer_is_an_ols_flag is set to 0 and vps_all_independent_layers_flag is set to 1, ols_mode_idc may be set to 2. When ols_mode_idc is set to / equal to 2, the total number of OLSs is explicitly indicated in the VPS. In addition, when ols_mode_idc is set to / equal to 2, the number of layers associated with each OLS and / or the number of output layers is explicitly indicated in the VPS. In a specific example, a vps_max_layers_minus1 syntax element may be included in the VPS to explicitly indicate the number of layers specified by the VPS, and therefore the number of layers that may be included in the OLS. In some examples, when vps_max_layers_minus1 is greater than 0, vps_all_independent_layers_flag may be indicated. In another specific example, when ols_mode_idc is equal to 2, num_output_layer_sets_minus1 may be included in the VPS. num_output_layer_sets_minus1 may indicate the total number of OLSs specified by the VPS. Therefore, when such data is explicitly indicated (e.g., when each_layer_is_an_ols_flag is set to 0, vps_all_independent_layers_flag is set to 1, and ols_mode_idc is set and / or inferred to be equal to 2), vps_max_layers_minus1 and num_output_layer_sets_minus1 may be indicated in the VPS to indicate the number of layers and the number of OLSs, respectively. In a specific example, when vps_all_independent_layers_flag is set to 1 and when each_layer_is_an_ols_flag is set to 0, ols_mode_idc may be inferred to be equal to 2.

[0157] In step 905, the code stream is stored for transmission to a decoder.

[0158] Figure 10 1 is a flow chart of an exemplary method 1000 for decoding a video sequence (e.g., from codestream 700) including an OLS for a simulcast layer in a multi-view (e.g., in multi-view sequence 600). A decoder (e.g., codec system 200, decoder 400, and / or video decoding device 800) may use method 1000 when performing method 100.

[0159] For example, after method 900 ends, method 1000 may begin with a decoder beginning to receive a bitstream comprising an OLS for simulcast multi-view layers. In step 1001, the decoder receives the bitstream. The bitstream may include one or more OLSs and one or more layers. For example, these layers may be simulcast layers and may not be encoded based on inter-layer prediction. Furthermore, these layers may be encoded to support multi-view video. Thus, the layers may be organized into OLSs, where one or more OLSs include two layers (e.g., one layer for display to each eye of the end user).

[0160] The codestream may also include a VPS. The VPS may include various syntax elements to indicate the layer / OLS configuration to the decoder for correct multi-view decoding and display. For example, the VPS may include vps_all_independent_layers_flag, which may be set to 1 to indicate that all layers specified by the VPS are independently coded without the need for inter-layer prediction. When vps_all_independent_layers_flag is set to 1, therefore when all layers specified by the VPS are independently coded without the need for inter-layer prediction, the VPS may also include each_layer_is_an_ols_flag. each_layer_is_an_ols_flag may indicate whether the OLS includes more than one layer. For example, when indicating that each OLS includes only one layer and / or each layer is an OLS where the single included layer is the only output layer, each_layer_is_an_ols_flag may be set to 1. Therefore, when multi-view is not in use, each_layer_is_an_ols_flag may be set to 1. For another example, when it is indicated that at least one OLS includes more than one layer and thus the codestream includes multi-view video, each_layer_is_an_ols_flag may be set to 0.

[0161] The VPS may also include an ols_mode_idc syntax element. For example, when each_layer_is_an_ols_flag is set to 0 and vps_all_independent_layers_flag is set to 1, ols_mode_idc may be set to 2. When ols_mode_idc is set to 2, the total number of OLSs is explicitly indicated in the VPS. In addition, when ols_mode_idc is set to / equal to 2, the number of layers associated with each OLS and / or the number of output layers is explicitly indicated in the VPS. In a specific example, a vps_max_layers_minus1 syntax element may be included in the VPS to explicitly indicate the number of layers specified by the VPS, and therefore the number of layers that may be included in the OLS. In some examples, when vps_max_layers_minus1 is greater than 0, vps_all_independent_layers_flag may be indicated. In another specific example, when ols_mode_idc is equal to 2, num_output_layer_sets_minus1 may be included in the VPS. num_output_layer_sets_minus1 may indicate the total number of OLSs specified by the VPS. Therefore, when such data is explicitly indicated (e.g., when each_layer_is_an_ols_flag is set to 0, vps_all_independent_layers_flag is set to 1, and ols_mode_idc is set and / or inferred to be equal to 2), vps_max_layers_minus1 and num_output_layer_sets_minus1 may be indicated in the VPS to indicate the number of layers and the number of OLSs, respectively. In a specific example, when vps_all_independent_layers_flag is set to 1 and when each_layer_is_an_ols_flag is set to 0, ols_mode_idc may be inferred to be equal to 2.

[0162] In step 1003, the decoder decodes the coded picture from the output layer of the OLS according to the each_layer_is_an_ols_flag in the VPS to produce a decoded picture. For example, the decoder can read the vps_all_independent_layers_flag to determine that all layers are simulcast. The decoder can also read the each_layer_is_an_ols_flag to determine that at least one OLS includes more than one layer. The decoder can also read the ols_mode_idc to determine that the number of OLSs and the number of layers are explicitly indicated. The decoder can then determine the number of OLSs and the number of layers by reading num_output_layer_sets_minus1 and vps_max_layers_minus1, respectively. The decoder can then use this information to locate the correct multi-view layer in the codestream. The decoder can also locate the correct coded picture from the layer. The decoder can then decode the picture to produce a decoded picture.

[0163] In step 1005, the decoder may forward the decoded picture for display as part of a decoded video sequence.

[0164] Figure 11 FIG1 is a diagram of an exemplary system 1100 for decoding a video sequence (e.g., in codestream 700) having an OLS for simulcast layers in multiple views (e.g., in multi-view sequence 600). System 1100 can be implemented by an encoder and a decoder, such as codec system 200, encoder 300, decoder 400, and / or video decoding device 800. Furthermore, system 1100 can utilize multi-layer video sequence 500. Furthermore, system 1100 can be utilized when implementing methods 100, 900, and / or 1000.

[0165] System 1100 includes a video encoder 1102. Video encoder 1102 includes an encoding module 1105 for encoding a bitstream including one or more layers of a coded picture. Encoding module 1105 is further configured to encode a VPS including a each_layer_is_an_ols_flag into the bitstream when all layers specified by the VPS are independently coded without requiring inter-layer prediction. The each_layer_is_an_ols_flag indicates whether each OLS includes only one layer. Video encoder 1102 also includes a storage module 1106 for storing the bitstream for transmission to a decoder. Video encoder 1102 also includes a transmission module 1107 for transmitting the bitstream to a video decoder 1110. Video encoder 1102 may also be configured to perform any of the steps of method 900.

[0166] System 1100 also includes a video decoder 1110. Video decoder 1110 includes a receiving module 1111 for receiving a bitstream including one or more layers of coded pictures and a VPS, wherein, when all layers specified by the VPS are independently coded without requiring inter-layer prediction, a "each_layer_is_an_ols_flag" is included in the VPS, indicating whether each OLS includes only one layer. Video decoder 1110 also includes a decoding module 1113 for decoding the coded pictures from the output layer of the OLS according to the "each_layer_is_an_ols_flag" in the VPS to generate decoded pictures. Video decoder 1110 also includes a forwarding module 1115 for forwarding the decoded pictures for display as part of a decoded video sequence. Video decoder 1110 can also be configured to perform any of the steps of method 1000.

[0167] A first component is directly coupled to a second component when there are no intervening components between them other than a wire, trace, or other medium. A first component is indirectly coupled to a second component when there are intervening components between them other than a wire, trace, or other medium. The term "coupled" and variations thereof encompass both direct and indirect couplings. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the following figure.

[0168] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and certain steps may be omitted or combined.

[0169] Although the present invention provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in a variety of other specific forms without departing from the spirit or scope of the present invention. The examples of the present invention are to be considered illustrative rather than restrictive, and the present invention is not limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0170] In addition, the techniques, systems, subsystems, and methods described and illustrated as separate or independent in various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present invention. Other changes, substitutions, and modifications can be determined by those skilled in the art and may be made without departing from the spirit and scope of the present invention.

Claims

1. A method implemented in a decoder, characterized in that The method comprises: A receiver of the decoder receives a codestream including one or more layers of a coded picture and a video parameter set (VPS), wherein when all layers specified by the VPS are independently coded without using inter-layer prediction, a flag (each_layer_is_an_ols_flag) for each layer being an output layer set (OLS) is included in the VPS, the flag each_layer_is_an_ols_flag being equal to 1 indicating that each OLS includes only one layer and that the one layer included in each OLS is the only output layer in the OLS, and the flag each_layer_is_an_ols_flag being equal to 0 indicating that at least one OLS includes more than one layer; The processor of the decoder decodes the coded image from the output layer of the OLS according to the each_layer_is_an_ols_flag in the VPS to generate a decoded image; The processor forwards the decoded image for display as part of a decoded video sequence.

2. The method according to claim 1, characterized in that The VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently coded without using inter-layer prediction.

3. The method according to claim 2, characterized in that The VPS includes a VPS maximum number of layers minus 1 (vps_max_layers_minus1) syntax element indicating the number of layers specified by the VPS, and when the vps_max_layers_minus1 is greater than 0, the VPS includes the vps_all_independent_layers_flag.

4. A method implemented in an encoder, characterized in that The method comprises: The processor of the encoder encodes a code stream comprising one or more layers of a coded image; The processor encodes a video parameter set (VPS) into the bitstream to obtain an encoded bitstream, wherein when all layers specified by the VPS are independently encoded without using inter-layer prediction, the VPS includes a flag (each_layer_is_an_ols_flag) for each layer to be an output layer set (OLS), wherein each_layer_is_an_ols_flag is equal to 1 to indicate that each OLS includes only one layer and the layer included in each OLS is the only output layer in the OLS, and each_layer_is_an_ols_flag is equal to 0 to indicate that at least one OLS includes more than one layer; A memory coupled to the processor stores the encoded code stream for transmission to a decoder.

5. The method according to claim 4, characterized in that The VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently coded without using inter-layer prediction.

6. The method according to claim 5, characterized in that The VPS includes a VPS maximum number of layers minus 1 (vps_max_layers_minus1) syntax element indicating the number of layers specified by the VPS, and when the vps_max_layers_minus1 is greater than 0, the VPS includes the vps_all_independent_layers_flag.

7. A video decoding device, characterized in that: include: A processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 6.

8. A non-transitory computer-readable medium, characterized in that A computer program product for use with a video decoding device, wherein the computer program product comprises computer-executable instructions stored in the non-transitory computer-readable medium, and when the computer-executable instructions are executed by a processor, the video decoding device performs the method according to any one of claims 1 to 6.

9. A decoder, characterized in that: include: a receiving module configured to receive a codestream comprising one or more layers of a coded image and a video parameter set (VPS), wherein when all layers specified by the VPS are independently coded without using inter-layer prediction, a flag (each_layer_is_an_ols_flag) for each layer being an output layer set (OLS) is included in the VPS, the flag being equal to 1 indicating that each OLS includes only one layer and that the layer included in each OLS is the only output layer in the OLS, and the flag being equal to 0 indicating that at least one OLS includes more than one layer; a decoding module for decoding the coded image from the output layer of the OLS according to the each_layer_is_an_ols_flag in the VPS to generate a decoded image; A forwarding module is configured to forward the decoded image to be displayed as part of a decoded video sequence.

10. The decoder according to claim 9, characterized in that The VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently coded without using inter-layer prediction.

11. The decoder according to claim 10, wherein: The VPS includes a VPS maximum number of layers minus 1 (vps_max_layers_minus1) syntax element indicating the number of layers specified by the VPS, and when the vps_max_layers_minus1 is greater than 0, the VPS includes the vps_all_independent_layers_flag.

12. An encoder, characterized in that include: Encoding module for: encoding a codestream comprising one or more layers of a coded picture; encoding a video parameter set (VPS) into the bitstream to obtain a coded bitstream, wherein, when all layers specified by the VPS are independently coded without using inter-layer prediction, the VPS includes a flag (each_layer_is_an_ols_flag) for each layer to be an output layer set (OLS), wherein each_layer_is_an_ols_flag is equal to 1 to indicate that each OLS includes only one layer and the layer included in each OLS is the only output layer in the OLS, and wherein each_layer_is_an_ols_flag is equal to 0 to indicate that at least one OLS includes more than one layer; The storage module is used to store the encoded code stream to send it to the decoder.

13. The encoder according to claim 12, wherein: The VPS includes vps_all_independent_layers_flag set to 1 to indicate that all layers specified by the VPS are independently coded without using inter-layer prediction.

14. The encoder according to claim 13, wherein The VPS includes a VPS maximum number of layers minus 1 (vps_max_layers_minus1) syntax element indicating the number of layers specified by the VPS, and when the vps_max_layers_minus1 is greater than 0, the VPS includes the vps_all_independent_layers_flag.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an encoded code stream obtained by the method according to any one of claims 4 to 6.

Images