Layer-based parameter set NAL unit constraints

Abstract

A video coding mechanism is disclosed. The mechanism includes receiving a bitstream comprising a picture parameter set (PPS) in a PPS network abstraction layer (NAL) unit and a coded picture comprising a set of video coding layer (VCL) NAL units. A particular VCL NAL unit does not reference a PPS NAL unit unless the value of the NAL unit header layer identifier (nuh_layer_id) of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of the particular VCL NAL unit. The coded picture is decoded from the set of VCL NAL units based on the PPS in the PPS NAL unit to generate a decoded picture. The decoded picture is transmitted for display as part of a decoded video sequence.

Description

Technical Field

[0001] Cross - Reference to Related Applications This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 905,236, filed on September 24, 2019, entitled "Improved Video Coding" by e - Kui Wang. 、Y

[0002] The present disclosure generally relates to video coding and, more particularly, to improvements in signaling parameters for supporting the coding of multi - layer bitstreams.

Background Art

[0003] Even for relatively short videos, the video data required to represent them can be quite large, which can cause difficulties when the data is streamed or transmitted across communication networks with limited bandwidth capacity. For this reason, video data is usually compressed before being transmitted over today's communication networks. When a video is stored in a memory device, the size of the video can also be a problem because of limited memory resources. Video compression devices often use software and / or hardware at the source to code the video data prior to transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device that decodes the video data at the destination. Due to limited network resources and the increasing demand for higher video quality, improved compression and decompression techniques that increase the compression ratio without sacrificing much or any of the picture quality are desired.

Summary of the Invention

Means for Solving the Problems

[0004] ​In one embodiment, the Disclosure includes a method implemented by a decoder, the method comprising: receiving a bitstream comprising a Picture Parameter Set (PPS) in a PPS Network Abstraction Layer (NAL) unit and a coded picture comprising a set of Video Coding Layer (VCL) NAL units, wherein a particular VCL NAL unit does not refer to a PPS NAL unit unless the value of the NAL unit header layer identifier (nuh_layer_id) of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit; and decoding the coded picture from the set of VCL NAL units based on the PPS in the PPS NAL unit by the decoder's processor in order to generate a coded picture.

[0005] Some video coding systems code video sequences into layers of pictures. Pictures in different layers have different properties. Therefore, an encoder can send different layers to a decoder depending on the decoder's constraints. To perform this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in NAL units. Each NAL unit receives a layer identifier (ID) known as nuh_layer_id. The sub-bitstream extraction process can then remove all NAL units with nuh_layer_ids that are outside the target range. The problem with this technique is that some NAL units are related to many layers. For example, PPS and Adaptive Parameter Set (APS) may contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the sub-bitstream extraction process can remove the PPS and / or APS. When this happens, the decoder does not have enough data to decode the extracted bitstream.

[0006] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0007] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the bitstream further includes an APS within the APS NAL unit, and a particular VCL NAL unit does not reference an APS NAL unit unless the value of the nuh_layer_id of the APS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit.

[0008] Optionally, in any of the embodiments described above, in other embodiments of the embodiment, a particular VCL NAL unit comprises a slice of coded picture and a header containing data elements related to the slice, the header specifying the APS ID of the APS.

[0009] Optionally, in any of the embodiments described above, in other embodiments of the embodiment, a particular VCL NAL unit comprises a slice of coded picture, the slice is associated with a header containing syntax elements to apply to the slice of coded picture, and the header specifies the PPS ID of the PPS.

[0010] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the bitstream is the result of a sub-bitstream extraction process that removes NAL units from the bitstream based on the nuh_layer_id value.

[0011] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, if the set of VCL NAL units all have a specific value for nuh_layer_id, then the set of VCL NAL units is part of a layer.

[0012] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the layer includes non-VCL NAL units associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id.

[0013] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the decoder receives a second bitstream comprising a second PPS in a second PPS NAL unit and a second coded picture comprising a second set of VCL NAL units, wherein the second VCL NAL unit refers to the second PPS NAL unit, provided that the value of the nuh_layer_id of the second PPS NAL unit is not less than or equal to the value of the nuh_layer_id of the second VCL NAL unit; and depending on the receiving step, takes several other corrective measures to ensure that a matching bitstream corresponding to the second bitstream is received before decoding the coded picture from the second set of VCL NAL units.

[0014] In one embodiment, the Disclosure includes a method implemented by an encoder, the method comprising: the processor of the encoder encoding a coded picture comprising a set of VCL NAL units into a bitstream; the processor encoding PPS in PPS NAL units into a bitstream based on the constraint that a particular VCL NAL unit does not refer to a PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit; and the memory coupled to the processor storing the bitstream for communication to a decoder.

[0015] Some video coding systems code video sequences into layers of pictures. Pictures in different layers have different properties. Therefore, an encoder can send different layers to a decoder depending on the decoder's constraints. To perform this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in NAL units. Each NAL unit receives a layer ID known as nuh_layer_id. The sub-bitstream extraction process can then remove all NAL units with nuh_layer_ids that are outside the target range. The problem with this technique is that some NAL units are related to many layers. For example, PPS and APS may contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the sub-bitstream extraction process can remove the PPS and / or APS. When this happens, the decoder does not have enough data to decode the extracted bitstream.

[0016] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0017] Optionally, in any of the embodiments described above, other embodiments of the embodiment further include the step of the processor encoding an adaptive parameter set (APS) in an APS NAL unit into a bitstream, based on the constraint that a particular VCL NAL unit does not refer to an APS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit.

[0018] Optionally, in any of the embodiments described above, in other embodiments of the embodiment, a particular VCL NAL unit comprises a slice of coded picture and a header containing data elements related to the slice, the header specifying the APS ID of the APS.

[0019] Optionally, in any of the embodiments described above, in other embodiments of the embodiment, a particular VCL NAL unit comprises a slice of coded picture, the slice is associated with a header containing syntax elements to apply to the slice of coded picture, and the header specifies the PPS ID of the PPS.

[0020] Optionally, in any of the embodiments described above, other embodiments of the embodiment further include the step of the processor performing a sub-bitstream extraction process to remove NAL units from the bitstream based on the nuh_layer_id value.

[0021] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, if the set of VCL NAL units all have a specific value for nuh_layer_id, then the set of VCL NAL units is part of a layer.

[0022] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the layer includes non-VCL NAL units associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id.

[0023] In one embodiment, the disclosure includes a video coding device comprising a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform any of the methods described above.

[0024] In one embodiment, the disclosure includes a non-temporary computer-readable medium comprising a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions stored on the non-temporary computer-readable medium to cause the video coding device to perform any of the methods described above when executed by a processor.

[0025] In one embodiment, the Disclosure includes a receiving means for receiving a bitstream comprising a PPS in a PPS NAL unit and a coded picture comprising a set of VCL NAL units, wherein a particular VCL NAL unit does not refer to a PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit; and a decoder comprising a decoding means for decoding a coded picture from a set of VCL NAL units based on the PPS in the PPS NAL unit to generate a coded picture; and a transfer means for transferring the coded picture for display as part of a coded video sequence.

[0026] Some video coding systems code a video sequence into picture layers. Pictures within different layers have different characteristics. Thus, an encoder can send different layers to a decoder according to the constraints on the decoder side. To fulfill this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in NAL units. Each NAL unit receives a layer ID known as nuh_layer_id. Subsequently, the sub-bitstream extraction process can remove all NAL units with nuh_layer_id outside the target range. The problem with this approach is that some NAL units are related to multiple layers. For example, PPS and APS can contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the PPS and / or APS can be removed by the sub-bitstream extraction process. When this occurs, the decoder does not have sufficient data to decode the extracted bitstream.

[0027] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0028] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the decoder is further configured to perform any of the methods described above.

[0029] In one embodiment, the present disclosure encodes a coded picture comprising a set of VCL NAL units into a bitstream, and encodes the PPS in the PPS NAL unit into the bitstream based on a constraint that a particular VCL NAL unit does not reference the PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of the particular VCL NAL unit, and includes an encoder comprising storage means for storing the bitstream for communication to a decoder.

[0030] Some video coding systems code a video sequence into picture layers. Pictures within different layers have different characteristics. Thus, an encoder can send different layers to a decoder according to constraints on the decoder side. To achieve this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be realized by including all data in NAL units. Each NAL unit receives a layer ID known as nuh_layer_id. Thereafter, the sub-bitstream extraction process can remove all NAL units with nuh_layer_id outside the target range. The problem with this approach is that some NAL units are related to multiple layers. For example, PPS and APS can contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the PPS and / or APS can be removed by the sub-bitstream extraction process. When this occurs, the decoder does not have sufficient data to decode the extracted bitstream.

[0031] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0032] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the encoder is further configured to perform any of the methods described above.

[0033] In one embodiment, the disclosure includes a method implemented by a decoder, the method comprising: receiving a bitstream comprising a PPS in a PPS NAL unit and a coded picture comprising a set of VCL NAL units, wherein the VCL NAL unit referencing the PPS is constrained to include a nuh_layer_id value that is less than or equal to the nuh_layer_id value of the PPS NAL unit; and decoding the coded picture from the set of VCL NAL units based on the PPS in the PPS NAL unit by the decoder's processor in order to generate a coded picture.

[0034] Some video coding systems code video sequences into layers of pictures. Pictures in different layers have different properties. Therefore, an encoder can send different layers to a decoder depending on the decoder's constraints. To perform this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in NAL units. Each NAL unit receives a layer ID known as nuh_layer_id. The sub-bitstream extraction process can then remove all NAL units with nuh_layer_ids that are outside the target range. The problem with this technique is that some NAL units are related to many layers. For example, PPS and APS may contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the sub-bitstream extraction process can remove the PPS and / or APS. When this happens, the decoder does not have enough data to decode the extracted bitstream.

[0035] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0036] Optionally, in any of the embodiments described above, in other embodiments of the embodiment, the bitstream further comprises an APS within the APS NAL unit, and the VCL NAL unit referencing the APS is constrained to contain a nuh_layer_id value less than or equal to the nuh_layer_id value of the APS NAL unit.

[0037] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, each VCL NAL unit comprises a slice of coded picture and a header containing data elements related to the slice, the header specifying the APS ID of the APS.

[0038] Optionally, in any of the embodiments described above, and in other embodiments of the embodiments, each VCL NAL unit comprises a slice of coded picture, the slice is associated with a header containing syntax elements to apply to the slice of coded picture, the header specifying the PPS ID of the PPS.

[0039] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the bitstream is the result of a sub-bitstream extraction process that removes NAL units from the bitstream based on the nuh_layer_id value.

[0040] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, if the set of VCL NAL units all have a specific value for nuh_layer_id, then the set of VCL NAL units is part of a layer.

[0041] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the layer includes non-VCL NAL units associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id.

[0042] In one embodiment, the Disclosure includes a method implemented by an encoder, the method comprising: encoding a coded picture comprising a set of VCL NAL units into a bitstream by the encoder's processor; encoding a PPS in a PPS NAL unit into a bitstream based on the constraint that the VCL NAL unit referencing the PPS contains a nuh_layer_id value that is less than or equal to the nuh_layer_id value of the PPS NAL unit; and storing the bitstream in a memory coupled to the processor for communication to a decoder.

[0043] Some video coding systems code video sequences into layers of pictures. Pictures in different layers have different properties. Therefore, an encoder can send different layers to a decoder depending on the decoder's constraints. To perform this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in NAL units. Each NAL unit receives a layer ID known as nuh_layer_id. The sub-bitstream extraction process can then remove all NAL units with nuh_layer_ids that are outside the target range. The problem with this technique is that some NAL units are related to many layers. For example, PPS and APS may contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or an APS NAL unit is outside the target range, the sub-bitstream extraction process can remove the PPS and / or APS. When this happens, the decoder does not have enough data to decode the extracted bitstream.

[0044] This example constrains PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by VCL NAL units containing slices of coded pictures. PPS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference PPS within the PPS NAL unit. Similarly, APS NAL units are constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Thus, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0045] Optionally, in any of the embodiments described above, other embodiments of the embodiment further include the step of the processor encoding the APS in the APS NAL unit into a bitstream based on the constraint that the VCL NAL unit referencing the APS contains a nuh_layer_id value that is less than or equal to the nuh_layer_id value of the APS NAL unit.

[0046] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, each VCL NAL unit comprises a slice of coded picture and a header containing data elements related to the slice, the header specifying the APS ID of the APS.

[0047] Optionally, in any of the embodiments described above, and in other embodiments of the embodiments, each VCL NAL unit comprises a slice of coded picture, the slice is associated with a header containing syntax elements to apply to the slice of coded picture, the header specifying the PPS ID of the PPS.

[0048] Optionally, in any of the embodiments described above, other embodiments of the embodiment further include the step of the processor performing a sub-bitstream extraction process to remove NAL units from the bitstream based on the nuh_layer_id value.

[0049] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, if the set of VCL NAL units all have a specific value for nuh_layer_id, then the set of VCL NAL units is part of a layer.

[0050] Optionally, in any of the embodiments described above, and in other embodiments of the embodiment, the layer includes non-VCL NAL units associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id.

[0051] For clarity, any one of the embodiments described above may be combined with any one or more of the other embodiments described above to bring about new embodiments within the scope of this disclosure.

[0052] These and other features can be better understood by interpreting the following detailed description in conjunction with the attached drawings and claims.

[0053] To better understand this disclosure, hereby refer to the following brief description, in which similar reference numbers represent similar parts in connection with the attached drawings and detailed description. [Brief explanation of the drawing]

[0054] [Figure 1] This is a flowchart illustrating one example of a method for coding video signals. [Figure 2] This is a schematic diagram of an example of a coding / decoding (codec) system for video coding. [Figure 3] This is a schematic diagram showing an example of a video encoder. [Figure 4] This is a schematic diagram showing an example of a video decoder. [Figure 5] This is a schematic diagram showing an example of a virtual reference decoder (HRD). [Figure 6] This is a schematic diagram showing an example of a multi-layer video sequence. [Figure 7] This is a schematic diagram showing an example of a bitstream. [Figure 8] This is a schematic diagram of an example of a video coding device. [Figure 9] This is a flowchart illustrating an example of a method for encoding a video sequence into a bitstream by constraining layer identifiers (IDs) for PPS and APS within the bitstream. [Figure 10] This is a flowchart illustrating an example of a method for decoding a video sequence from a bitstream where the layer IDs for PPS and APS within the bitstream are constrained. [Figure 11] This is a schematic diagram of an example system for coding video sequences using a bitstream where the layer IDs for PPS and APS within the bitstream are constrained. [Modes for carrying out the invention]

[0055] Firstly, while exemplary implementations of one or more embodiments are provided below, it should be understood that the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and practices illustrated or described herein, and may be modified within the scope of the appended claims, along with the entire scope of their equivalents.

[0056] The following terms are defined as follows, unless used in the opposite context herein. Specifically, the following definitions are intended to further clarify this disclosure. However, terms may be described differently in different contexts. Therefore, the following definitions should be considered supplementary and should not be considered to limit other definitions given to such terms herein.

[0057] A bitstream is a sequence of bits containing video data that is compressed for transmission between an encoder and a decoder. An encoder is a device configured to use the encoding process to compress video data into a bitstream. A decoder is a device configured to use the decoding process to reconstruct video data from the bitstream for display. A picture is an array of luma samples and / or chroma samples that make up a frame or its fields. A slice is an integer number of complete tiles or a contiguous row of complete coding tree units (CTUs) of an integer number of pictures that are exclusively contained within a single network abstraction layer (NAL) unit (e.g., within a tile). For clarity, the picture being encoded or decoded can be called the current picture. A coded picture is a coded representation of a picture that includes all of the coding tree units (CTUs) of the picture, along with a video coding layer (VCL) NAL unit with a specific value for the NAL unit header layer identifier (nuh_layer_id) in an access unit (AU). A decoded picture is a picture produced by applying the decoding process to a coded picture.

[0058] An AU is a set of coded pictures that are contained in different layers and associated at the same time for output from a Decoding Picture Buffer (DPB). A NAL unit is a syntax structure containing data in the form of a raw byte sequence payload (RBSP), an indication of the data type, and scattered as desired with an emulation-preventing byte. A VCL NAL unit is a NAL unit coded to contain video data, such as a coded slice of a picture. A non-VCL NAL unit is a NAL unit containing non-video data, such as syntax and / or parameters that support decoding video data, performing conformance checks, or other operations. A layer is a set of VCL NAL units that share specified characteristics (e.g., common resolution, frame rate, image size, etc.), as indicated by the layer ID and associated non-VCL NAL units. The NAL unit header layer identifier (nuh_layer_id) is a syntax element that specifies the identifier of the layer containing the NAL unit.

[0059] A Picture Parameter Set (PPS) is a syntax structure containing syntax elements that apply to the entire coded picture, determined by the syntax elements found in each picture header. A PPS NAL is a NAL unit containing the PPS. The PPS ID, also called pps_pic_parameter_set_id, is an identifier that identifies the PPS for reference by other syntax elements. An Adaptive Parameter Set (APS) is a syntax structure containing syntax elements that apply to a slice, determined by the syntax elements found in the slice header. An APS NAL is a NAL unit containing the APS. The APS ID, also called aps_adaptation_parameter_set_id, is an identifier for the APS for reference by other syntax elements. A picture header is a syntax structure containing syntax elements that apply to all slices of a coded picture. A slice header is part of a coded slice containing data elements about the CTU rows within a tile represented by the entire tile or slice. A coded video sequence is a set of one or more coded pictures. A decoded video sequence is a set of one or more decoded pictures. The sub-bitstream extraction process removes NAL units from the bitstream that are not included in the target set, determined by the target output layer set (OLS) index and the target highest temporal identifier (TemporalId), and the output sub-bitstream contains the NAL units from the bitstream that are included in the target set.

[0060] In this specification, the following acronyms are used: Access Unit (AU), Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Codified Layer Video Sequence (CLVS), Codified Layer Video Sequence Start (CLVSS), Codified Video Sequence (CVS), Codified Video Sequence Start (CVSS), Joint Video Expert Team (JVET), Hypothetical Critical Decoder (HRD), Motion Constrained Tilesets (MCTS), Maximum Transmission Unit (MTU), Network Abstraction Layer (NAL), Output Layer Set (OLS), Picture Order Count (POC), Random Access Point (RAP), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), Video Parameter Set (VPS), and Multipurpose Video Coding (VVC).

[0061] Many video compression techniques can be used to reduce the size of video files while minimizing data loss. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or eliminate data redundancy in a video sequence. In block-based video coding, a video slice (e.g., a video picture or part of a video picture) may be divided into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coding (I) slice of a picture are coded using spatial predictions for reference samples in adjacent blocks within the same picture. Video blocks in an intercoding unidirectional prediction (P) or bidirectional prediction (B) slice of a picture may be coded using spatial predictions for reference samples in adjacent blocks within the same picture, or temporal predictions for reference samples in other reference pictures. A picture may also be referred to as a frame and / or image, and a reference picture may be referred to as a reference frame and / or reference image. Spatial or temporal predictions become prediction blocks representing image blocks. Residual data represents the pixel difference between the original image block and the prediction block. Thus, an intercoding block is encoded according to a motion vector pointing to a block of reference samples forming the prediction block, and residual data showing the difference between the coding block and the prediction block. An intracoding block is encoded according to the intracoding mode and residual data. For further compression, the residual data may be transformed from a pixel region to a transformation region. These result in residual transformation coefficients that can be quantized. The quantized transformation coefficients may initially be arranged in a two-dimensional array. The quantized transformation coefficients may be scanned to generate a one-dimensional vector of transformation coefficients. Entropy coding may be applied to achieve further compression.Such video compression techniques will be explained in more detail below.

[0062] To ensure accurate decoding of encoded video, video is encoded and decoded according to the corresponding video coding standard. Video coding standards include Advanced Video Coding (AVC), also known as ITU-T H.261, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding Plus Depth (MVC+D), and Three-Dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The Joint Video Expert Team (JVET) of ITU-T and ISO / IEC has begun development of a video coding standard called General-Purpose Video Coding (VVC). VVC is included in working drafts (WDs), including JVET-O2001-v14.

[0063] Some video coding systems code video sequences into layers of pictures. Pictures in different layers have different properties. Therefore, an encoder can send different layers to a decoder depending on the decoder's constraints. To perform this function, the encoder can encode all layers into a single bitstream. Upon request, the encoder can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream. The result is an extracted bitstream containing only the data for decoding the requested layer. This process can be achieved by including all data in a Network Abstraction Layer (NAL) unit. Each NAL unit receives a layer identifier (ID) known as the NAL unit header layer identifier (nuh_layer_id). The sub-bitstream extraction process can then remove all NAL units with nuh_layer_ids that are outside the target range. The problem with this technique is that some NAL units are related to many layers. For example, a Picture Parameter Set (PPS) and an Adaptive Parameter Set (APS) may contain data related to one or more layers. If the nuh_layer_id of a PPS NAL unit or APS NAL unit is outside the target range, the sub-bitstream extraction process can remove the PPS and / or APS. When this happens, the decoder does not have enough data to decode the extracted bitstream.

[0064] This specification discloses a mechanism for constraining PPS and APS to prevent corresponding NAL units from being inadvertently removed from the bitstream during the sub-bitstream extraction process. Specifically, PPS and APS are referenced by video coding layer (VCL) NAL units containing slices of coded pictures. A PPS NAL unit is constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference the PPS within the PPS NAL unit. Similarly, an APS NAL unit is constrained to contain nuh_layer_ids less than or equal to the nuh_layer_ids of all VCL NAL units that reference the APS within the APS NAL unit. The sub-bitstream extraction process removes NAL units with nuh_layer_ids higher than the target nuh_layer_id. By ensuring that PPS NAL units and APS NAL units have nuh_layer_ids lower than all VCL NAL units that reference them, PPS / APS cannot be removed by the sub-bitstream extraction process, which also retains the VCL NAL units that reference PPS / APS. Therefore, errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0065] Figure 1 is a flowchart of one example of how video signal coding works. Specifically, the video signal is encoded by an encoder. The encoding process compresses the video signal using various techniques to reduce the size of the video file. A smaller file size allows the compressed video file to be transmitted to the user, reducing the associated bandwidth overhead. The decoder then decodes the compressed video file and restores the original video signal for display to the end user. To ensure that the decoder can reliably restore the video signal, the decoding process usually reflects the encoding process.

[0066] In step 101, a video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. A video file may contain both audio and video components. The video component contains a series of image frames, which, when viewed in sequence, give the impression of visual motion. Each frame contains pixels, which are represented by light, here called the lumina component (or lumina sample), and color, here called the chroma component (or color sample). In some examples, frames may also contain depth values ​​to support three-dimensional viewing.

[0067] In step 103, the video is divided into blocks. This division involves subdividing the pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be subdivided into coding tree units (CTUs), which are blocks of a predetermined size (e.g., 64 pixels × 64 pixels). A CTU contains both luminous and chroma samples. The CTUs can then be divided into blocks using the coding tree, and these blocks can be recursively subdivided until a configuration supporting further encoding is achieved. For example, the luminous component of a frame may be subdivided until each individual block contains relatively uniform illumination values. Furthermore, the chroma component of a frame may be subdivided until each individual block contains relatively uniform color values. Therefore, the division technique varies depending on the content of the video frame.

[0068] In step 105, the image blocks separated in step 103 are compressed using various compression techniques. For example, interpretation and / or intrapretation may be used. Interpretation takes advantage of the fact that objects in a typical scene tend to appear in consecutive frames. Therefore, a block representing an object in a reference frame does not need to be described repeatedly in adjacent frames. Specifically, an object such as a desk may remain in the same position across multiple frames. Therefore, the desk is described once, and adjacent frames can refer to the reference frame. By using pattern matching techniques, objects can be aligned across multiple frames. Furthermore, objects that move across multiple frames may be represented, for example, due to the movement of the object or the movement of the camera. As a specific example, a video may display a car crossing the screen across multiple frames. Such movement can be described using motion vectors. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in a given frame to the coordinates of that object in a reference frame. Therefore, interpretation can encode the image block in the current frame as a set of motion vectors indicating the offset from the corresponding block in the reference frame.

[0069] Intra-prediction encodes blocks within a typical frame. It leverages the fact that luma and chroma components tend to cluster within a frame. For example, a green patch in one part of a tree tends to be located near similar green patches. Intra-prediction uses multiple directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional modes indicate that a block is similar to / identical to samples of adjacent blocks in its corresponding direction. Planar mode indicates that a series of blocks along a row / column (e.g., a plane) can be interpolated based on adjacent blocks at the edges of the row. Planar mode effectively shows smooth light / color transitions across rows / columns by using a relatively constant slope of changing values. DC mode is used for boundary smoothing, indicating that a block is similar to / identical to the average value associated with samples of all adjacent blocks in the angular direction of the directional prediction mode. Thus, intra-prediction blocks can represent image blocks as various relational prediction mode values ​​instead of actual values. Furthermore, intra-prediction blocks can represent image blocks as motion vector values ​​instead of actual values. In either case, the predicted block may not accurately represent the image block in some cases. The difference is stored in the residual block. Transformations can be applied to the residual block to further compress the file.

[0070] In step 107, various filtering techniques are applied. In HEVC, filters are applied according to the in-loop filtering method. The block-based prediction method described above can create blocky images in the decoder. Furthermore, the block-based prediction method encodes the blocks, and the encoded blocks can be reconstructed later using them as reference blocks. The in-loop filtering method repeatedly applies noise suppression filters, deblocking filters, adaptive loop filters, and sample-adaptive offset (SAO) filters to blocks / frames. These filters reduce such blocking artifacts, so the encoded file can be accurately reconstructed. Moreover, these filters reduce artifacts in the reconstructed reference blocks, so there is less likelihood of further artifacts occurring in subsequent blocks encoded based on the reconstructed reference blocks.

[0071] Once the video signal has been segmented, compressed, and filtered, the resulting data is encoded into a bitstream in step 109. This bitstream includes the aforementioned data as well as desirable signaling data to assist in the proper reconstruction of the video signal by the decoder. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory so that it can be transmitted to the decoder on demand. The bitstream may also be broadcast and / or multicast to multiple decoders. The creation of the bitstream is a repetitive process. Therefore, steps 101, 103, 105, 107, and 109 may be performed sequentially and / or simultaneously across a large number of frames and blocks. The order shown in Figure 1 is presented for clarity and simplicity of explanation and is not intended to limit the video coding process to a specific order.

[0072] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data from the bitstream to determine the division of frames. The division must be consistent with the result of the block division in step 103. Now, let's explain the entropy encoding / decoding used in step 111. The encoder makes numerous choices during the compression process, such as selecting a block division scheme from several options based on the spatial arrangement of values ​​in the input image. Multiple bins may be used to communicate the perfect choice. The bins used here are binary values ​​treated as variables (e.g., bit values ​​that can change depending on the situation). In entropy coding, the encoder can discard options that are clearly invalid in a particular situation, leaving a set of acceptable options. Each acceptable option is then assigned a codeword. The length of the codeword is based on the number of acceptable options (e.g., one bin for two options, two bins for three or four options, etc.). The encoder then encodes a codeword for the selected option. This method reduces the size of the codeword because it is desired to uniquely indicate a selection from a smaller subset of acceptable options, rather than a larger set of all possible options. The decoder then decodes the selection by determining the set of acceptable options, similar to the encoder. The decoder can read the codeword by determining the set of acceptable options and determine the selection made by the encoder.

[0073] In step 113, the decoder performs block decoding. Specifically, the decoder generates residual blocks using the inverse transform. The decoder then uses the residual blocks and corresponding prediction blocks to reconstruct the image blocks based on the partitioning. The prediction blocks may include both intra-prediction blocks and inter-prediction blocks generated by the encoder in step 105. The reconstructed image blocks are then placed within the frames of the video signal reconstructed based on the partitioning data determined in step 111. The syntax of step 113 may also be transmitted as a bitstream by the entropic coding described above.

[0074] In step 115, the frames of the restored video signal are filtered, similar to step 107 in the encoder. To remove blocking artifacts, for example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be applied to the frames. Once the frames are filtered, in step 117 the video signal can be output to a display for viewing by the end user.

[0075] Figure 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functions to assist in the implementation of operation method 100. The codec system 200 is outlined to depict the components used in both the encoder and the decoder. The codec system 200 receives a video signal as described in steps 101 and 103 of operation method 100, partitions it, and as a result obtains a partitioned video signal 201. The codec system 200 then compresses the partitioned video signal 201 into a coded bitstream when functioning as an encoder, as described in steps 105, 107, and 109 of method 100. When functioning as a decoder, the codec system 200 generates an output video signal from the bitstream, as described in steps 111, 113, 115, and 117 of operation method 100. The codec system 200 includes a general-purpose coder control component 211, a transform scaling quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, an inverse scaling component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting context-adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown in the figure. In Figure 2, the black lines show the motion of the data being encoded / decoded, and the dashed lines show the motion of the control data that controls the operation of the other components. Any of the components of the codec system 200 may reside within the encoder. The decoder may contain a subset of the components of the codec system 200. For example, the decoder may include the intra-picture prediction component 217, the motion compensation component 219, the inverse scaling component 229, the in-loop filter component 225, and the decoded picture buffer component 223 may be included. These components will now be explained.

[0076] The segmented video signal 201 is a captured video sequence segmented into blocks of pixels by a coding tree. The coding tree subdivides blocks of pixels into smaller blocks of pixels using various split modes. These blocks can then be further subdivided into even smaller blocks. Blocks are sometimes called nodes in the coding tree. Larger parent nodes are split into smaller child nodes. The number of times a node is subdivided is called the node / coding tree depth. The segmented blocks may, in some cases, be contained within a coding unit (CU). For example, a CU may be a sub-part of a CTU, containing a lumen block, a red difference chroma (Cr) block, and a blue difference chroma (Cb) block, along with the corresponding CU syntax instructions. Split modes include binary trees (BT), triple trees (TT), and quad trees (QT), which are used to segment a node into two, three, or four child nodes, which differ in shape depending on the split mode used. The segmented video signal 201 is transferred to the general-purpose coder control component 211, the conversion scaling quantization component 213, the intrapicture estimation component 215, the filter control analysis component 227, and the motion estimation component 221 for compression.

[0077] The general-purpose coder control component 211 is configured to make decisions related to coding the images of a video sequence into a bitstream, depending on the constraints of the application. For example, the general-purpose coder control component 211 manages the optimization of bitrate / bitstream size and restoration quality. Such decisions may be made based on the available storage space / bandwidth and the image resolution requirements. The general-purpose coder control component 211 also manages buffer usage in light of the transmission speed to mitigate buffer underrun and overrun problems. To manage these problems, the general-purpose coder control component 211 manages the sorting, prediction, and filtering by other components. For example, the general-purpose coder control component 211 can dynamically increase the complexity of compression to increase resolution and bandwidth usage, or decrease the complexity of compression to decrease resolution and bandwidth usage. Therefore, the general-purpose coder control component 211 balances the restoration quality of the video signal with the bitrate problem by controlling other components of the codec system 200. The general-purpose coder control component 211 generates control data that controls the operation of other components. Furthermore, the control data is transferred to the header formatting CABAC component 231 to signal parameters for decoding by the decoder, and is encoded as a bitstream.

[0078] The segmented video signal 201 is also transmitted to the motion estimation component 221 and the motion compensation component 219 for interpretation. A frame or slice of the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform interpretation coding of the received video blocks based on one or more blocks in one or more reference frames to provide time prediction. The codec system 200 can execute multiple coding passes to select the appropriate coding mode for each block of video data, for example.

[0079] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors can, for example, represent the movement of a coded object relative to a prediction block. A prediction block is a block that is known to closely match the block to be coded in terms of pixel difference. Prediction blocks may also be called reference blocks. Such pixel differences can be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. HEVC uses several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be split into CTBs, which can then be split into CBs to be included in CUs. A CU can be encoded as a prediction unit containing prediction data and / or as a transformed unit (TU) containing the transformed residual data of the CU. The motion estimation component 221 generates motion vectors, prediction units, and TUs by using rate distortion analysis as part of a rate distortion optimization process. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc., for the current block / frame, and select the reference blocks, motion vectors, etc., that have the best rate distortion characteristics. The best rate distortion characteristics are those that balance the quality of video restoration (e.g., amount of data loss due to compression) and coding efficiency (e.g., the size of the final encoding).

[0080] In some cases, the codec system 200 can calculate the values ​​of sub-integer pixel positions of the reference picture stored in the decoded picture buffer component 223. For example, the video codec system 200 can interpolate the values ​​of quarter-pixel, eighth-pixel, or other fractional pixel positions of the reference picture. Thus, the motion estimation component 221 can perform motion search based on all pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates motion vectors for the predicted units of the video blocks in the intercoding slice by comparing the positions of the predicted units with the positions of the predicted blocks in the reference picture. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header formatting CABAC component 231 for encoding and outputs motion to the motion compensation component 219.

[0081] Motion compensation performed by the motion compensation component 219 allows for the acquisition or generation of predicted blocks based on motion vectors determined by the motion estimation component 221. Again, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Having received the motion vectors of the predicted units of the current video block, the motion compensation component 219 can determine the position of the predicted block pointed to by the motion vectors. The residual video block is then formed by subtracting the pixel values ​​of the predicted block from the pixel values ​​of the current video block to be coded, thereby forming a pixel difference value. Typically, the motion estimation component 221 performs motion estimation for the lumens component, and the motion compensation component 219 uses motion vectors calculated based on the lumens component for both the chromens and lumens components. The predicted block and residual block are then transferred to the transformation scaling quantization component 213.

[0082] The segmented video signal 201 is also transmitted to the intra-picture estimation component 215 and the intra-picture prediction component 217. Similar to the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated, but are shown separately for conceptual purposes. As previously mentioned, the intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block based on the block in the current frame, instead of inter-predicting from frame to frame by the motion estimation component 221 and the motion compensation component 219. Specifically, the intra-picture estimation component 215 determines the intra-prediction mode to use for encoding the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode for encoding the current block from several proven intra-prediction modes. The selected intra-prediction mode is then forwarded to the header formatting CABAC component 231 for encoding.

[0083] For example, the intrapicture estimation component 215 calculates rate distortion values ​​for various proven intra-prediction modes using rate distortion analysis and selects the intra-prediction mode with the best rate distortion characteristics from among the proven modes. Rate distortion analysis typically determines the amount of distortion (or error) between the encoded block and the original unencoded block that was encoded to generate the encoded block, and the bitrate (e.g., number of bits) used to generate the encoded block. The intrapicture estimation component 215 determines which intra-prediction mode exhibits the best rate distortion value for the block by calculating a ratio from the distortion and rate of various encoded blocks. In addition, the intrapicture estimation component 215 may be configured to code depth blocks of the depth map using a depth modeling mode (DMM) based on rate distortion optimization (RDO).

[0084] The intrapicture prediction component 217, when running on an encoder, can generate residual blocks from prediction blocks based on a selected intrapicture prediction mode determined by the intrapicture estimation component 215, or, when running on a decoder, reads residual blocks from the bitstream. Residual blocks contain the difference in values ​​between the prediction blocks and the original blocks, expressed as a matrix. The residual blocks are then transferred to the transformation scaling quantization component 213. The intrapicture estimation component 215 and the intrapicture prediction component 217 can act on both the luma and chroma components.

[0085] The transformation scaling quantization component 213 is configured to further compress the residual blocks. The transformation scaling quantization component 213 applies a transformation, such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transformation, to the residual blocks to generate video blocks containing residual transformation coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transformations can also be used. The transformation can convert residual information from the pixel value domain to a transformation domain, such as the frequency domain. The transformation scaling quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling applies a scaling factor to the residual information, thereby quantizing various frequency information at different granularities, which can affect the final visual quality of the restored video. Furthermore, the transformation scaling quantization component 213 is configured to quantize the transformation coefficients to further reduce the bitrate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be changed by adjusting the quantization parameters. In some cases, the transformation scaling quantization component 213 can then perform a scan of the matrix containing the quantized transformation coefficients. The quantized transformation coefficients are encoded in a bitstream and then transferred to the header formatting CABAC component 231.

[0086] The inverse scaling component 229 applies the inverse operations of the transform-scaling-quantization component 213 to assist in motion estimation. The inverse scaling component 229 reconstructs the residual block in pixel region by applying inverse scaling, transform, and / or quantization, for example, to be used later as a reference block that could be a predicted block for another current block. The motion estimation component 221 and / or motion compensation component 219 can compute the reference block by adding the residual block to the corresponding predicted block used for motion estimation of a later block / frame. Filters are applied to the reconstructed reference block to mitigate artifacts that occur during scaling, quantization, and transform; otherwise, such artifacts could lead to inaccurate predictions (and further artifacts) when subsequent blocks are predicted.

[0087] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual blocks and / or the restored image blocks. For example, to restore the original image blocks, the transformed residual blocks from the inverse scaling component 229 can be combined with the corresponding predicted blocks from the intra-picture prediction component 217 and / or the motion compensation component 219. Filters can then be applied to the restored image blocks. In some examples, the filters can be applied to the residual blocks instead. As with the other components in Figure 2, the filter control analysis component 227 and the in-loop filter component 225 may be highly integrated and run together, but are depicted separately for conceptual purposes. Filters applied to the restored reference blocks are applied to a specific spatial region and include several parameters that adjust how such filters are applied. The filter control analysis component 227 analyzes the restored reference blocks to determine the circumstances under which such filters should be applied and sets the appropriate parameters. Such data is transferred to the header formatting CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on filter control data. These filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be applied, as examples, in the spatial / pixel domain (e.g., on a reconstructed pixel block) or in the frequency domain.

[0088] When operating as an encoder, the filtered, restored image blocks, residual blocks, and / or predicted blocks are stored in the decoded picture buffer component 223 for later use in the motion estimation described above. When operating as a decoder, the decoded picture buffer component 223 stores the restored and filtered blocks and transfers them to the display as part of the output video signal. The decoded picture buffer component 223 may be any memory device capable of storing the predicted blocks, residual blocks, and / or restored image blocks.

[0089] The header formatting CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to the decoder. Specifically, the header formatting CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. Furthermore, prediction data, including intra-prediction and motion data, as well as residual data in the form of quantized conversion coefficient data, are all encoded into a bitstream. The final bitstream contains all the information necessary for the decoder to reconstruct the original segmented video signal 201. Such information may include an intra-prediction mode index table (also called a codeword mapping table), definitions of encoding contexts for various blocks, indications for the most likely intra-prediction mode, and indications for segmentation information. Such data may be encoded using entropycoding. For example, information can be encoded using context-adaptive variable-length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), stochastic interval partitioned entropy (PIPE) coding, or other entropy coding techniques. After entropy coding, the coded bitstream can be transmitted to another device (e.g., a video decoder) or stored in an archive for later transmission or retrieval.

[0090] Figure 3 is a block diagram showing an exemplary video encoder 300. The video encoder 300 may be used to perform the encoding function of the codec system 200 and / or to perform steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 divides the input video signal, and the resulting divided video signal 301 is substantially the same as the divided video signal 201. The divided video signal 301 is then compressed by the components of the encoder 300 and encoded into a bitstream.

[0091] Specifically, the segmented video signal 301 is transferred to the intra-picture prediction component 317 for intra-prediction. The intra-picture prediction component 317 may be substantially the same as the intra-picture estimation component 215 and the intra-picture prediction component 217. The segmented video signal 301 is also transferred to the motion compensation component 321 for intra-prediction based on the reference block in the decoded picture buffer component 323. The motion compensation component 321 may be substantially the same as the motion estimation component 221 and the motion compensation component 219. The prediction block and residual block from the intra-picture prediction component 317 and the motion compensation component 321 are transferred to the transformation quantization component 313 for transformation and quantization of the residual block. The transformation quantization component 313 may be substantially the same as the transformation scaling quantization component 213. The transformed and quantized residual block and the corresponding prediction block (along with the associated control data) are transferred to the entropycoding component 331 for coding into a bitstream. The entropic coding component 331 may be substantially the same as the header formatting CABAC component 231.

[0092] The transformed and quantized residual blocks and / or corresponding predicted blocks are also transferred from the transform quantization component 313 to the inverse transform quantization component 329 for restoration to a reference block used by the motion compensation component 321. The inverse transform quantization component 329 may be substantially the same as the inverse scaling component 229. In some cases, the in-loop filter in the in-loop filter component 325 is also applied to the residual blocks and / or the restored reference block. The in-loop filter component 325 may be substantially the same as the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may contain multiple filters as described with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded picture buffer component 323 for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially the same as the decoded picture buffer component 223.

[0093] Figure 4 is a block diagram showing an exemplary video decoder 400. The video decoder 400 may be used to perform the decoding function of the codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of the operation method 100. The decoder 400 receives a bitstream from, for example, the encoder 300 and generates an output video signal restored based on the bitstream for display to the end user.

[0094] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 can use header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes information desirable for decoding the video signal, such as general control data, filter control data, segmentation information, motion data, prediction data, and quantized conversion coefficients from the residual blocks. The quantized conversion coefficients are transferred to the inverse quantization component 429 for reconstruction to the residual blocks. The inverse quantization component 429 may be similar to the inverse quantization component 329.

[0095] The restored residual blocks and / or predicted blocks are transferred to the intra-picture prediction component 417 for restoration to image blocks based on the intra-prediction operation. The intra-picture prediction component 417 may be the same as the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 uses a prediction mode to locate the reference block in the frame and applies the residual block to the result to restore the intra-predicted image block. The restored intra-predicted image block and / or residual block and the corresponding inter-prediction data are transferred to the decoded picture buffer component 423 via the in-loop filter component 425, which may be substantially the same as the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the restored image block, residual block, and / or predicted block, and such information is stored in the decoded picture buffer component 423. The restored image blocks are transferred from the decoded picture buffer component 423 to the motion compensation component 421 for interpretation. The motion compensation component 421 may be substantially the same as the motion estimation component 221 and / or motion compensation component 219. Specifically, the motion compensation component 421 generates a prediction block using the motion vector from the reference block and applies the residual block to the result to restore the image block. The thus restored block may also be transferred to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 continues to store further restored image blocks, which can be restored into frames by segmentation information. Such frames may also be arranged in a sequence. This sequence is output to the display as a restored output video signal.

[0096] Figure 5 is a schematic diagram showing an exemplary HRD 500. The HRD 500 may be used with an encoder such as a codec system 200 and / or encoder 300. The HRD 500 may inspect the bitstream generated in step 109 of method 100 before the bitstream is transferred to a decoder such as a decoder 400. In some examples, once the bitstream is encoded, the bitstream may be continuously transferred through the HRD 500. If a portion of the bitstream does not conform to the relevant constraints, the HRD 500 may indicate such anomalies to the encoder so that the encoder re-encodes the corresponding portion of the bitstream using a different mechanism.

[0097] The HRD 500 includes a virtual stream scheduler (HSS) 541, a component configured to perform a virtual distribution mechanism. The virtual distribution mechanism is used to check the conformity of a bitstream or decoder with respect to the timing and data flow of the bitstream 551 input to the HRD 500. For example, the HSS 541 may receive the bitstream 551 output from an encoder and manage the process of conformity testing of the bitstream 551. In a specific example, the HSS 541 may control the rate at which coded pictures pass through the HRD 500 and verify that the bitstream 551 does not contain non-conforming data.

[0098] HSS 541 can transfer bitstream 551 to CPB 543 at a predetermined rate. HRD 500 can manage the data in decoding unit (DU) 553. DU 553 is an access unit (AU), or a subset of AUs and associated non-video coding layer (VCL) network abstraction layer (NAL) units. Specifically, an AU contains one or more pictures associated with the output time. For example, an AU may contain a single picture in a single layer bitstream, or a picture per layer in a multi-layer bitstream. Each picture in an AU may be divided into slices, each contained in the corresponding VCL NAL unit. Thus, DU 553 can contain one or more pictures, one or more slices of pictures, or a combination thereof. Parameters used to decode AUs, pictures, and / or slices may also be contained in non-VCL NAL units. Therefore, DU 553 contains non-VCL NAL units containing the data necessary to support the decoding of VCL NAL units within DU 553. CPB 543 is the first-in, first-out buffer for HRD 500. CPB 543 contains DU 553 which holds the video data in decoding order. CPB 543 stores the video data for use during bitstream conformance verification.

[0099] The CPB 543 forwards the DU 553 to the decoding process component 545. The decoding process component 545 is a component conforming to the VVC standard. For example, the decoding process component 545 may emulate a decoder 400 used by an end user. The decoding process component 545 decodes the DU 553 at a rate achievable by an exemplary end-user decoder. If the decoding process component 545 cannot decode the DU 553 fast enough to prevent an overflow of the CPB 543, the bitstream 551 is not conforming to the standard and requires re-encoding.

[0100] The decoding process component 545 decodes the DU 553 and generates a decoded DU 555. The decoded DU 555 contains the decoded picture. The decoded DU 555 is transferred to the DPB 547. The DPB 547 may be substantially similar to the decoded picture buffer components 223, 323, and / or 423. For the purpose of assisting interpretation, the picture taken from the decoded DU 555 and marked for use as a reference picture 556 is returned to the decoding process component 545 to assist in further decoding. The DPB 547 outputs the decoded video sequence as a series of pictures 557. Picture 557 is a restored picture that typically reflects the picture encoded into the bitstream 551 by the encoder.

[0101] Picture 557 is transferred to the output cropping component 549. The output cropping component 549 is configured to apply a conformance cropping window to picture 557. This yields the output cropped picture 559. The output cropped picture 559 is a fully restored picture. Thus, the output cropped picture 559 mimics what the end user would see when decoding the bitstream 551. In this way, the encoder can examine the output cropped picture 559 to confirm that the encoding is good.

[0102] The HRD 500 is initialized based on HRD parameters in bitstream 551. For example, the HRD 500 can read HRD parameters from VPS, SPS, and / or SEI messages. The HRD 500 can then perform conformity testing operations on bitstream 551 based on the information in such HRD parameters. Specifically, the HRD 500 can determine one or more CPB delivery schedules from the HRD parameters. The delivery schedules specify the timing of video data delivery to and from memory locations such as the CPB and / or DPB. Thus, the CPB delivery schedule specifies the timing of AU, DU 553, and / or picture delivery to and from the CPB 543. It should be noted that the HRD 500 can use a DPB delivery schedule in DPB 547 that is similar to the CPB delivery schedule.

[0103] Video can be coded into different layers and / or OLS for use by decoders with varying levels of hardware capabilities and for various network conditions. CPB distribution schedules are selected to reflect these issues. Thus, upper-layer subbitstreams are specified with respect to optimal hardware and network conditions, and therefore, the upper layers can receive one or more CPB distribution schedules that use a large amount of memory in CPB 543 and a short delay for the transfer of DU 553 toward DPB 547. Similarly, lower-layer subbitstreams are specified with respect to limited decoder hardware capabilities and / or poor network conditions. Thus, the lower layers can receive one or more CPB distribution schedules that use a small amount of memory in CPB 543 and a longer delay for the transfer of DU 553 toward DPB 547. The OLS, layers, sublayers, or combinations thereof can then be tested according to the corresponding distribution schedules to ensure that the resulting subbitstreams can be correctly decoded under the expected conditions with respect to the subbitstreams. Therefore, the HRD parameters in bitstream 551 may include data sufficient to indicate the CPB delivery schedule and enable HRD 500 to determine the CPB delivery schedule and associate it with the corresponding OLS, layer, and / or sublayer.

[0104] Figure 6 is a schematic diagram showing an example of a multilayer video sequence 600. The multilayer video sequence 600 can be encoded by an encoder such as codec system 200 and / or encoder 300, for example, according to method 100, and decoded by a decoder such as codec system 200 and / or decoder 400. Furthermore, the multilayer video sequence 600 can be checked for compliance with standards by an HRD such as HRD 500. The multilayer video sequence 600 is included to illustrate an example of the application of layers within a coded video sequence. The multilayer video sequence 600 is any video sequence using multiple layers, such as layer N 631 and layer N+1 632.

[0105] In one example, a multilayer video sequence 600 can use interlayer prediction 621. Interlayer prediction 621 is applied between pictures 611, 612, 613, 614 and pictures 615, 616, 617, 618 in different layers. In the illustrated example, pictures 611, 612, 613, and 614 are part of layer N+1 632, and pictures 615, 616, 617, and 618 are part of layer N 631. Layers such as layer N 631 and / or layer N+1 632 are groups of pictures all associated with similar values ​​of characteristics such as similar size, quality, resolution, signal-to-noise ratio, and capability. A layer can be formally defined as a set of VCL NAL units that share the same layer ID and associated non-VCL NAL units. A VCL NAL unit is a NAL unit that is coded to contain video data, such as a coding slice of a picture. A non-VCL NAL unit is a NAL unit that contains non-video data such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations.

[0106] In the illustrated example, layer N+1 632 is associated with a larger image size than layer N 631. Therefore, in this example, pictures 611, 612, 613, and 614 in layer N+1 632 have larger picture sizes (e.g., greater height and width, and thus more samples) than pictures 615, 616, 617, and 618 in layer N 631. However, such pictures may be separated between layer N+1 632 and layer N 631 by other characteristics. Although only two layers, layer N+1 632 and layer N 631, are shown, a set of pictures can be separated into any number of layers based on relevant characteristics. Furthermore, layers N+1 632 and N 631 may be indicated by layer IDs. A layer ID is a data item associated with a picture, indicating that the picture is part of the indicated layer. Therefore, each picture 611-618 may be associated with a corresponding layer ID to indicate which layer N+1 632 or layer N 631 contains the corresponding picture. For example, the layer ID may include a NAL unit header layer identifier (nuh_layer_id), which is a syntax element that specifies the identifier of the layer containing NAL units (e.g., slices and / or parameters of the pictures in the layer). Layers associated with lower quality / smaller image size / smaller bitstream size, such as layer N 631, are generally assigned a lower layer ID and are called lower layers. Furthermore, layers associated with higher quality / larger image size / larger bitstream size, such as layer N+1 632, are generally assigned a higher layer ID and are called upper layers.

[0107] Pictures 611-618 in different layers 631-632 are configured to be displayed in alternative forms. For example, the decoder may decode and display picture 615 at the current display time if a smaller picture is desired, or the decoder may decode and display picture 611 at the current display time if a larger picture is desired. Thus, pictures 611-614 in the upper layer N+1 632 contain substantially the same image data as the corresponding pictures 615-618 in the lower layer N 631 (regardless of the difference in picture size). Specifically, picture 611 contains substantially the same image data as picture 615, picture 612 contains substantially the same image data as picture 616, and so on.

[0108] Pictures 611-618 can be coded by referencing other pictures 611-618 in the same layer N 631 or N+1 632. Coding a picture by referencing other pictures in the same layer yields an interprediction 623. The interprediction 623 is indicated by a solid arrow. For example, picture 613 may be coded using an interprediction 623 that references one or two of pictures 611, 612 and / or 614 in layer N+1 632, in which case one picture is referenced for a unidirectional interprediction and / or two pictures are referenced for a bidirectional interprediction. Furthermore, picture 617 may be coded using an interprediction 623 that references one or two of pictures 615, 616 and / or 618 in layer N 631, in which case one picture is referenced for a unidirectional interprediction and / or two pictures are referenced for a bidirectional interprediction. When performing interprediction 623, if one picture is used as a reference to another picture within the same layer, that picture may be called a reference picture. For example, picture 612 may be a reference picture used to code picture 613 according to interprediction 623. Interprediction 623 may also be called intralayer prediction in a multilayer context. Thus, interprediction 623 is a mechanism for coding a sample of the current picture by referencing an indicated sample in a different reference picture, when the reference picture and the current picture are in the same layer.

[0109] Pictures 611-618 can also be coded by referencing other pictures 611-618 in different layers. This process is known as interlayer prediction 621 and is indicated by a dashed arrow. Interlayer prediction 621 is a mechanism for coding a sample of the current picture by referencing a specified sample in a reference picture when the current picture and the reference picture are in different layers and therefore have different layer IDs. For example, a picture in a lower layer N 631 can be used as a reference picture to code the corresponding picture in a higher layer N+1 632. As a specific example, picture 611 may be coded by referencing picture 615 according to interlayer prediction 621. In such a case, picture 615 is used as the interlayer reference picture. The interlayer reference picture is the reference picture used in interlayer prediction 621. In most cases, the interlayer prediction 621 is constrained to use only interlayer reference pictures that are in lower layers, such as picture 615, and that are contained within the same AU 627 as the current picture, such as picture 611. If multiple layers (e.g., more than two layers) are available, the interlayer prediction 621 can encode / decode the current picture based on multiple interlayer reference pictures that are at a lower level than the current picture.

[0110] The video encoder can encode pictures 611-618 using a multilayer video sequence 600 via many different combinations and / or permutations of inter-prediction 623 and inter-layer prediction 621. For example, picture 615 may be coded according to intra-prediction. Then, pictures 616-618 may be coded according to inter-prediction 623 by using picture 615 as a reference picture. Furthermore, picture 611 may be coded according to inter-layer prediction 621 by using picture 615 as an inter-layer reference picture. Then, pictures 612-614 may be coded according to inter-prediction 623 by using picture 611 as a reference picture. Thus, the reference picture can function as both a single-layer reference picture and an inter-layer reference picture for different coding mechanisms. By coding the picture of the upper layer N+1 632 based on the picture of the lower layer N 631, the upper layer N+1 632 can avoid using intra-prediction, which has much lower coding efficiency than inter-prediction 623 and inter-layer prediction 621. Thus, the insufficient coding efficiency of intra-prediction may be limited to the smallest / lowest quality picture, and therefore limited to coding the smallest amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures can be indicated by entries in the reference picture list contained in the reference picture list structure.

[0111] Furthermore, pictures 611-618 may be included in Access Unit (AU) 627. AU 627 is a set of coded pictures that are included in different layers and associated with the same output time during decoding. Therefore, coded pictures within the same AU 627 are scheduled to be output from the DPB in the decoder at the same time. For example, pictures 614 and 618 are in the same AU 627. Pictures 613 and 617 are in a different AU 627 than pictures 614 and 618. In an alternative example, pictures 614 and 618 within the same AU 627 may be displayed. For example, picture 618 may be displayed when a smaller picture size is desired, and picture 614 may be displayed when a larger picture size is desired. If a larger picture size is desired, picture 614 is output, and picture 618 is used only for interlayer prediction 621. In this case, picture 618 is discarded without being output once interlayer prediction 621 is complete.

[0112] AU 627 may be further divided into one or more Picture Units (PUs) 628. A PU 628 is a subset of AU 627 containing a single coded picture. A PU 628 can be formally defined as a set of NAL units that are related to each other according to a specified classification rule, are consecutive in decoding order, and each contains exactly one coded picture. It should be noted that a PU 628 may be referred to as a Decoding Unit (DU) when described in terms of HRD and / or related conformity testing.

[0113] It should also be noted that pictures 611-618, and therefore AU 627 and PU 628, are each associated with a TemporalId 629. TemporalId 629 is an identifier indicating the relative position of a NAL unit in a video sequence. Pictures and / or PU 628 in the same AU 627 are associated with the same value of TemporalId 629. For example, the first AU 627 in a sequence may contain a TemporalId 629 of 0, and subsequent AU 627s may contain successively increasing TemporalId 629s. Non-VCL NAL units may also be associated with TemporalId 629. For example, a parameter set may be contained in an AU 627 and associated with one or more pictures within that AU 627. In such cases, the TemporalId 629 of the parameter set may be less than or equal to the TemporalId 629 of the AU 627.

[0114] Figure 7 is a schematic diagram showing an example of a bitstream 700. For example, the bitstream 700 may be generated by the codec system 200 and / or encoder 300 for decoding by the codec system 200 and / or decoder 400 according to method 100. Furthermore, the bitstream 700 may include a multilayer video sequence 600. In addition, the bitstream 700 may include various parameters for controlling the operation of an HRD, such as an HRD 500. Based on such parameters, the HRD can check whether the bitstream 700 conforms to the standard before sending it to the decoder for decoding.

[0115] Bitstream 700 includes a VPS 711, one or more SPS 713s, multiple Picture Parameter Sets (PPS) 715s, multiple Adaptive Parameter Sets (APS) 716s, multiple Picture Headers 718s, multiple Slice Headers 717s, Image Data 720, and SEI Messages 719. The VPS 711 contains data relating to the entire bitstream 700. For example, the VPS 711 may include data-related OLSs, layers, and / or sublayers used in bitstream 700. The SPS 713 contains sequence data common to all pictures in the coded video sequences included in bitstream 700. For example, each layer may contain one or more coded video sequences, and each coded video sequence may refer to the SPS 713 for its corresponding parameters. Parameters in the SPS 713 may include picture sizing, bit depth, coding tool parameters, bitrate limits, etc. Each sequence refers to an SPS 713, but it should be noted that in some examples, a single SPS 713 can contain data for multiple sequences. The PPS 715 contains parameters that apply to the entire picture. Therefore, each picture in a video sequence may refer to a PPS 715. It should be noted that in some examples, while each picture refers to a PPS 715, a single PPS 715 can contain data for multiple pictures. For example, several similar pictures may be coded according to similar parameters. In such cases, a single PPS 715 may contain data for such similar pictures. The PPS 715 may indicate coding tools, quantization parameters, offsets, etc., available for slices within the corresponding picture. A PPS 715 may be associated with a particular set of pictures 725, and a bitstream 700 may contain many PPS 715s. Therefore, each PPS 715 may contain a PPS ID 732 that identifies the PPS 715 for reference by other syntax elements.In some cases, PPS ID 732 may be stored in the PPS picture parameter set ID (pps_pic_parameter_set_id) syntax element within PPS 715.

[0116] An APS 716 is a syntax structure containing syntax elements / parameters that apply to one or more slices 727 within one or more pictures 725. Such correlations can be determined based on the syntax elements found in the slice header 717 associated with the slices 727. For example, an APS 716 can apply to at least one but fewer than all slices 727 in a first picture 721, at least one but fewer than all slices 727 in a second picture 725, and so on. An APS 716 can be separated into several types based on the parameters it contains. Such types can include Adaptive Loop Filter (ALF) APS, Luma Mapping with Chroma Scaling (LMCS) APS, and Scaling List (Scaling) APS. An ALF is an adaptive block-based filter that contains a transfer function controlled by variable parameters and uses feedback from a feedback loop to refine the transfer function. Furthermore, ALFs are used to correct coding artifacts (e.g., errors) that result from block-based coding, such as blurring and ringing artifacts. Therefore, the ALF parameters included in the ALF APS may include parameters selected by the encoder to cause ALF to remove block-based coding artifacts during decoding in the decoder. LMCS is a process applied as part of the decoding process, mapping luminous samples to specific values ​​and potentially applying scaling operations to the values ​​of chroma samples. The LMCS tool can reshape luminous components based on their mapping to corresponding chroma components to reduce rate distortion. Therefore, the LMCS APS includes parameters selected by the encoder to cause the LMCS tool to reshape luminous components. The scaling list APS includes the coding tool parameters associated with the quantization matrix used by the specified filter.Therefore, APS 716 may include parameters used to apply various filters to coded slices 727 during conformance testing in HRD and / or decoding in the decoder. APS 716 may be associated with a particular set of slices 727, and bitstream 700 may contain many APS 716s. Thus, each APS 716 may include an APS ID 731 that identifies the APS 716 for reference by other syntax elements. In some examples, the APS ID 731 may be stored in the APS adaptation parameter set ID (aps_adaptation_parameter_set_id) syntax element within the APS 716.

[0117] The picture header 718 is a syntax structure containing syntax elements that apply to all slices 727 of the coded picture 725. For example, the picture header 718 may contain picture sequence count information, reference picture data, data related to intra-random access point (IRAP) pictures, and data related to filter application for picture 725. A PU may contain exactly one picture header 718 and exactly one picture 725. Thus, the bitstream 700 can contain exactly one picture header 718 for each picture 725. The picture header 718 may also specify a PPS ID 715 that indicates which PPS 732 contains the parameters related to the picture 725 corresponding to the picture header 718. In a specific example, the PPS ID 732 may be stored in the picture header 718 as a picture header (PH) picture parameter set ID (ph_pic_parameter_set_id). The picture header 718 may also specify an APS ID 731 that indicates which APS 716 contains parameters related to the picture 725 corresponding to the picture header 718. In a specific example, the APS ID 731 may be stored in the picture header 718 as either a PH scaling list APS ID (ph_scaling_list_aps_id) or a PH LMCS APS ID (ph_lmcs_aps_id), depending on the example.

[0118] The slice header 717 contains parameters specific to each slice 727 within the picture 725. Therefore, in a video sequence, there may be one slice header 717 for each slice 727. The slice header 717 may include slice type information, filtering information, prediction weights, tile entry points, deblocking parameters, etc. In some cases, the syntax elements may be the same for all slices 727 within the picture 725. To reduce redundancy, the picture header 718 and slice header 717 may share certain types of information. For example, certain parameters (e.g., filtering parameters) may be included in the picture header 718 if they apply to the entire picture 725, or in the slice header 717 if they apply to a group of slices 727 that are a subset of the entire picture 725. Additionally, the slice header 717 may specify an APS ID 731 indicating which APS 716 contains parameters related to the slice 727 corresponding to the slice header 717. In a specific example, APS ID 731 may be stored in slice header 717 as, as appropriate, a slice header (SH) ALF APS ID (sh_alf_aps_id_luma) for the luma component or an SH ALF APS ID (sh_alf_aps_id_chroma) for the chroma component.

[0119] Image data 720 includes video data encoded according to interpretation and / or intrapretation, as well as the corresponding transformed and quantized residual data. For example, image data 720 may include layer 723, picture 725, and / or slice 727. Layer 723 is a set of VCL NAL units and associated non-VCL NAL units that share specified characteristics (e.g., common resolution, frame rate, image size, etc.) as indicated by a layer ID such as nuh_layer_id 729. For example, layer 723 may include a set of picture 725 that share the same nuh_layer_id 729. Layer 723 may be substantially the same as layers 631 and / or 632. nuh_layer_id 729 is a syntax element that specifies an identifier for layer 723 containing at least one NAL unit. For example, the lowest quality layer 723, known as the base layer, can have the lowest value of nuh_layer_id 729, with an increasing value of nuh_layer_id 729 for higher quality layers 723. Therefore, lower layers 723 are layers 723 with a small value of nuh_layer_id 729, and higher layers are layers 723 with a large value of nuh_layer_id 729.

[0120] A picture 725 is an array of luma samples and / or chroma samples that generate a frame or its fields. For example, a picture 725 is a coded image that can be output for display or used to support the coding of other picture 725s for output. A picture 725 contains one or more slices 727. A slice 727 can be defined as an integer of complete tiles of picture 725 exclusively contained in a single NAL unit or an integer of consecutive complete coding tool unit (CTU) rows (e.g., within a tile). A slice 727 is further divided into CTUs and / or coding tree blocks (CTBs). A CTU is a group of samples of a given size that can be separated by a coding tree. A CTB is a subset of a CTU and contains the luma or chroma components of the CTU. A CTU / CTB is further divided into coding blocks based on the coding tree. A coding block can then be encoded / decoded according to a prediction mechanism.

[0121] SEI message 719 is a syntactic structure with specified semantics that conveys information not required by the decoding process to determine the values ​​of samples in the decoded picture. For example, SEI message 719 may include data to support the HRD process or other supporting data not directly related to decoding bitstream 700 in the decoder.

[0122] Bitstream 700 can be coded as a sequence of NAL units. A NAL unit is a container for video data and / or supporting syntax. A NAL unit can be a VCL NAL unit 747 or a non-VCL NAL unit. A VCL NAL unit 747 is a NAL unit coded to contain video data. Specifically, a VCL NAL unit 747 includes a slice 727 and an associated slice header 717. A non-VCL NAL unit is a NAL unit that contains non-video data such as syntax and / or parameters that support decoding of video data, performing conformance checks, or other operations. Non-VCL NAL units may include VPS NAL unit 741, SPS NAL unit 743, PPS NAL unit 745, APS NAL unit 746, PH NAL unit 748, and SEI NAL unit 749, which include VPS 711, SPS 713, PPS 715, APS 716, picture header 718, and SEI message 719, respectively. It should be noted that the above list of NAL units is typical and not exhaustive.

[0123] As mentioned above, pictures 725 in different layers 723 have different characteristics. For example, pictures 725 in different layers 723 may be encoded with different sizes, resolutions, frame rates, etc. This allows the encoder to send different layers 723 to the decoder according to the decoder's constraints. To perform this function, the encoder can encode all layers 723 into a single bitstream 700. Upon request, the encoder (or slicer in the content hosting server) can perform a sub-bitstream extraction process to remove irrelevant information from the bitstream 700 before transmission. The result is an extracted bitstream 700 containing only the data for decoding the requested layer 723. This process can be achieved by using NAL units. Specifically, each NAL unit contains a nuh_layer_id 729. Therefore, nuh_layer_id 729 is included in VPS NAL unit 741, SPS NAL unit 743, PPS NAL unit 745, APS NAL unit 746, PH NAL unit 748, VCL NAL unit 747, and SEI NAL unit 749. The subbitstream extraction process receives a target range describing the IDs of the requested layers 723. The subbitstream extraction process can remove all NAL units with nuh_layer_id 729 that are outside the target range. This leaves only the NAL units with nuh_layer_id 729 that are within the target range and remain in the bitstream 700 extracted for transmission to the decoder. A problem with this technique is that some NAL units are related to many layers 723. For example, PPS NAL unit 745, which includes PPS 715, and APS NAL unit 746, which includes APS 716, may contain data relating to one or more layers 723. If the nuh_layer_id 729 of PPS NAL unit 745 or APS NAL unit 746 is outside the target range, PPS 715 and / or APS 716 can be removed by the subbitstream extraction process.If this occurs, PPS 715 and / or APS 716 will not be sent to the decoder. In this case, the decoder will not have enough data to decode the extracted bitstream 700.

[0124] Therefore, bitstream 700 imposes constraints on PPS 715 and APS 716 to prevent PPS NAL unit 745 and APS NAL unit 746 from being inadvertently removed from bitstream 700 during the sub-bitstream extraction process. Specifically, PPS 715 and APS 716 are referenced by picture header 718 and / or slice header 717 by using PPS ID 732 and APS ID 731, respectively. PPS 715 and / or PPS NAL unit 745 are constrained to contain a value of nuh_layer_id 729 that is less than or equal to the value of nuh_layer_id 729 of all VCL NAL units 747 that reference PPS 715 within PPS NAL unit 745. The sub-bitstream extraction process can act on groups of layers based on requirements for values ​​less than or equal to nuh_layer_id 729. Therefore, the constraint ensures that when PPS 715 is associated with VCL NAL unit 747 with a nuh_layer_id 729 value within the requested range, PPS NAL unit 745 contains a nuh_layer_id 729 value within the requested range. Thus, a particular VCL NAL unit 747 does not refer to PPS NAL unit 745 unless the nuh_layer_id 729 value of PPS NAL unit 745 is less than or equal to the nuh_layer_id 729 value of that particular VCL NAL unit 747. In a particular implementation, the nuh_layer_id 729 of PPS 715 is constrained to be less than or equal to the nuh_layer_id 729 in VCL NAL unit 747 associated with picture 725 when ph_pic_parameter_set_id in picture header 718 is set to be equal to pps_pic_parameter_set_id in PPS 725.

[0125] Similarly, APS 716 and / or APS NAL unit 746 are constrained to contain a value of nuh_layer_id 729 that is less than or equal to the value of nuh_layer_id 729 of all VCL NAL units 747 that reference APS 716 within APS NAL unit 746. The subbitstream extraction process can act on groups of layers based on requests for values ​​less than or equal to nuh_layer_id 729. Thus, the constraint ensures that when APS 716 is associated with a VCL NAL unit 747 that has a nuh_layer_id 729 value within the requested range, APS NAL unit 746 contains a value of nuh_layer_id 729 that is within the requested range. Therefore, a particular VCL NAL unit 747 does not reference APS NAL unit 746 unless the value of nuh_layer_id 729 of APS NAL unit 746 is less than or equal to the value of nuh_layer_id 729 of that particular VCL NAL unit 747. In a particular embodiment, the nuh_layer_id 729 of APS 716 is constrained to be less than or equal to the nuh_layer_id 729 in the VCL NAL unit 747 associated with slice 727 when ph_scaling_list_aps_id, ph_lmcs_aps_id, sh_alf_aps_id_luma, and / or sh_alf_aps_id_chroma in picture header 718 and / or slice header 717 are set to equal to aps_adaptation_parameter_set_id in APS 716.

[0126] By ensuring that PPS NAL units 745 and APS NAL units 746 have a lower nuh_layer_id 729 than all VCL NAL units 747 that reference them, the sub-bitstream extraction process, which also retains VCL NAL units 747 that reference PPS / APS, cannot remove PPS 715 and APS 716. Therefore, sub-bitstream extraction-related errors are avoided. As a result, the functionality of the encoder and decoder is improved. Furthermore, coding efficiency is enhanced by correctly removing unnecessary NAL units, thereby reducing the use of processor, memory, and / or network signaling resources in both the encoder and decoder.

[0127] Here, the information mentioned above will be explained in more detail. Layered video coding is also called scalable video coding or video coding with scalability. Scalability in video coding can be supported by using multilayer coding techniques. A multilayer bitstream comprises a base layer (BL) and one or more enhancement layers (EL). Examples of scalability include spatial scalability, signal-to-noise ratio (SNR) scalability, multiview scalability, and frame rate scalability. When multilayer coding techniques are used, a picture or part thereof may be coded without using a reference picture (intra-prediction), by referencing a reference picture in the same layer (inter-prediction), and / or by referencing a reference picture in another layer (inter-layer prediction). A reference picture used for inter-layer prediction of the current picture is called an inter-layer reference picture (ILRP). Figure 6 shows an example of multilayer coding for spatial scalability where pictures in different layers have different resolutions.

[0128] Several video coding families provide scalability support in profiles separated from profiles for single-layer coding. Scalable Video Coding (SVC) is a scalable extension of Advanced Video Coding (AVC) that provides support for spatial, temporal, and quality scalability. In SVC, each macroblock (MB) in an EL picture is signaled to indicate whether an EL MB is predicted using array blocks from lower layers. Predictions from array blocks can include textures, motion vectors, and / or coding modes. SVC implementations may not directly reuse AVC implementations that are not modified in their design. The SVC EL macroblock syntax decoding process is different from the AVC syntax decoding process.

[0129] Scalable HEVC (SHVC) is an extension of HEVC that provides support for spatial and quality scalability. Multiview HEVC (MV-HEVC) is an extension of HEVC that provides support for multiview scalability. 3D HEVC (3D-HEVC) is an extension of HEVC that provides support for more advanced and efficient 3D video coding than MV-HEVC. Temporal scalability can be included as an integral part of the single-layer HEVC codec. In the multi-layer extensions of HEVC, decoded pictures used for interlayer prediction are brought by the same AU only and are treated as Long-Term Reference Pictures (LTRPs). Such pictures are assigned a reference index in the reference picture list, along with other temporal reference pictures in the current layer. Interlayer prediction (ILP) is implemented at the prediction unit level by setting the value of the reference index to refer to the interlayer reference picture in the reference picture list. Spatial scalability involves resampling a reference picture or a portion thereof if the ILRP has a different spatial resolution than the current picture being encoded or decoded. Reference picture resampling can be implemented at either the picture level or the coding block level.

[0130] VVC can also support layered video coding. A VVC bitstream can contain multiple layers. All layers may be independent of each other. For example, each layer can be coded without using interlayer prediction. In this case, the layers are also called simulcast layers. In some cases, some layers are coded using ILP. Flags in the VPS can indicate whether a layer is a simulcast layer or whether some layers use ILP. If some layers use ILP, layer dependencies between layers are also signaled in the VPS. Unlike SHVC and MV-HEVC, VVC may not specify an OLS. The OLS contains a specified set of layers, in which case one or more layers in the set of layers are specified to be output layers. The output layers are the layers of the output OLS. In some implementations of VVC, if a layer is a simulcast layer, only one layer may be selected for decoding and output. In some implementations of VVC, when any layer uses ILP, the entire bitstream containing all layers is specified to be decoded. Also, some of these layers are specified to be output layers. The output layers may be shown as only the top layer, all layers, or the top layer plus the set of lower layers shown.

[0131] The aforementioned embodiments have several problems. For example, the nuh_layer_id value in SPS, PPS, and APS NAL units may not be properly constrained. Furthermore, the TemporalId value in SEI NAL units may not be properly constrained. Also, if resampling of the reference picture is enabled and the spatial resolution of the pictures in the CLVS is different, the NoOutputOfPriorPicsFlag setting may not be specified properly. In addition, some video coding systems cannot include suffix SEI messages in scalable nesting SEI messages. As another example, buffering period, picture timing, and decoding unit information SEI messages may contain analysis dependencies on VPS and / or SPS.

[0132] Generally, this disclosure describes video coding improvement techniques. The technical description is based on VVC. However, these techniques are also applicable to layered video coding based on other video codec specifications.

[0133] One or more of the aforementioned problems can be resolved as follows: The nuh_layer_id value in SPS, PPS, and APS NAL units is appropriately constrained herein. The TemporalId value in SEI NAL units is appropriately constrained herein. The NoOutputOfPriorPicsFlag setting is appropriately specified when resampling of the reference picture is enabled and the pictures in the CLVS have different spatial resolutions. Suffix SEI messages can be included in scalable nesting SEI messages. Parsing dependencies of BP, PT, and DUI SEI messages on a VPS or SPS can be removed by repeating the syntax element decoding_unit_hrd_params_present_flag in the BP SEI message syntax, decoding_unit_hrd_params_present_flag and decoding_unit_cpb_params_in_pic_timing_sei_flag in the PT SEI message syntax, and decoding_unit_cpb_params_in_pic_timing_sei_flag in the DUI SEI message syntax.

[0134] An example of the implementation of the aforementioned mechanism is as follows. An example of general NAL unit semantics is shown below.

[0135] nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit. The value of nuh_temporal_id_plus1 should not be equal to 0. The variable TemporalId can be derived as follows: TemporalId=nuh_temporal_id_plus1-1 When nal_unit_type is within the range of IDR_W_RADL or greater and RSV_IRAP_13 or less, TemporalId must be equal to 0. When nal_unit_type is equal to STSA_NUT, TemporalId must not be equal to 0.

[0136] The TemporalId value must be the same for all VCL NAL units in an access unit. The TemporalId value of a coded picture, layer access unit, or access unit may be the TemporalId value of the VCL NAL units in the coded picture, layer access unit, or access unit. The TemporalId value of a sublayer display may be the maximum value of the TemporalIds of all VCL NAL units within the sublayer display.

[0137] The value of TemporalId in a non-VCL NAL unit is constrained as follows: If nal_unit_type is equal to DPS_NUT, VPS_NUT, or SPS_NUT, then TemporalId must be equal to 0, and the TemporalId of the access unit containing the NAL unit must be equal to 0. Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, then TemporalId must be equal to 0. Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, then TemporalId must be equal to the TemporalId of the access unit containing the NAL unit. Otherwise, if nal_unit_type is equal to PPS_NUT or APS_NUT, then TemporalId must be greater than or equal to the TemporalId of the access unit containing the NAL unit. If a NAL unit is a non-VCL NAL unit, the value of TemporalId must be equal to the minimum TemporalId value of all access units to which the non-VCL NAL unit applies. If nal_unit_type is equal to PPS_NUT or APS_NUT, TemporalId may be greater than or equal to the TemporalId of the containing access unit, because all PPS and APS may be included at the beginning of the bitstream. Furthermore, the first coded picture has a TemporalId equal to 0.

[0138] An example of sequence parameter set RBSP semantics is as follows: The SPS RBSP must be available to the decoding process before it is referenced. The SPS may be contained in at least one access unit with a TemporalId equal to 0, or it may be provided via an external mechanism. The SPS NAL unit containing the SPS may be constrained to have a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit referencing the SPS.

[0139] An example of picture parameter set RBSP semantics is as follows: The PPS RBSP must be available to the decoding process before it is referenced. The PPS should be contained in or provided via an external mechanism in at least one access unit with a TemporalId less than or equal to the TemporalId of the PPS NAL unit. The PPS NAL unit containing the PPS RBSP should have a nuh_layer_id equal to the lowest nuh_layer_id value of the coding slice NAL unit referencing the PPS.

[0140] An example of the semantics of the adaptive parameter set is as follows: Each APS RBSP must be available to the decoding process before being referenced. Also, an APS should be contained in at least one access unit with a TemporalId less than or equal to the TemporalId of the coded slice NAL unit that references the APS or is provided via an external mechanism. An APS NAL unit can be shared by multiple layer pictures / slices. The nuh_layer_id of the APS NAL unit must be equal to the lowest nuh_layer_id value of the coding slice NAL unit that references the APS NAL unit. Alternatively, an APS NAL unit does not have to be shared by multiple layer pictures / slices. The nuh_layer_id of the APS NAL unit must be equal to the nuh_layer_id of the slice that references the APS.

[0141] In one example, the removal of a picture from the DPB before decoding the current picture is described as follows: The removal of a picture from the DPB before decoding the current picture (but after parsing the slice header of the first slice of the current picture) may occur during the CPB removal time of the first decoding unit of access unit n (which contains the current picture). This proceeds as follows: The decoding process for building the reference picture list is invoked, and the decoding process for marking the reference picture is invoked.

[0142] If the current picture is a coded layer video sequence start (CLVSS) picture that is not picture 0, the following ordered steps apply. For the decoder under test, the variable NoOutputOfPriorPicsFlag is derived as follows: If the values ​​of pic_width_in_luma_samples, pic_height_max_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[Htid] derived from the SPS differ from the values ​​of pic_width_in_luma_samples, pic_height_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[Htid] derived from the SPS referenced by the preceding picture, NoOutputOfPriorPicsFlag may be set to 1 by the decoder under test, regardless of the value of no_output_of_prior_pics_flag. Under these conditions, it may be preferable to set NoOutputOfPriorPicsFlag to be equal to no_output_of_prior_pics_flag, but it should be noted that in this case, the decoder under test should be able to set NoOutputOfPriorPicsFlag to 1. Otherwise, NoOutputOfPriorPicsFlag may be set to be equal to no_output_of_prior_pics_flag.

[0143] The value of NoOutputOfPriorPicsFlag derived for the decoder under test is applied to the HRD, so that when the value of NoOutputOfPriorPicsFlag is equal to 1, all picture storage buffers in the DPB are emptied without the output of the pictures they contain, and the DPB fullness is set to 0. If both of the following conditions are met for any picture k in the DPB, then all such picture k in the DPB are removed from the DPB: the picture k is marked as unused for reference, the picture k has a PictureOutputFlag equal to 0, or the corresponding DPB output time is less than or equal to the CPB removal time of the first decoding unit (indicated as decoding unit m) of the current picture n. This can occur if DpbOutputTime[k] is less than or equal to DuCpbRemovalTime[m]. For each picture removed from the DPB, the DPB fullness is decremented by 1.

[0144] In one example, the output and removal of pictures from the DPB is described as follows: The output and removal of pictures from the DPB before decoding the current picture (but after parsing the slice header of the first slice of the current picture) may occur when the first decoding unit of the access unit containing the current picture is removed from the CPB and proceeds as follows: The decoding process for building the reference picture list and the decoding process for marking the reference picture are invoked.

[0145] If the current picture is a CLVSS picture that is not picture 0, the following ordered steps apply. The variable NoOutputOfPriorPicsFlag is derived for the decoder under test as follows. If the values ​​of pic_width_max_in_luma_samples, pic_height_max_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[Htid] derived from the SPS differ from the values ​​of pic_width_in_luma_samples, pic_height_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[Htid] derived from the SPS referenced by the preceding picture, NoOutputOfPriorPicsFlag may be set to 1 by the decoder under test, regardless of the value of no_output_of_prior_pics_flag. Under these conditions, it is preferable to set NoOutputOfPriorPicsFlag to be equal to no_output_of_prior_pics_flag, but it should be noted that in this case, the decoder under test can set NoOutputOfPriorPicsFlag to 1. Otherwise, NoOutputOfPriorPicsFlag can be set to be equal to no_output_of_prior_pics_flag.

[0146] The value of NoOutputOfPriorPicsFlag derived for the decoder under test may be applied to the HRD as follows: If NoOutputOfPriorPicsFlag is equal to 1, all picture storage buffers in the DPB are emptied without outputting the pictures they contain, and the DPB fullness is set to 0. Otherwise (NoOutputOfPriorPicsFlag is equal to 0), all picture storage buffers containing pictures marked as not needed for output and not used for reference are emptied (without output), and all non-empty picture storage buffers in the DPB are emptied by repeatedly calling the bumping process, and the DPB fullness is set to 0.

[0147] Otherwise (the current picture is not a CLVSS picture), all picture storage buffers containing pictures marked as not needed for output and not used for reference are emptied (without output). For each picture storage buffer emptied, the DPB fullness is decremented by 1. The bumping process is repeatedly called, decrementing the DPB fullness by 1 for each further picture storage buffer emptied until any of the following conditions are no longer true: The condition is that the number of pictures in the DPB marked as needed for output is greater than sps_max_num_reorder_pics[Htid]. The other condition is that there is at least one picture in the DPB marked as not needed for output, where sps_max_latency_increase_plus1[Htid] is not equal to 0 and the associated variable PicLatencyCount is greater than or equal to SpsMaxLatencyPictures[Htid]. Another condition is that the number of pictures in the DPB is greater than or equal to SubDpbSize[Htid].

[0148] An example of a typical SEI message syntax is as follows:

[0149] [Table 1]

[0150] An example of scalable nesting SEI message syntax is as follows:

[0151] [Table 2]

[0152] An example of scalable nesting SEI message semantics is as follows: Scalable nesting SEI messages provide a mechanism for associating an SEI message with a specific layer associated with a particular OLS or a specific layer not associated with an OLS. A scalable nesting SEI message contains one or more SEI messages. SEI messages contained within a scalable nesting SEI message are also called scalably nested SEI messages. Bitstream compliance may require the following restrictions to apply when an SEI message is contained within a scalable nesting SEI message.

[0153] SEI messages with payloadType equal to 132 (decoded picture hash) or 133 (scalable nesting) should not be included in a scalable nesting SEI message. If a scalable nesting SEI message includes a buffering period, picture timing, or decoding unit information SEI message, the scalable nesting SEI message should not include any other SEI messages whose payloadType is not equal to 0 (buffering period), 1 (picture timing), or 130 (decoding unit information).

[0154] Furthermore, bitstream compliance may also require the following restrictions to apply to the value of nal_unit_type of an SEI NAL unit containing a scalable nesting SEI message: If a scalable nesting SEI message contains an SEI message with a payloadType equal to 0 (buffering period), 1 (picture timing), 130 (decoded unit information), 145 (dependent RAP representation), or 168 (frame field information), the SEI NAL unit containing the scalable nesting SEI message should have a nal_unit_type equal to PREFIX_SEI_NUT. If a scalable nesting SEI message contains an SEI message with a payloadType equal to 132 (decoded picture hash), the SEI NAL unit containing the scalable nesting SEI message should have a nal_unit_type set equal to SUFFIX_SEI_NUT.

[0155] The nesting_ols_flag can be set to equal to 1 to specify that scalably nested SEI messages apply to a specific layer associated with a particular OLS. The nesting_ols_flag can be set to equal to 0 to specify that scalably nested SEI messages apply to a specific layer in general (e.g., not associated with an OLS).

[0156] Bitstream compliance may require the following restrictions to apply to the value of nesting_ols_flag: If a scalable nesting SEI message contains an SEI message with payloadType equal to 0 (buffering period), 1 (picture timing), or 130 (decoding unit information), the value of nesting_ols_flag must be equal to 1. If a scalable nesting SEI message contains an SEI message with payloadType equal to a value in VclAssociatedSeiList, the value of nesting_ols_flag must be equal to 0.

[0157] The value of nesting_num_olss_minus1 plus 1 specifies the number of OLS to which the scalably nested SEI message applies. The value of nesting_num_olss_minus1 can be in the range of 0 or greater and TotalNumOlss-1 or less. nesting_ols_idx_delta_minus1[i] is used to derive the variable NestingOlsIdx[i], which specifies the OLS index of the i-th OLS to which the scalably nested SEI message applies when nesting_ols_flag is 1. The value of nesting_ols_idx_delta_minus1[i] must be in the range of 0 or greater and TotalNumOlss-2 or less. The variable NestingOlsIdx[i] can be derived as follows: if(i==0) NestingOlsIdx[i]=nesting_ols_idx_delta_minus1[i] else NestingOlsIdx[i]=NestingOlsIdx[i-1]+nesting_ols_idx_delta_minus1[i]+1

[0158] nesting_num_ols_layers_minus1[i]+1 specifies the number of layers to which scalably nested SEI messages are applied in relation to the NestingOlsIdx[i]th OLS. The value of nesting_num_ols_layers_minus1[i] must be in the range of 0 or greater and less than or equal to NumLayersInOls[NestingOlsIdx[i]]-1.

[0159] The variable NestingOlsLayerIdx[i][j] is used to derive the variable NestingOlsLayerIdx[i][j] which specifies the OLS layer index of the j-th layer to which a scalably nested SEI message in relation to the NestingOlsIdx[i]-th OLS applies when nesting_ols_flag is equal to 1. The value of nesting_ols_layer_idx_delta_minus1[i] must be in the range of 0 or greater and NumLayersInOls[nestingOlsIdx[i]]-2 or less.

[0160] The variable NestingOlsLayerIdx[i][j] can be derived as follows: (j==0) NestingOlsLayerIdx[i][j]=nesting_ols_layer_idx_delta_minus1[i][j] else NestingOlsLayerIdx[i][j]=NestingOlsLayerIdx[i][j-1]+ nesting_ols_layer_idx_delta_minus1[i][j]+1

[0161] The lowest value among all the values ​​of LayerIdInOls[NestingOlsIdx[i]][NestingOlsLayerIdx[i][0]] for i in the range of 0 to nesting_num_olss_minus1 must be equal to the nuh_layer_id of the current SEI NAL unit (e.g., the SEI NAL unit containing scalable nesting SEI messages). nesting_all_layers_flag may be set to equal to 1 to specify that scalable nested SEI messages generally apply to all layers having a nuh_layer_id greater than or equal to the nuh_layer_id of the current SEI NAL unit. nesting_all_layers_flag may be set to equal to 0 to specify that scalable nested SEI messages may or may not apply to all layers having a nuh_layer_id greater than or equal to the nuh_layer_id of the current SEI NAL unit.

[0162] The value of nesting_num_layers_minus1 plus 1 specifies the number of layers to which scalably nested SEI messages generally apply. If nuh_layer_id is the nuh_layer_id of the current SEI NAL unit, the value of nesting_num_layers_minus1 must be between 0 and vps_max_layers_minus1 - GeneralLayerIdx[nuh_layer_id]. nesting_layer_id[i] specifies the nuh_layer_id value of the i-th layer to which scalably nested SEI messages generally apply when nesting_all_layers_flag is 0. The value of nesting_layer_id[i] must be greater than nuh_layer_id, in which case nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

[0163] If nesting_ols_flag is equal to 1, the variable NestingNumLayers specifies the number of layers to which scalably nested SEI messages generally apply, and the list NestingLayerId[i] specifies a list of nuh_layer_id values ​​for i in the range from 0 to NestingNumLayers-1, which is greater than or equal to 0. In this case, nuh_layer_id is the nuh_layer_id of the current SEI NAL unit. if(nesting_all_layers_flag){ NestingNumLayers=vps_max_layers_minus1+1-GeneralLayerIdx[nuh_layer_id] for(i=0;i <NestingNumLayers;i++) NestingLayerId[i]=vps_layer_id[GeneralLayerIdx[nuh_layer_id]+i](D-2) else{ NestingNumLayers=nesting_num_layers_minus1+1 for(i=0;i <NestingNumLayers;i++) NestingLayerId[i]=(i==0)?nuh_layer_id:nesting_layer_id[i] }

[0164] The value of nesting_num_seis_minus1 plus 1 specifies the number of scalably nested SEI messages. The value of nesting_num_seis_minus1 should be between 0 and 63 (inclusive). nesting_0_bit should be set to equal to 0.

[0165] Figure 8 is a schematic diagram of an example of a video coding device 800. The video coding device 800 is suitable for carrying out the disclosed examples / embodiments as described herein. The video coding device 800 includes a downstream port 820, an upstream port 850, and / or a transceiver unit (Tx / Rx) 810 including a transmitter and / or receiver for transmitting data upstream and / or downstream over a network. The video coding device 800 also includes a processor 830 including a logic unit and / or a central processing unit (CPU) for processing data, and a memory 832 for storing data. The video coding device 800 may also include electrical, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 850 and / or downstream port 820 for data communication over an electrical, optical, or wireless communication network. The video coding device 800 may also include input and / or output (I / O) devices 860 for exchanging data with a user. The I / O device 860 may include output devices such as a display for showing video data and a speaker for outputting audio data. The I / O device 860 may also include input devices such as a keyboard, mouse, or trackball, and / or corresponding interfaces for interacting with such output devices.

[0166] The processor 830 is implemented by hardware and software. The processor 830 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 830 communicates with downstream port 820, Tx / Rx 810, upstream port 850, and memory 832. The processor 830 includes a coding module 814. The coding module 814 implements embodiments disclosed herein, such as methods 100, 900, and 1000, which can use a multi-layer video sequence 600 and / or bitstream 700. The coding module 814 may also implement any other methods / mechanisms described herein. Furthermore, the coding module 814 may implement a codec system 200, an encoder 300, a decoder 400, and / or HRD 500. For example, the coding module 814 may be used to signal and / or read various parameters as described herein. Furthermore, the coding module may be used to encode and / or decode a video sequence based on such parameters. Thus, the signaling modifications described herein can increase efficiency and / or avoid errors in the coding module 814. Therefore, the coding module 814 may be configured to perform a mechanism to address one or more of the aforementioned problems. Thus, the coding module 814 provides additional functionality and / or coding efficiency when the video coding device 800 codes video data. In this way, the coding module 814 improves the functionality of the video coding device 800 and addresses problems inherent in video coding technology. Furthermore, the coding module 814 enables the video coding device 800 to perform conversions to different states. Alternatively, the coding module 814 can be implemented as instructions stored in memory 832 (for example, stored on a non-temporary medium). It can be run by processor 830 (as a computer program product).

[0167] Memory 832 includes one or more memory types, such as disks, tape drives, solid-state drives, read-only memory (ROM), random-access memory (RAM), flash memory, tri-level associative memory (TCAM), and static random-access memory (SRAM). Memory 832 may also be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution.

[0168] Figure 9 is a flowchart of an example of method 900 for encoding a video sequence into a bitstream such as bitstream 700 by restricting the layer IDs in the PPS and APS within the bitstream. Method 900 may be used with an encoder such as a codec system 200, encoder 300, and / or video coding device 800 when performing method 100. Furthermore, method 900 can operate on HRD 500 and therefore can perform conformance testing on a multilayer video sequence 600.

[0169] Method 900 can be initiated when an encoder receives a video sequence and decides to encode that video sequence into a multilayer bitstream, for example, based on user input. In step 901, the encoder encodes the coded pictures in a set of VCL NAL units into a bitstream. For example, the coded pictures may be included in layers. Furthermore, the encoder can encode one or more layers containing coded pictures into a multilayer bitstream. A layer may include a set of VCL NAL units with the same layer ID and associated non-VCL NAL units. For example, a set of VCL NAL units is part of a layer if all of the VCL NAL units have a specific value of nuh_layer_id. A layer may include a set of VCL NAL units containing the video data of the encoded pictures and an arbitrary set of parameters used to code such pictures. One or more layers may be output layers. Layers that are not output layers are encoded to support the restoration of the output layers, but such support layers are not intended to be output at the decoder. In this way, the encoder can encode various combinations of layers to send to the decoder as needed. Layers may be sent as necessary to allow the decoder to obtain different representations of the video sequence depending on network conditions, hardware capabilities, and / or user settings.

[0170] Furthermore, a layer contains various non-VCL NAL units. Each non-VCL NAL unit is associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id. Constraints may be applied to the nuh_layer_id of non-VCL NAL units to ensure that they are properly associated with VCL NAL units and are not removed from the bitstream by the sub-bitstream extraction process before being sent to the decoder.

[0171] In step 903, the encoder encodes the PPS into a bitstream. The PPS is encoded into PPS NAL units. A PPS NAL unit is an example of a non-VCL NAL unit associated with one or more layers. PPS NAL units are coded under the constraint that a particular VCL NAL unit does not / cannot reference a PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit. In other words, the PPS and / or PPS NAL units are constrained to contain a nuh_layer_id value that is less than or equal to the value of the nuh_layer_id of all VCL NAL units that reference the PPS within the PPS NAL unit. Such constraints or requirements ensure that the bitstream conforms to the VVC or any other standard, as modified, for example, as shown herein. However, the encoder may also operate in other less constrained modes, such as when operating under different standards or different versions of the same standard. A specific VCL NAL unit described within a constraint may comprise a slice of coded pictures and may be associated with a header, such as a slice header and / or picture header, which contains data elements about the slice. The header specifies the PPS ID of the PPS. The PPS ID in the header indicates that the slice refers to a PPS NAL unit and therefore the constraint applies.

[0172] In step 905, the encoder encodes the APS into a bitstream. The APS is encoded into an APS NAL unit. An APS NAL unit is an example of a non-VCL NAL unit associated with one or more layers. An APS NAL unit is coded under the constraint that no particular VCL NAL unit references an APS NAL unit unless the value of the nuh_layer_id of the APS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit. In other words, an APS and / or APS NAL unit is constrained to contain a nuh_layer_id value that is less than or equal to the value of the nuh_layer_id of all VCL NAL units that reference the APS within the APS NAL unit. A particular VCL NAL unit may comprise a slice of coded pictures and may be associated with a header such as a slice header and / or picture header that contains data elements about the slice. The header specifies the APS ID of the APS. The APS ID in the header indicates that the constraint applies because the slice references an APS NAL unit. The aforementioned constraints and / or requirements ensure that the bitstream conforms to, for example, VVC or any other standard as modified as shown herein. However, the encoder may also operate in less restrictive modes, such as when operating under different standards or different versions of the same standard.

[0173] In step 907, the encoder may store a bitstream for communication to the decoder upon request. Upon receiving a request, the encoder and / or the corresponding slicer may also, in step 909, perform a sub-bitstream extraction process to remove NAL units from the bitstream based on the nuh_layer_id value. This process results in a bitstream containing only VCL NAL units and non-VCL NAL units associated with the requested layer. Due to the aforementioned constraints, PPS / PPS NAL units and / or APS / APS NAL units are retained in the bitstream when VCL NAL units in the requested layer refer to them, and removed from the bitstream when VCL NAL units in the requested layer do not refer to them. The extracted bitstream may be sent to the decoder for decoding and output.

[0174] Figure 10 is a flowchart of an exemplary method 1000 for decoding a video sequence from a bitstream such as bitstream 700, where the layer IDs of the PPS and APS in the bitstream are constrained. Method 1000 may be used with a decoder such as a codec system 200, a decoder 400, and / or a video coding device 800 when performing Method 100. Furthermore, Method 1000 may be used on a multilayer video sequence 600 whose compatibility has been checked by an HRD such as HRD 500.

[0175] Method 1000 can be initiated when the decoder begins receiving a bitstream of coded data representing a multilayer video sequence, for example, as a result of Method 900 and / or at the request of the decoder. In step 1001, the decoder receives a bitstream containing coded pictures in sets of PPS NAL units, APS NAL units, and VCL NAL units. For example, the coded pictures may be included in layers. Furthermore, the bitstream may contain one or more layers containing coded pictures. A layer may contain a set of VCL NAL units with the same layer ID and associated non-VCL NAL units. For example, a set of VCL NAL units is part of a layer if all of the VCL NAL units have a specific value of nuh_layer_id. A layer may contain a set of VCL NAL units containing video data of coded pictures and an arbitrary set of parameters used to code such pictures. One or more layers may be output layers. Layers that are not output layers are encoded to support the restoration of output layers, but such support layers are not intended to be output. In this way, the decoder can obtain different representations of the video sequence depending on network conditions, hardware capabilities, and / or user settings. The layers also include various non-VCL NAL units. Each non-VCL NAL unit is associated with a set of VCL NAL units, all of which have a specific value for nuh_layer_id. Constraints may be applied to the nuh_layer_id of non-VCL NAL units to ensure they are properly associated with VCL NAL units and not removed from the bitstream by the sub-bitstream extraction process before transmission to the decoder.

[0176] A PPS NAL unit is an example of a non-VCL NAL unit associated with one or more layers. A PPS NAL unit is coded under the constraint that a particular VCL NAL unit (e.g., from a set of VCL NAL units) does not / cannot reference a PPS NAL unit unless the value of the PPS NAL unit's nuh_layer_id is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit. In other words, a PPS and / or PPS NAL unit is constrained to contain a nuh_layer_id value that is less than or equal to the nuh_layer_id values ​​of all VCL NAL units that reference the PPS within the PPS NAL unit. A particular VCL NAL unit may comprise a slice of coded pictures and may be associated with headers such as a slice header and / or picture header, which contain data elements about the slice. The header specifies the PPS ID of the PPS. The PPS ID in the header indicates that the slice references a PPS NAL unit, and therefore the constraint applies.

[0177] An APS NAL unit is another example of a non-VCL NAL unit associated with one or more layers. An APS NAL unit is coded under the constraint that a particular VCL NAL unit does not reference an APS NAL unit unless the nuh_layer_id value of the APS NAL unit is less than or equal to the nuh_layer_id value of that particular VCL NAL unit. In other words, an APS and / or APS NAL unit is constrained to contain a nuh_layer_id value that is less than or equal to the nuh_layer_id value of all VCL NAL units that reference the APS within the APS NAL unit. A particular VCL NAL unit may comprise a slice of coded pictures and may be associated with headers such as a slice header and / or picture header, which contain data elements about the slice. The header specifies the APS ID of the APS. The APS ID in the header indicates that the constraint applies because the slice references an APS NAL unit.

[0178] The bitstream received in step 1001 is the result of a sub-bitstream extraction process that removes NAL units from the bitstream based on the nuh_layer_id value. This process yields a bitstream containing only VCL NAL units and non-VCL NAL units associated with the requested layer. Due to the aforementioned constraints, PPS / PPS NAL units and / or APS / APS NAL units are retained in the bitstream when VCL NAL units in the received layer refer to them, and removed from the bitstream when VCL NAL units in the received layer do not refer to them. In one embodiment, the video decoder expects the VCL NAL units, PPS NAL units, and APS NAL units to contain a nuh_layer_id constrained as described above based on VVC or some other standard. However, if the decoder determines that this condition is not true, the decoder may detect the error, notify the error, request that a corrected bitstream (or part thereof) be retransmitted, or take several other corrective measures to ensure that a matching bitstream is received.

[0179] In step 1003, the decoder may decode the coded picture from the set of VCL NAL units based on the PPS in the PPS NAL unit and / or the APS in the APS NAL unit to generate the coded picture. In step 1005, the decoder may transfer the coded picture for display as part of the coded video sequence.

[0180] Figure 11 is a schematic diagram of an example of a system 1100 for coding a video sequence using a bitstream where the layer IDs of PPS and APS within the bitstream are constrained. System 1100 may be implemented by encoders and decoders such as a codec system 200, encoder 300, decoder 400, and / or video coding device 800. Furthermore, system 1100 can perform conformance testing on a multilayer video sequence 600 and / or bitstream 700 using HRD 500. Additionally, system 1100 may be used when implementing methods 100, 900 and / or 1000.

[0181] System 1100 includes a video encoder 1102. The video encoder 1102 includes an encoding module 1105 for encoding an encoded picture in a set of VCL NAL units in a bitstream. The encoding module 1105 further encodes the PPS in a PPS NAL unit into a bitstream, based on the constraint that a particular VCL NAL unit (e.g., from a set of VC NAL units) does not refer to a PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of that particular VCL NAL unit. The video encoder 1102 further includes a storage module 1106 for storing the bitstream for communication to the decoder. The video encoder 1102 further includes a transmission module 1107 for sending the bitstream to the video decoder 1110. The video encoder 1102 may be further configured to perform any step of method 900.

[0182] System 1100 further includes a video decoder 1110. The video decoder 1110 includes a receiving module 1111 for receiving a bitstream comprising PPS in PPS NAL units and coded pictures in a set of VCL NAL units, where a particular VCL NAL unit does not refer to a PPS NAL unit unless the value of the nuh_layer_id of the PPS NAL unit is less than or equal to the value of the nuh_layer_id of the particular VCL NAL unit. The video decoder 1110 further includes a decoding module 1113 for decoding the coded pictures from the set of VCL NAL units based on the PPS in the PPS NAL units in order to produce a coded picture. The video decoder 1110 further includes a transfer module 1115 for transferring the coded pictures for display as part of a coded video sequence. The video decoder 1110 may be further configured to perform any step of method 1000.

[0183] If there are no intervening elements between the first and second elements, other than lines, traces, or other mediating elements, the first element is directly coupled to the second element. If there are intervening elements between the first and second elements other than lines, traces, or other mediating elements, the first element is indirectly coupled to the second element. The term "coupled" and its variations include both direct and indirect coupling. The use of the term "about" means a range including ±10% of the following number, unless otherwise specified.

[0184] The steps of the typical methods described herein do not necessarily have to be performed in the order described, and it should be understood that the order of steps in such methods is merely a typical example. Similarly, in methods consistent with various embodiments of this disclosure, such methods may include further steps, and certain steps may be omitted or combined.

[0185] While several embodiments have been provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. These examples should be considered illustrative rather than restrictive, and their intent should not be limited to the details given herein. For example, various elements or components may be combined or incorporated in other systems, or certain functions may be omitted or not implemented.

[0186] In addition, techniques, systems, subsystems, and methods described and illustrated individually or separately in various embodiments may be combined with or incorporated into other systems, components, techniques, or methods without departing from the scope of this disclosure. Other examples of modifications, substitutions, and alterations are evident to those skilled in the art and may be made without departing from the ideas and scope disclosed herein. [Explanation of symbols]

[0187] 100 How it works 200 codec systems 201 Video signal 211 General-purpose coder control component 213 Transformation Scaling Quantization Component 215 Intrapicture Estimation Components 217 Intrapicture Prediction Component 219 Motion compensation component 221 Motion Estimation Component 223 Picture Buffer Component 225 In-loop filter component 227 Filter Control Analysis Component 229 Inverse Scaling Transform Component 231 Header Formatting CABAC Component 300 video encoders 301 Video signal 313 Transformation Quantization Components 317 Intrapicture Prediction Component 321 Motion Compensation Component 323 Picture Buffer Component 325 In-Loop Filter Components 329 Inverse Transform Quantization Components 331 Entropic Coding Components 400 video decoders 417 Intrapicture Prediction Component 421 Motion Compensation Component 423 Picture Buffer Component 425 In-loop filter component 429 Inverse Transform Quantization Component 433 Entropy Decoding Component 500 Virtual Reference Decoder (HRD) 541 Virtual Stream Scheduler (HSS) 543 Coded Picture Buffer (CPB) 545 Decoding Process Components 547 Decoded Picture Buffer (DPB) 549 Output Cropping Component 551 bitstream 553 Decoding Unit (DU) 555 Decoded DU 556 Reference Picture 557 Pictures 559 Output cropped picture 600 multilayer video sequences 611-618 Pictures 621 Interlayer Prediction 623 Inter forecast 627 Access Unit (AU) 628 Picture Units (PU) 629 Temporal ID Layers 631-632 700 bitstream 711 Video Parameter Set (VPS) 713 Sequence Parameter Set (SPS) 715 Picture Parameter Set (PPS) 716 APS 717 Slice Header 718 Picture Header 719 SEI Message 720 image data 721 First Picture 723 Layers 725 Pictures 727 slices 729 nuh_layer_id 741 NAL Unit 743 NAL Unit 745 NAL Unit 746 NAL Unit 747 NAL Unit 748NAL Unit 749 NAL Unit 800 video coding devices 814 Coding Modules 820 Downstream Ports 830 Processor 832 memory 850 Upstream Port 860 I / O devices 1100 System 1102 Video Encoder 1105 Encoding Module 1106 Memory Module 1107 Transmitter Module 1110 Video Decoder 1111 Receiver Module 1113 Decoding Module 1115 Transfer Module

Claims

1. A method carried out by an encoder, wherein the method is The steps of transmitting a bitstream comprising a Picture Parameter Set (PPS) in a Network Abstraction Layer (NAL) unit and a set of Video Coding Layer (VCL) NAL units using the encoder's transmitter, The PPS NAL unit is a non-VCL NAL unit, The PPS NAL unit has a PPS raw byte sequence payload (RBSP) having a nuh_layer_id that is equal to the lowest NAL unit header layer identifier (nuh_layer_id) of a plurality of coded slice NAL units referencing the PPS, and the steps are as follows: To generate a decoded picture, the encoder's processor decodes the coded picture from a set of VCL NAL units based on the PPS in the PPS NAL unit, The VCL NAL unit comprises a slice of the coded picture and a header containing data elements related to the slice, the header specifying the APS identifier (ID) of the Adaptive Parameter Set (APS). method.

2. An encoding device, A transmitter configured to transmit a bitstream comprising a Picture Parameter Set (PPS) in a Network Abstraction Layer (NAL) unit and a set of Video Coding Layer (VCL) NAL units, wherein the PPS comprises a Picture Parameter Set (PPS) in a Network Abstraction Layer (NAL) unit and a set of Video Coding Layer (VCL) NAL units, The PPS NAL unit is a non-VCL NAL unit, The PPS NAL unit includes a transmitter having a PPS raw byte sequence payload (RBSP) having a nuh_layer_id equal to the lowest NAL unit header layer identifier (nuh_layer_id) of a plurality of coded slice NAL units referencing the PPS, A memory coupled to the transmitter, comprising a memory for storing instructions, A processor coupled to the memory, configured to execute the instructions and cause the encoding device to perform the method according to claim 1, An encoding device equipped with the following features.

Images