Sequence parameter set based multi-layer bitstream conformance check and corresponding apparatus

Abstract

A method for bitstream conformance check is disclosed. Information representative of the size of pictures contained in the same access unit and associated with a current output layer set is obtained (S500). For a level signaled for the current output layer set, the size of a decoded picture buffer is determined (S502) in response to the obtained information. Bitstream conformance for the current output layer set is then checked (S504) by comparing the determined size of the decoded picture buffer with the size of the decoded picture buffer required for decoding the current output layer set.

Description

Cross-reference to related applications

[0001] This application claims the benefit of European Application No. 24305388.1, filed on 15 March 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] At least one of these embodiments generally relates to a method and apparatus for performing consistency checks on multi-layer video bitstreams. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to utilize spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to fully leverage intra-frame or inter-frame image correlations, followed by transform, quantization, and entropy coding of the differences between the original and predicted blocks (typically represented as prediction error or prediction residual). To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction.

[0004] Coding standards such as VVC (Video Coding Commons) define coding configurations through profiles, tiers, and levels. These configurations impose operational constraints, limiting the range of coding tools and parameters that the encoder can use for video compression. The encoder signals the profile (and associated tiers and levels) upon which it generates the encoded video bitstream to the decoder, allowing the decoder to verify whether computational resources can be allocated for decoding that bitstream. Similarly, multi-layer video can be encoded according to multiple profiles. A decoder conforming to a multi-layer profile needs the processing power to decode that multi-layer video. In principle, the complete set of tools available in the encoder can be used for each layer, so the processing power required by a multi-layer decoder may be the sum of the processing power required to decode each layer. However, in practice, and based on specific application requirements, different toolsets can be applied to the encoding of each layer to control the processing power required to decode each layer while maintaining acceptable compression performance. Summary of the Invention

[0005] In one implementation, bitstream consistency can be checked. In an example, information representing the size of images contained in the same access unit and associated with the current output layer set is obtained; for the level signaled for the current output layer set, the size of the decoded image buffer is determined in response to the obtained information; and bitstream consistency for the current output layer set is checked by comparing the determined size of the decoded image buffer with the size of the decoded image buffer required to decode the current output layer set.

[0006] For example, obtaining information representing the size of an image contained in the same access unit and associated with the current output layer set may include a decoding syntax element whose value is equal to the size.

[0007] In other examples, obtaining information representing the size of an image contained in the same access unit and associated with the current output layer set may include parsing a first syntax element representing the image height and a second syntax element representing the image width, wherein the product of the values ​​of the first and second syntax elements is equal to the size of the image contained in the same access unit and associated with the current output layer set.

[0008] In another example, information indicating the size of the images contained in the current output layer set is derived from the sequence parameter set syntax elements. More precisely, the requirements for each layer are obtained from the information signaled in the sequence parameter set. In the example, the size required to decode the current output layer set for the decoded image buffer can be parsed from the decoded image buffer syntax structure.

[0009] In another example, it can be determined whether real-time decoding of the current output layer set is feasible by comparing the brightness sampling rate of the level notified by the signal with the information obtained.

[0010] In other examples, the requirements for each layer can be obtained; information for the current output layer set can be parsed; the requirements for each output layer set can be derived from the parsed information for the current output layer set; and the bitstream consistency for the current output layer set can be checked in response to the derived requirements for each output layer set. Attached Figure Description

[0011] Figure 1 A block diagram of a system that can implement aspects of this embodiment is shown; Figure 2 A block diagram illustrating an embodiment of a video encoder is shown; Figure 3 A block diagram illustrating an embodiment of the video decoder is shown; Figure 4 This is a diagram illustrating a multi-layer encoding / decoding workflow; Figure 5A A flowchart of a decoding method according to a specific embodiment is depicted; Figure 5B A flowchart illustrating an encoding method according to a specific embodiment is provided; Figure 6A A flowchart is depicted for a decoding method with accompanying grammatical changes according to a specific embodiment; Figure 6B A flowchart of a decoding method according to another specific embodiment is depicted; Figure 7AA flowchart is depicted for a decoding method that accompanies the decoding requirement for each layer according to a specific embodiment; Figure 7B A flowchart depicts an encoding method according to a specific embodiment, which accompanies the decoding requirement of each layer; and Figure 8 A flowchart is depicted for a decoding method that accompanies the decoding requirement for each layer according to another specific embodiment. Detailed Implementation

[0012] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail, and are generally described in a manner that may sound restrictive, at least for the purpose of illustrating the various features. However, this is for the purpose of clarity and does not limit the scope of this application or these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0013] The aspects described and envisioned in this application can be implemented in many different forms. The following... Figure 1 , 2 Some embodiments are provided in sections 3 and 4, but other embodiments are contemplated, and... Figure 1 , 2 The discussion in section 3 does not limit the breadth of implementation. At least one of these aspects generally relates to video encoding and decoding, and at least another aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0014] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "encode" and "code," "pixel" and "sample," and "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0015] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding." The use of these terms does not imply the order of the modified operations unless specifically required. Therefore, in this example, the first decoding need not be performed before the second decoding, and can occur, for example, before, during, or within an overlapping period of the second decoding.

[0016] For clarity, in all embodiments described herein, met conditions, unmet conditions, and configuration condition parameters are described relative to a threshold (e.g., greater than or less than), a value (e.g., a threshold), a configuration value (e.g., a threshold), etc. For example, met conditions may be described as being above (e.g., a threshold) value, and unmet conditions (e.g., performance criteria) may be described as being below (e.g., a threshold) value. The embodiments described herein are not limited to threshold-based conditions. Any other types of conditions and parameters (e.g., belonging to or not belonging to a value range) may be applied to the embodiments described herein.

[0017] This aspect is not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically impractical, the aspects described in this application may be used alone or in combination.

[0018] Figure 1 A block diagram illustrating examples of a system 100 that can implement various aspects and embodiments is shown. System 100 can be embodied as a device including various components described below and configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, and servers. Elements of system 100 can be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0019] System 100 includes at least one processor 110 configured to execute instructions loaded thereon to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0020] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 130 may be implemented as a separate element of system 100, or may be incorporated into processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0021] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items of various kinds during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0022] In some embodiments, memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, a new standard developed by JVET).

[0023] Inputs to the components of system 100 may be provided by a variety of input devices as indicated in box 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) component (COMP) input terminals (or sets of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.

[0024] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band to select, for example, a signal band (which may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as a frequency selector, signal selector, band limiter, channel selector, filter, downconverter, demodulator, error corrector, and demultiplexer. The RF section may include a tuner performing a variety of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0025] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 110. Similarly, aspects of USB or HDMI interface processing may be implemented as needed within a separate interface IC or within processor 110. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on the output device.

[0026] Various components of system 100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using suitable connection means 115, such as internal buses as known in the art, including I2C buses, wiring, and printed circuit boards.

[0027] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data on the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0028] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 190 and a communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router, providing access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 100, which transmits data via an HDMI connection to input box 105. Still other embodiments use an RF connection to input box 105 to provide streaming data to system 100. As described above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0029] System 100 can provide output signals to a variety of output devices, including a display 165, a speaker 175, and other peripheral devices 185. The display 165 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used in a television, tablet computer, laptop computer, cellular phone (mobile phone), or other device. The display 165 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., as an external monitor for a laptop computer). In various embodiment examples, other peripheral devices 185 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functionality based on the output of system 100. For example, an optical disc player performs the function of playing the output of system 100.

[0030] In various embodiments, control signals communicate between system 100 and display 165, speaker 175, or other peripheral devices 185 using signaling protocols such as AV.Link, CEC, or other communication protocols. These communication protocols enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 100 via dedicated connections through corresponding interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated with other components of system 100 into a single unit within an electronic device (e.g., a television). In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0031] Display 165 and speaker 175 may alternatively be separated from one or more other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0032] The embodiments can be executed by computer software implemented by processor 110 or by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. Memory 120 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as a non-limiting example. Processor 110 can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as a non-limiting example.

[0033] Figure 2 An example video encoder 200, such as a VVC (Various Video Coding) encoder, is shown. Figure 2 Encoders that improve upon the VVC standard or employ VVC-like technologies can also be shown.

[0034] Before encoding, the video sequence may undergo pre-coding (201), for example, applying a color transformation to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution more robust to compression (e.g., using histogram equalization on one of the color components). Metadata may be associated with the pre-coding process and appended to the bitstream.

[0035] In encoder 200, the image is encoded by encoder elements as described below. The image to be encoded is divided (202) and processed in units such as CUs (coding units). Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260), for example using an intra-frame prediction tool such as decoder-side intra-frame mode derivation (DIMD). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame or inter-frame modes to use to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. The prediction residual is calculated by, for example, subtracting (210) the predicted block (also called the prediction block) from the original image block.

[0036] The prediction residual is then transformed (225) into transformation coefficients c (also known as prediction residual transformation coefficients), which are quantized (230) into quantization indices. (Also known as the transform coefficient level on the encoder side or the quantized transform coefficient). Quantization level (also known as the quantization index). Motion vectors and other syntax elements (such as image segmentation information) are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly encoded without applying the transform or quantization process.

[0037] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) (also known as scaling) and inversely transformed (250) to decode the prediction residuals. The decoded prediction residuals and the predicted blocks are combined (255) to reconstruct the image blocks. A loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (Sample Adaptive Offset) / ALF (Adaptive Loop Filter) filtering to reduce coding artifacts. The filtered image is stored in a reference image buffer (280). Thus, the loop filter (265) is used to enhance the reconstructed image before storing it in the reference image buffer (280). Loop filters form a complete family. Among them, the deblocking filter (DBF) is designed to reduce blocking artifacts that occur along block boundaries. Deblocking filters are often designed to improve subjective quality, i.e., the attention of the human psychological visual system to such coding errors. In typical video coding standards such as HEVC and VVC, deblocking filters are predetermined based on coding information (such as prediction modes, motion vectors, and transform coefficients) and local variations at block boundaries. Adaptive Loop Filters (ALFs), on the other hand, are learned on the encoder side to minimize the mean square error relative to the source image, and the learned filter weights are then encoded into the bitstream. ALFs are typically applied at the CTU level, while deblocking filters are applied along block boundaries.

[0038] Figure 3 A block diagram of an example video decoder 300 is shown. In decoder 300, the bitstream is decoded by decoder elements as described below. Video decoder 300 typically performs the following operations: Figure 2 The encoding process described herein is the inverse of the decoding process. Encoder 200 also typically performs video decoding as part of the video data encoding.

[0039] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy-decoded (330) to obtain the quantization level. (Also known as the transform coefficient level or quantization level on the decoder side), prediction mode, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) images based on the decoded image segmentation information. Quantization level The dequantized (340) are the reconstructed transform coefficients. Dequantization is also known as scaling. The reconstructed transform coefficients. The image is inversely transformed (350) to obtain the prediction residual. The prediction residual (355) and the predicted block (also called the prediction block) are combined to reconstruct the image block. The predicted block can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). A loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380), also called the DPB (“decoded picture buffer”). Note that for a given picture, the contents of the reference picture buffer 380 on the decoder 300 side are the same as the contents of the reference picture buffer 280 on the encoder 200 side for the same picture.

[0040] The decoded image can also undergo post-decoding processing (385), such as inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which is the inverse of the remapping process performed in the pre-encoding process (201). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0041] The aspects disclosed herein are described with reference to CU; however, the described aspects are similarly applicable to any video frame region (i.e., video data region) to which the encoder 200 or decoder 300 may apply encoding tools. Generally, the aspects described herein can be applied to video data regions of any shape or size formed by video partitions. CU includes the luminance component Y and the chrominance components Cr and Cb (either of which is also referred to as C herein).

[0042] VVC supports multi-layer encoding through multiple layers, such as for scalability, multi-view, or 360° use cases. Each layer corresponding to a source is encoded separately or with inter-layer dependencies, and then the bitstreams are merged into a single bitstream, where layer information is... Figure 4 As shown.

[0043] Figure 4 This is a diagram illustrating a multi-layer encoding / decoding workflow 500. (See diagram for example.) Figure 4 As shown in the example, encoder 510 and decoder 570 are connected via multiplexer 540, channel 550 (e.g., a wired or wireless communication link), and demultiplexer 560. In encoder 510, a separate encoder instance can be used for each layer. For example, a separate encoder instance can be used... N Layer encoders, such as encoder layer 0 (530.0), encoder layer 1 (530.1), and encoder layer... N (530.N), which encode their respective inputs: source layer 0 (520.0), source layer 1 (520.1), and source layer... N (520.N). For example, each source can correspond to different resolution levels of the same input image—from the lowest resolution level (source layer 520.0), through intermediate resolution levels 520.1 to 520.N-1, to the highest resolution level 520.N (source layer 520.N). Furthermore, the source layer (e.g., such as 520.1) can be encoded relative to a reference image provided by a lower-level encoder (e.g., encoder 530.0) (e.g., by encoder 530.1). This creates inter-layer dependencies at both ends of encoder 510 and decoder 570 due to inter-layer predictions (indicated by dashed arrows).

[0044] Therefore, for each frame of the video sequence, the layer encoder 530.0-N is initiated to encode the image of the corresponding layer, where the lower-layer encoder is used before the higher-layer encoder due to the dependencies introduced by inter-layer prediction. For each frame, the output bitstream (i.e., sub-bitstream) of the corresponding layer encoder 530.0-N is multiplexed 540 into a single bitstream. Layer ID information (identifiers) is also encoded in this bitstream, associating the sub-bitstream with its respective layer. The layer ID information can be signaled, for example, in the Network Abstraction Unit (NALU) header (the NALU is a syntax structure containing an indication of the subsequent data type, and bytes containing that data in the form of a raw byte sequence payload, which is interspersed with anti-race bytes as needed). The layer encoder 530.0-N shares layer-independent data structures, such as APS (Adaptive Parameter Set) and VPS (Video Parameter Set), which can also be encoded into the bitstream.

[0045] The bitstream generated by encoder 510 is transmitted to decoder 570 via channel 550 for layered decoding. The transmitted bitstream is first demultiplexed 560 into sub-bitstreams for each layer. Therefore, the sub-bitstream of layer 0 can be independently decoded by decoder layer 0 (580.0) to generate reconstructed source layer 0 (590.0), and the sub-bitstream of layer 1 can be decoded by decoder layer 1 (580.1) depending on the output from decoder layer 0 (580.0) (from which a reference image is received) to generate reconstructed source layer 1 (590.1), and the layers... N The sub-bitstream can be generated by the decoder layer. N (580.N) depends on the decoder layer. N The output of -1 (from which it receives the reference image) is decoded to generate the reconstructed source layer. N (590.N). Therefore, for each frame, decoding a multi-layer bitstream requires first decoding the layers from lower levels, and then decoding the layers from higher levels. Figure 4 In the example, layer 0 (580.0) is decoded first, then layer 1 (580.1) can be decoded, and finally layer... N (580.N). It's important to note that each layer of the decoder (580.0-N) requires full decoding capabilities, including entropy decoding, inverse quantization, inverse transform, prediction, and loop filtering. This can lead to high processing complexity at the decoder end, proportional to the number of layers that must be decoded for each frame. Furthermore, multi-layer encoding / decoding increases memory requirements. In fact, the decoded image buffer (DPB) that holds the image in the decoder before outputting the image must be large enough to handle multiple layers.

[0046] The VVC standard defines an Output Layer Set (OLS) as a set of layers that designates one or more layers as output layers. Therefore, a given layer only needs to be decoded if it is defined as an output layer or is a dependent layer that needs to be used as an output layer. OLSs are typically defined within a VPS. The OLS is signaled by indicating the output layers of the corresponding OLS. Other layers belonging to that OLS are inferred from the layer dependencies indicated in the VPS. OLSs are used to (i) signal which layers can be output (and if multiple layers are indicated as output layers in the same OLS, they are all output together) and which layers will be decoded, and (ii) associate certain attributes with each VPS, such as associating a PTL (Profile, Tier, Level) with each OLS. In practice, for spatially scalable video, the first OLS contains only the base layer that is output, the second OLS contains the second layer as output and the base layer as a dependency, and the third OLS contains the third layer as output and the second and base layers as dependencies (the second layer is a direct dependency of the third layer, and the base layer is an indirect dependency required to decode the second layer). For each OLS, the complexity constraints can be different, as indicated by its associated PTL.

[0047] To ensure practicality in implementing the full syntax defined in the VVC standard, a finite number of subsets of the syntax are defined in VVC through combinations of capabilities (also known as constraints) called “grades,” “hierarchies,” and “levels” (i.e., PTLs). A grade and its associated hierarchies and levels represent coding configurations that translate into different processing capability requirements that the decoder must meet when decoding the corresponding bitstream. The VVC PTLs are defined in Appendix A of the VVC specification (see Versatile Video Coding Editorial Refinements on Draft 10 (JVET-T2001), 20th Meeting, via teleconference, October 7-16, 2020).

[0048] A “grade” is a subset of the entire bitstream syntax specified in VVC. In other words, a grade defines the set of tools that the canonical decoder should support. Within the boundaries defined by a given grade, the performance of the encoder and decoder still varies considerably depending on the values ​​used by the syntax elements encoded into the bitstream (such as the specified size of the decoded image). In many applications, implementing a decoder capable of handling all assumed values ​​of syntax elements within a particular grade is neither practical nor economical. To address this issue, “hierarchies” and “levels” are specified within each grade. Thus, a level is a specified set of constraints applied to the values ​​of syntax elements encoded into (or signaled within) the bitstream. Therefore, a level specifies quantitative constraints on the maximum canonical decoder capability, such as, for example, the maximum luma sampling rate (the maximum number of luma samples per second that the decoder should process and produce), the maximum bit rate, and the maximum image resolution. Successive levels typically correspond to incremental capabilities. Therefore, a decoder conforming to a given level (e.g., level 5.0) also conforms to all lower levels (e.g., levels 1.0 through 4.1). Some of these constraints are expressed as simple restrictions on values, while others take the form of constraints on arithmetic combinations of values ​​(e.g., image width multiplied by image height multiplied by the number of images decoded per second). A hierarchy is a specified category of level constraints imposed on syntax element values ​​or variable values ​​in a bitstream. Levels specified for lower levels are more restrictive than those specified for higher levels. Ranks, hierarchies, and levels are also used to indicate the capabilities of individual decoder implementations and the interoperability points between the encoder and decoder. Level constraints are contained within a hierarchy, and a decoder conforming to a certain hierarchy and level will be able to decode all bitstreams conforming to the same hierarchy, lower levels of that hierarchy, or any level below that level.

[0049] Any encoding process that produces bitstream data conforming to the specified bitstream syntax format requirements of a specification (such as the VVC Recommendation | International Standard) is considered compliant with that specification. A corresponding decoding process is specified such that all decoders conforming to a specified combination of capabilities (also known as levels and constraints) will produce numerically identical cropped decoded output pictures when calling the decoding process associated with that level for a bitstream conforming to that level, level, and constraint. Any decoding process that produces the same cropped decoded output pictures (with the correct output order or output timing, as specified) as those produced by the process described in the specification is considered compliant with the specification.

[0050] The constraints defined by the PLT can be used on the decoder side to check if the bitstream conforms to the video coding specification, or if the decoder conforms to the grade / level indicated by signals in the bitstream. If it does not conform, the client receiving and decoding the video bitstream can request another version of the video (another bitstream) that conforms to the grade / level supported by the decoder. Different video decoders (the hardware core of the video decoder, or the software implementation of the video decoder) can conform to different grades and levels. This means that a video decoder may be able to decode a bitstream that conforms to a first given grade / level, but it may not be able to decode a bitstream that conforms to a second given grade / level that is different from the first given grade / level.

[0051] These constraints can be used on the encoder side to ensure that the encoded bitstream conforms to the profile / tier / level targeted by the encoder. The encoder, for example, uses syntax structures such as the profile_tier_level() syntax structure to signal profile, tier, and level information in the bitstream.

[0052] The `profile_tier_level()` syntax provides level information, and optionally, tiers, levels, subtiers, and general constraint information to which one or more OLSs conform. The OLS associated with a given `profile_tier_level()` syntax is defined as a scoped OLS (i.e., `OlsInScope`). When the `profile_tier_level()` syntax is included in a VPS, `OlsInScope` is one or more OLSs specified by the VPS. A list of PTLs is defined in the VPS, and for each OLS, an index of which PTL the current OLS relates to is signaled. Therefore, a given PTL can be associated with one or more OLSs. When the `profile_tier_level()` syntax is included in a Sequence Parameter Set (SPS), `OlsInScope` is an OLS that only includes the lowest level among the layers referencing the SPS, and that the lowest level is an independent layer. Furthermore, a given level can be specified for a sub-layer of a layer, associated with a TemporalId value (suitable for time scalability use cases).

[0053] The VVC standard also defines Network Abstraction Layer (NAL) units, Decoded Picture Buffers (DPBs), Picture Units (PUs), and Access Units (AUs). A NAL unit is a syntax structure containing an indication of the subsequent data type, along with bytes containing that data in the form of a Raw Byte Sequence Payload (RBSP), interspersed with anti-race bytes as needed. A DPB is a buffer that holds decoded pictures for reference, output reordering, or output delays specified for hypothetical reference decoders (e.g., reference picture buffers 280 and 380). A PU is a set of NAL units that are associated with each other according to specified classification rules, are consecutive in decoding order, and contain exactly one encoded picture. In other words, a PU represents a bitstream associated with a picture. As an example, a PU can contain several slices (in this case, the PU contains consecutive Slice NAL units, as well as associated SPS, APS, and SEI NAL units). For example, a PU contains VPS, APS, slice data NALUs, and NALUs (short for NAL Units) related to the SEI of a picture. An AU is a set of PUs belonging to different layers and containing encoded images to be output from the DPB simultaneously. In other words, in a multi-layer context, an AU represents a bitstream associated with a single image represented by multiple layers. When there is only one layer, the AU contains only one image.

[0054] The `dpb_parameters()` syntax structure provides information on the DPB size, maximum number of image reorderings, and maximum latency required to decode one or more OLSs. Its syntax is defined in Table 1. When the `dpb_parameters()` syntax structure is included in a VPS, the OLS to which the `dpb_parameters()` syntax structure applies is specified by the VPS. When the `dpb_parameters()` syntax structure is included in an SPS, it applies to OLSs that only include the following layers: the lowest layer among the layers referencing the SPS, and that lowest layer is an independent layer.

[0055]

[0056] The semantics of the syntax elements used in Table 1 are specified below.

[0057] Increasing 1 to `dpb_max_dec_pic_buffering_minus1[i]` specifies the maximum required size of the DPB in units of the image storage buffer when `Htid` equals `i`. The value of `dpb_max_dec_pic_buffering_minus1[i]` should be in the range of 0 to `MaxDpbSize-1` (inclusive), where `MaxDpbSize` is defined as follows: When `i` is greater than 0, `dpb_max_dec_pic_buffering_minus1[i-1]` should be greater than or equal to `dpb_max_dec_pic_buffering_minus1[i-1]`. When `dpb_max_dec_pic_buffering_minus1[i]` does not exist for `i` in the range of 0 to `MaxSubLayersMinus1-1` (inclusive) because `subLayerInfoFlag` equals 0, it is inferred to be equal to `dpb_max_dec_pic_buffering_minus1[MaxSubLayersMinus1]`.

[0058] `dpb_max_num_reorder_pics[i]` specifies the maximum allowed number of pictures in the OLS that can precede any picture in the OLS in the decoding order and follow any picture in the output order when `Htid` equals `i`. The value of `dpb_max_num_reorder_pics[i]` should be in the range of 0 to `dpb_max_dec_pic_buffering_minus1[i]` (inclusive). When `i` is greater than 0, `dpb_max_num_reorder_pics[i]` should be greater than or equal to `dpb_max_num_reorder_pics[i-1]`. When `dpb_max_num_reorder_pics[i]` does not exist for `i` in the range of 0 to `MaxSubLayersMinus1-1` (inclusive) because `subLayerInfoFlag` equals 0, it is inferred that it is equal to `dpb_max_num_reorder_pics[MaxSubLayersMinus1]`.

[0059] dpb_max_latency_increase_plus1[i] is not equal to 0 and is used to calculate the value of MaxLatencyPictures[i], which specifies the maximum number of pictures in the OLS that can precede any picture in the OLS in the output order and follow any picture in the decoding order when Htid equals i.

[0060] When dpb_max_latency_increase_plus1[i] is not equal to 0, the value of MaxLatencyPictures[i] is specified as follows: MaxLatencyPictures[i]=dpb_max_num_reorder_pics[i]+dpb_max_latency_increase_plus1[i]-1 When dpb_max_latency_increase_plus1[i] equals 0, it does not represent the corresponding limit.

[0061] The value of dpb_max_latency_increase_plus1[i] should be in the range of 0 to 2^32-2 (inclusive). When dpb_max_latency_increase_plus1[i] does not exist for i in the range of 0 to MaxSubLayersMinus1-1 (inclusive) because subLayerInfoFlag is equal to 0, it is inferred that it is equal to dpb_max_latency_increase_plus1[MaxSubLayersMinus1].

[0062] The variable Htid identifies the highest temporal sublayer to be decoded, as deduced below. The following applies to the first AU of the bitstream: If some external means are available to set Htid, then Htid is set by the external means. Otherwise, if opi_htid_plus1 exists in the OPI NAL unit of the first AU of the bitstream, then Htid is set to equal to ((opi_htid_plus1>0)?opi_htid_plus1-1:0). Otherwise, Htid is set to equal to vps_ptl_max_tid[vps_ols_ptl_idx[TargetOlsIdx]]. Note 3 - When sps_video_parameter_set_id equals 0, Htid will be set to equal to sps_max_sublayers_minus1.

[0063] To compare the capabilities of tiers, a tier with a general_tier_flag value of 0 (i.e., the main tier) is considered lower than a tier with a general_tier_flag value of 1 (i.e., the higher tier). To compare the capabilities of levels, if the value of general_level_idc or sublayer_level_idc[i] of a specific level within a specific tier is less than the values ​​of other levels, that specific level is considered lower than some other level within the same tier.

[0064] For an OLS with an OLS index TargetOlsIdx, the derivation of the variables PicWidthMaxInSamplesY, PicHeightMaxInSamplesY, and PicSizeMaxInSamplesY, along with the applicable dpb_parameters() syntax structure, is as follows: If NumLayersInOls[TargetOlsIdx] equals 1, then PicWidthMaxInSamplesY is set to equal sps_pic_width_max_in_luma_samples, PicHeightMaxInSamplesY is set to equal sps_pic_height_max_in_luma_samples, and PicSizeMaxInSamplesY is set to equal PicWidthMaxInSamplesY. PicHeightMaxInSamplesY, where sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are found in the SPS referenced by the layer in OLS, and the applicable dpb_parameters() syntax structure is also found in that SPS.

[0065] Otherwise (NumLayersInOls[TargetOlsIdx] is greater than 1), PicWidthMaxInSamplesY is set to equal vps_ols_dpb_pic_width[MultiLayerOlsIdx[TargetOlsIdx]], PicHeightMaxInSamplesY is set to equal vps_ols_dpb_pic_height[MultiLayerOlsIdx[TargetOlsIdx]], and PicSizeMaxInSamplesY is set to equal PicWidthMaxInSamplesY. PicHeightMaxInSamplesY, and the applicable dpb_parameters() syntax structure is identified by vps_ols_dpb_params_idx[MultiLayerOlsIdx[TargetOlsIdx]] found in the VPS.

[0066] Table 2 specifies the restrictions used for each level within each tier (except for level 15.5) (i.e., Table 135 in the VVC standard). Level 15.5 is an unrestricted level. The tier and level of the bitstream conformance are indicated by the syntax elements general_tier_flag and general_level_idc, respectively; and the level of the sublayer representation conformance is indicated by the syntax element sublayer_level_idc[i], as follows: If the specified level is not level 15.5, then according to the hierarchical constraints specified in Table 2, a general_tier_flag of 0 indicates compliance with the primary level, and a general_tier_flag of 1 indicates compliance with the higher-level level. Furthermore, for levels lower than level 4 (corresponding to entries marked "-" in Table 2), general_tier_flag should be equal to 0. Otherwise (if the specified level is level 15.5), the requirement for bitstream consistency is that general_tier_flag should be equal to 1, and the value 0 for general_tier_flag is reserved for future use by ITU-T|ISO / IEC; the decoder should ignore this value of general_tier_flag.

[0067] general_level_idc and sublayer_level_idc[i] should be set to the same value as the general_level_idc value for the level number specified in Table 2.

[0068]

[0069] The encoded picture buffer (CPB) is a first-in, first-out buffer containing DUs in decoding order. A decoded unit (DU) is an AU if DecodingUnitHrdFlag equals 0, otherwise a subset of the AU, consisting of one or more VCL NAL units and associated non-VCL NAL units within the AU. An encoded picture is an encoded representation of a picture, comprising VCL NAL units within an AU with a specific nuh_layer_id value and containing all CTUs of the picture.

[0070] The following describes the level-specific limitations. When the specified level is not level 15.5, the value of dpb_max_dec_pic_buffering_minus1[Htid]+1 should be less than or equal to MaxDpbSize, as deduced below:

[0071] MaxLumaPs is specified in Table 2, maxDpbPicBuf is equal to 8, and dpb_max_dec_pic_buffering_minus1[Htid] is found in or derived from the applicable dpb_parameters() syntax structure. Assuming numDecPics is the number of images in AU n, the variable AuSizeMaxInSamplesY[n] is set to equal PicSizeMaxInSamplesY. numDecPics. Therefore, it represents the maximum number of samples that the decoder should be able to decode for each access unit (for each image).

[0072] Table 3 shows the tier and level restrictions for video tiers (i.e., Table 136 in the VVC standard). According to Table 3, the restrictions for a given tier, level, and tier apply to all layers of a given OLS. That is, if, for example, two layers must be decoded, the MaxLumaSr of one level will be doubled compared to a single layer to set the appropriate level to use.

[0073]

[0074] Below is an example of Multilayer Main 10. Bitstreams conforming to Multilayer Main 10 should adhere to the following constraints: - The referenced SPS should have sps_chroma_format_idc (grayscale or 4:2:0 YUV video) equal to 0 or 1.

[0075] - The referenced SPS should have sps_bitdepth_minus8 (8 or 10 bits) in the range of 0 to 2 (inclusive).

[0076] - The referenced SPS should have a sps_palette_enabled_flag equal to 0 (palette tool not used).

[0077] - In a bitstream conforming to Multilayer Main 10 level, the general_level_idc and sublayer_level_idc[i] in the referenced VPS (when available) and referenced SPS should not be equal to 255 for all i values ​​(which indicates level 15.5).

[0078] - The hierarchical and level constraints specified for Multilayer Main 10 should be satisfied where applicable.

[0079] The bitstream conforms to Multilayer Main 10 profiles, indicated by general_profile_idc with a value of 17.

[0080] A decoder conforming to Multilayer Main 10 at a specific level within a specific layer should be able to decode all bitstreams that satisfy all of the following conditions: Condition 1: The bitstream is indicated to conform to Multilayer Main 10, Main 10 or Main 10 StillPicture format.

[0081] Condition 2: The bitstream is indicated to conform to a level lower than or equal to the specified level.

[0082] Condition 3: The bitstream is indicated to be a level that is not level 15.5 and is lower than or equal to the specified level.

[0083] The SPS contains information about the video sequence. Example SPS information is provided in Table 4.

[0084]

[0085] The semantics of the syntax elements used in Table 4 are specified below.

[0086] sps_seq_parameter_set_id provides an identifier for SPS to be referenced by other syntax elements.

[0087] When sps_video_parameter_set_id is greater than 0, it specifies the vps_video_parameter_set_id value used for the VPS referenced by SPS.

[0088] sps_max_sublayers_minus1 plus 1 specifies the maximum number of time sublayers that can exist in each CLVS referencing SPS.

[0089] sps_chroma_format_idc specifies the chroma sampling relative to the luminance sampling.

[0090] The fact that sps_ptl_dpb_hrd_params_present_flag is equal to 1 indicates that the profile_tier_level() syntax structure and the dpb_parameters() syntax structure exist in SPS, and the general_timing_hrd_parameters() syntax structure and the ols_timing_hrd_parameters() syntax structure can also exist in SPS.

[0091] sps_pic_width_max_in_luma_samples specifies the maximum width in luminance samples for each decoded image referencing SPS. sps_pic_width_max_in_luma_samples should not be equal to 0 and should be an integer multiple of Max(8, MinCbSizeY).

[0092] The requirement for bitstream consistency is that the value of sps_pic_width_max_in_luma_samples should be less than or equal to the value of vps_ols_dpb_pic_width[i] when sps_video_parameter_set_id is greater than 0 and SPS is referenced by a layer included in the i-th multilayer OLS specified by VPS (for any i in the range of 0 to NumMultiLayerOlss-1, inclusive).

[0093] sps_pic_height_max_in_luma_samples specifies the maximum height in luminance samples for each decoded image referencing SPS. sps_pic_height_max_in_luma_samples should not be equal to 0 and should be an integer multiple of Max(8, MinCbSizeY).

[0094] When sps_video_parameter_set_id is greater than 0 and SPS is referenced by a layer included in the i-th multilayer OLS specified by VPS (for any i in the range of 0 to NumMultiLayerOlss-1 (inclusive), the requirement for bitstream consistency is that the value of sps_pic_height_max_in_luma_samples should be less than or equal to the value of vps_ols_dpb_pic_height[i].

[0095] The VPS contains information about the video, particularly regarding multi-tier configurations. For example, the number of tiers, dependencies between tiers, and the output tier set (OLS – layers that can be output together) are transmitted within the VPS. This is transmitted in NAL units of the VPS type. It is optional for single-tier video but mandatory for multi-tier video.

[0096] The syntax of VPS in VVC is described in Table 5, and the semantics of the syntax elements are explained below.

[0097]

[0098] The semantics of the syntax elements used in Table 5 are specified below.

[0099] The VPS RBSP (Raw Byte Sequence Payload: a syntax structure containing an integer number of bytes, encapsulated in a NAL unit) should be available for the decoding process before being referenced, either in at least one AU with a TemporalId equal to 0 or provided by external means. All VPS NAL units with a specific vps_video_parameter_set_id value in the CVS should have the same content.

[0100] `vps_video_parameter_set_id` provides an identifier for the VPS that can be referenced by other syntax elements. The value of `vps_video_parameter_set_id` should be greater than 0.

[0101] The increment of 1 in vps_max_layers_minus1 specifies the number of layers specified by the VPS, which is the maximum number of layers allowed in each CVS referencing the VPS.

[0102] Increasing 1 to `vps_max_sublayers_minus1` specifies the maximum number of time-limited sublayers that can exist within the layers specified by the VPS. The value of `vps_max_sublayers_minus1` should be in the range of 0 to 6 (inclusive).

[0103] A value of 1 for `vps_default_ptl_dpb_hrd_max_tid_flag` indicates that the syntax elements `vps_ptl_max_tid[i]`, `vps_dpb_max_tid[i]`, and `vps_hrd_max_tid[i]` do not exist and are inferred to be equal to the default value `vps_max_sublayers_minus1`. A value of 0 for `vps_default_ptl_dpb_hrd_max_tid_flag` indicates that the syntax elements `vps_ptl_max_tid[i]`, `vps_dpb_max_tid[i]`, and `vps_hrd_max_tid[i]` exist. When they do not exist, the value of `vps_default_ptl_dpb_hrd_max_tid_flag` is inferred to be 1.

[0104] A `vps_all_independent_layers_flag` value of 1 indicates that all layers specified by the VPS are encoded independently without using inter-layer prediction. A `vps_all_independent_layers_flag` value of 0 indicates that one or more layers specified by the VPS can use inter-layer prediction. When it does not exist, the value of `vps_all_independent_layers_flag` is inferred to be equal to 1.

[0105] vps_layer_id[i] specifies the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​of m and n, when m is less than n, the value of vps_layer_id[m] should be less than vps_layer_id[n].

[0106] A value of 1 for `vps_independent_layer_flag[i]` indicates that the layer at index `i` does not use inter-layer prediction. A value of 0 for `vps_independent_layer_flag[i]` indicates that the layer at index `i` can use inter-layer prediction, and the syntax element `vps_direct_ref_layer_flag[i][j]` exists in the VPS, where `j` is in the range of 0 to `i-1` (inclusive). When it does not exist, the value of `vps_independent_layer_flag[i]` is inferred to be 1.

[0107] A value of 1 for `vps_max_tid_ref_present_flag[i]` indicates that the syntax element `vps_max_tid_il_ref_pics_plus1[i][j]` can exist. A value of 0 for `vps_max_tid_ref_present_flag[i]` indicates that the syntax element `vps_max_tid_il_ref_pics_plus1[i][j]` does not exist.

[0108] `vps_direct_ref_layer_flag[i][j]` equal to 0 indicates that the layer at index `j` is not a direct reference layer for the layer at index `i`. `vps_direct_ref_layer_flag[i][j]` equal to 1 indicates that the layer at index `j` is a direct reference layer for the layer at index `i`. `vps_direct_ref_layer_flag[i][j]` is inferred to be 0 if neither `i` nor `j` exists in the range 0 to `vps_max_layers_minus1` (inclusive). When `vps_independent_layer_flag[i]` equals 0, there should exist at least one `j` value in the range 0 to `i-1` (inclusive) such that `vps_direct_ref_layer_flag[i][j]` equals 1. The variables `NumDirectRefLayers[i]`, `DirectRefLayerIdx[i][d]`, `NumRefLayers[i]`, `ReferenceLayerIdx[i][r]`, and `LayerUsedAsRefLayerFlag[j]` are derived as follows:

[0109] The derivation of the variable GeneralLayerIdx[i], which specifies the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i], is as follows:

[0110] For any two distinct values ​​of i and j (both within the range of 0 to vps_max_layers_minus1 (inclusive)), when dependencyFlag[i][j] equals 1, the requirement for bitstream consistency is that the values ​​of sps_chroma_format_idc and sps_bitdepth_minus8 applied to layer i should be equal to the values ​​of sps_chroma_format_idc and sps_bitdepth_minus8 applied to layer j, respectively.

[0111] A value of 0 for `vps_max_tid_il_ref_pics_plus1[i][j]` indicates that the image from layer j is not used for the ILRP (inter-layer reference pictures) used for decoding the image from layer i. A value greater than 0 for `vps_max_tid_il_ref_pics_plus1[i][j]` indicates that, for decoding the image from layer i, images from layer j with a TemporalId greater than `vps_max_tid_il_ref_pics_plus1[i][j] - 1` are not used for the ILRP, and APSs with nuh_layer_id equal to `vps_layer_id[j]` and a TemporalId greater than `vps_max_tid_il_ref_pics_plus1[i][j] - 1` are not referenced. When not present, the value of `vps_max_tid_il_ref_pics_plus1[i][j]` is inferred to be equal to `vps_max_sublayers_minus1 + 1`.

[0112] `vps_each_layer_is_an_ols_flag` equal to 1 indicates that each OLS specified by the VPS contains only one layer, and each layer specified by the VPS is a single-container OLS with a single output layer. `vps_each_layer_is_an_ols_flag` equal to 0 indicates that at least one OLS specified by the VPS contains more than one layer. If `vps_max_layers_minus1` equals 0, the value of `vps_each_layer_is_an_ols_flag` is inferred to be 1. Otherwise, when `vps_all_independent_layers_flag` equals 0, the value of `vps_each_layer_is_an_ols_flag` is inferred to be 0.

[0113] The value of vps_ols_mode_idc equal to 0 indicates that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1. The i-th OLS includes layers with layer indices from 0 to i (inclusive of endpoints), and for each OLS, the highest layer in the OLS is the output layer.

[0114] The value of vps_ols_mode_idc equals 1, which means that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1. The i-th OLS includes layers with layer indices from 0 to i (inclusive of endpoints), and for each OLS, all layers in the OLS are output layers.

[0115] The value of vps_ols_mode_idc equal to 2 specifies that the total number of OLS specified by the VPS is explicitly signaled, and for each OLS, the output layer is explicitly signaled, while the other layers are layers that serve as direct or indirect reference layers to the OLS output layer.

[0116] The value of vps_ols_mode_idc should be in the range of 0 to 2 (inclusive). A value of 3 for vps_ols_mode_idc is reserved for future use by ITU-T|ISO / IEC. Decoders conforming to this specification version should ignore OLS values ​​where vps_ols_mode_idc equals 3.

[0117] When vps_all_independent_layers_flag equals 1 and vps_each_layer_is_an_ols_flag equals 0, the value of vps_ols_mode_idc is inferred to be equal to 2.

[0118] The addition of 2 to vps_num_output_layer_sets_minus2 specifies the total number of OLS specified by the VPS when vps_ols_mode_idc equals 2.

[0119] The derivation of the variable olsModeIdc is as follows:

[0120] The derivation of the variable TotalNumOlss, which specifies the total number of OLS values ​​by the VPS, is as follows:

[0121] A value of 1 for `vps_ols_output_layer_flag[i][j]` indicates that when `vps_ols_mode_idc` equals 2, the layer whose `nuh_layer_id` equals `vps_layer_id[j]` is the output layer of the i-th OLS. A value of 0 for `vps_ols_output_layer_flag[i][j]` indicates that when `vps_ols_mode_idc` equals 2, the layer whose `nuh_layer_id` equals `vps_layer_id[j]` is not the output layer of the i-th OLS.

[0122] Note: When vps_ols_mode_idc equals 2, vps_ols_output_layer_flag[0][j] does not exist, and if j == 0, it can be inferred that it is equal to 1, otherwise it is 0 (the first OLS only contains the first layer).

[0123] The variables NumOutputLayersInOls[i], specifying the number of output layers in the i-th OLS; NumSubLayersInLayerInOLS[i][j], specifying the number of sublayers in the j-th layer of the i-th OLS; OutputLayerIdInOls[i][j], specifying the nuh_layer_id value of the j-th output layer in the i-th OLS; and LayerUsedAsOutputLayerFlag[k], specifying whether the k-th layer is used as an output layer in at least one OLS, are derived from the VPS information as follows:

[0124] The derivation of the variables NumLayersInOls[i] (specifying the number of layers in the i-th OLS), LayerIdInOls[i][j] (specifying the nuh_layer_id value of the j-th layer in the i-th OLS), NumMultiLayerOlss (specifying the number of multi-layer OLSs, i.e., OLSs containing more than one layer), and MultiLayerOlsIdx[i] (specifying the index of the i-th OLS in the multi-layer OLS list when NumLayersInOls[i] is greater than 0) is as follows:

[0125] Note 1 - The 0th OLS contains only the lowest layer (i.e., the layer whose nuh_layer_id is equal to vps_layer_id[0]), and for the 0th OLS, the only layer contained is output.

[0126] The lowest layer in each OLS should be an independent layer. In other words, for each i in the range of 0 to TotalNumOlss-1 (inclusive), the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] should be equal to 1. Each layer should be included in at least one OLS specified by the VPS. In other words, for each layer with a nuh_layer_id-specific value nuhLayerId equal to one of vps_layer_id[k] (k being in the range of 0 to vps_max_layers_minus1 (inclusive)), there should exist at least one pair of values ​​for i and j (where i is in the range of 0 to TotalNumOlss-1 (inclusive) and j is in the range of 0 to NumLayersInOls[i]-1 (inclusive)) such that the value of LayerIdInOls[i][j] is equal to nuhLayerId.

[0127] Incrementing `vps_num_ptls_minus1` by 1 specifies the number of `profile_tier_level()` syntax structures in the VPS. The value of `vps_num_ptls_minus1` should be less than `TotalNumOlss`. If it does not exist, the value of `vps_num_ptls_minus1` is inferred to be equal to 0.

[0128] A value of 1 for `vps_pt_present_flag[i]` indicates that tier, level, and general constraint information exists in the i-th `profile_tier_level()` syntax structure of the VPS. A value of 0 for `vps_pt_present_flag[i]` indicates that tier, level, and general constraint information does not exist in the i-th `profile_tier_level()` syntax structure of the VPS. The value of `vps_pt_present_flag[0]` is inferred to be 1. When `vps_pt_present_flag[i]` is 0, the tier, level, and general constraint information used for the i-th `profile_tier_level()` syntax structure of the VPS is inferred to be the same as the tier, level, and general constraint information used for the (i-1)-th `profile_tier_level()` syntax structure of the VPS.

[0129] `vps_ptl_max_tid[i]` specifies the TemporalId of the highest sublayer representation with level information in the `i`th `profile_tier_level()` syntax structure in the VPS, and the TemporalId of the highest sublayer representation in the OLS where `olsIdx` is equal to `vps_ols_ptl_idx[olsIdx]`. The value of `vps_ptl_max_tid[i]` should be in the range of 0 to `vps_max_sublayers_minus1` (inclusive). When `vps_default_ptl_dpb_hrd_max_tid_flag` is equal to 1, the value of `vps_ptl_max_tid[i]` is inferred to be equal to `vps_max_sublayers_minus1`.

[0130] vps_ptl_alignment_zero_bit should be equal to 0.

[0131] vps_ols_ptl_idx[i] specifies the index of the profile_tier_level() syntax structure in the list of profile_tier_level() syntax structures in the VPS that applies to the i-th OLS. When present, the value of vps_ols_ptl_idx[i] should be in the range of 0 to vps_num_ptls_minus1 (inclusive).

[0132] When it does not exist, the value of vps_ols_ptl_idx[i] is inferred as follows: - If vps_num_ptls_minus1 equals 0, then the value of vps_ols_ptl_idx[i] is inferred to be equal to 0.

[0133] - Otherwise (vps_num_ptls_minus1 is greater than 0 and vps_num_ptls_minus1+1 equals TotalNumOlss), the value of vps_ols_ptl_idx[i] is inferred to be equal to i.

[0134] When NumLayersInOls[i] equals 1, the profile_tier_level() syntax structure applicable to the i-th OLS also exists in the SPS referenced by the layer in the i-th OLS. The requirement for bitstream consistency is that when NumLayersInOls[i] equals 1, the profile_tier_level() syntax structure signaled for the i-th OLS in the VPS and SPS should be the same.

[0135] Each profile_tier_level() syntax structure in a VPS should be referenced by at least one value of vps_ols_ptl_idx[i], where i is in the range of 0 to TotalNumOlss-1 (inclusive).

[0136] vps_num_dpb_params_minus1 (if it exists) incremented by 1 specifies the number of dpb_parameters() syntax structures in the VPS. The value of vps_num_dpb_params_minus1 should be in the range of 0 to NumMultiLayerOlss-1 (inclusive).

[0137] The derivation of the variable VpsNumDpbParams, which specifies the number of dpb_parameters() syntax structures in a VPS, is as follows:

[0138] The `vps_sublayer_dpb_params_present_flag` flag controls the presence of the syntax elements `dpb_max_dec_pic_buffering_minus1[j]`, `dpb_max_num_reorder_pics[j]`, and `dpb_max_latency_increase_plus1[j]` in the `dpb_parameters()` syntax structure of the VPS when `vps_dpb_max_tid[i]` is greater than 0. Here, `j` is in the range of 0 to `vps_dpb_max_tid[i]-1` (inclusive). When `j` does not exist, the value of `vps_sublayer_dpb_params_present_flag` is inferred to be equal to 0.

[0139] `vps_dpb_max_tid[i]` specifies the TemporalId of the highest sublayer representation of DPB parameters that can exist in the `i`th `dpb_parameters()` syntax structure of the VPS. The value of `vps_dpb_max_tid[i]` should be in the range of 0 to `vps_max_sublayers_minus1` (inclusive). When it does not exist, the value of `vps_dpb_max_tid[i]` is inferred to be equal to `vps_max_sublayers_minus1`.

[0140] For each m-th multilayer OLS in the range m from 0 to NumMultiLayerOlss-1 (inclusive), the value of vps_dpb_max_tid[vps_ols_dpb_params_idx[m]] should be greater than or equal to vps_ptl_max_tid[vps_ols_ptl_idx[n]], where n is the OLS index of the m-th multilayer OLS among all OLS.

[0141] vps_ols_dpb_pic_width[i] specifies the width, in luminance samples, of each image storage buffer used for the i-th multi-layer OLS.

[0142] vps_ols_dpb_pic_height[i] specifies the height in luminance samples for each image storage buffer used in the i-th multi-layer OLS.

[0143] vps_ols_dpb_chroma_format[i] specifies the maximum allowed value of sps_chroma_format_idc for all SPS referenced by CLVS in CVS for the i-th multi-level OLS.

[0144] vps_ols_dpb_bitdepth_minus8[i] specifies the maximum allowed value of sps_bitdepth_minus8 for all SPS referenced by CLVS in CVS for the i-th multi-level OLS. The value of vps_ols_dpb_bitdepth_minus8[i] should be in the range of 0 to 2 (inclusive).

[0145] Note 2 - In order to decode the i-th multi-layer OLS, the decoder can safely allocate memory for DPB based on the values ​​of the syntax elements vps_ols_dpb_pic_width[i], vps_ols_dpb_pic_height[i], vps_ols_dpb_chroma_format[i], and vps_ols_dpb_bitdepth_minus8[i].

[0146] `vps_ols_dpb_params_idx[i]` specifies the index of the `dpb_parameters()` syntax structure in the list of `dpb_parameters()` syntax structures in the VPS that applies to the `i`-th tier of OLS. When present, the value of `vps_ols_dpb_params_idx[i]` should be in the range of 0 to `VpsNumDpbParams-1` (inclusive).

[0147] When vps_ols_dpb_params_idx[i] does not exist, the following inference is made: - If VpsNumDpbParams equals 1, then the value of vps_ols_dpb_params_idx[i] equals 0.

[0148] - Otherwise (VpsNumDpbParams is greater than 1 and equal to NumMultiLayerOlss), the value of vps_ols_dpb_params_idx[i] is inferred to be equal to i.

[0149] For a single-layer OLS, the applicable dpb_parameters() syntax structure exists in the SPS referenced by the layer in the OLS.

[0150] Each dpb_parameters() syntax structure in a VPS should be referenced by at least one value of vps_ols_dpb_params_idx[i], where i is in the range of 0 to NumMultiLayerOlss-1 (inclusive).

[0151] A value of 1 for `vps_timing_hrd_params_present_flag` indicates that the VPS includes the `general_timing_hrd_parameters()` syntax structure and other HRD parameters. A value of 0 for `vps_timing_hrd_params_present_flag` indicates that the VPS does not include the `general_timing_hrd_parameters()` syntax structure or other HRD parameters.

[0152] When NumLayersInOls[i] equals 1, the general_timing_hrd_parameters() syntax structure and the ols_timing_hrd_parameters() syntax structure applicable to the i-th OLS exist in the SPS referenced by the layer in the i-th OLS.

[0153] A value of 1 for `vps_sublayer_cpb_params_present_flag` indicates that the i-th `ols_timing_hrd_parameters()` syntax structure in the VPS contains HRD parameters for the sublayer representation of the TemporalId in the range of 0 to `vps_hrd_max_tid[i]` (inclusive). A value of 0 for `vps_sublayer_cpb_params_present_flag` indicates that the i-th `ols_timing_hrd_parameters()` syntax structure in the VPS contains only HRD parameters for the sublayer representation of the TemporalId equal to `vps_hrd_max_tid[i]`. When `vps_max_sublayers_minus1` equals 0, the value of `vps_sublayer_cpb_params_present_flag` is inferred to be 0.

[0154] When `vps_sublayer_cpb_params_present_flag` equals 0, the HRD parameters for sublayer representations with TemporalId in the range of 0 to `vps_hrd_max_tid[i]-1` (inclusive) are inferred to be the same as the HRD parameters for sublayer representations with TemporalId equal to `vps_hrd_max_tid[i]`. These include the HRD parameters from the `fixed_pic_rate_general_flag[i]` syntax element up to the `sublayer_hrd_parameters(i)` syntax structure immediately following the condition `if(general_vcl_hrd_params_present_flag)` in the `ols_timing_hrd_parameters` syntax structure.

[0155] Increasing `vps_num_ols_timing_hrd_params_minus1` by 1 specifies the number of `ols_timing_hrd_parameters()` syntax structures present in the VPS when `vps_timing_hrd_params_present_flag` equals 1. The value of `vps_num_ols_timing_hrd_params_minus1` should be in the range of 0 to `NumMultiLayerOlss-1` (inclusive).

[0156] `vps_hrd_max_tid[i]` specifies the TemporalId of the highest sublayer representation containing the HRD parameters in the `i`th `ols_timing_hrd_parameters()` syntax structure. The value of `vps_hrd_max_tid[i]` should be in the range of 0 to `vps_max_sublayers_minus1` (inclusive). When it does not exist, the value of `vps_hrd_max_tid[i]` is inferred to be equal to `vps_max_sublayers_minus1`.

[0157] For each m-th multilayer OLS in the range m from 0 to NumMultiLayerOlss-1 (inclusive), the value of vps_hrd_max_tid[vps_ols_timing_hrd_idx[m]] should be greater than or equal to vps_ptl_max_tid[vps_ols_ptl_idx[n]], where n is the OLS index of the m-th multilayer OLS among all OLS.

[0158] `vps_ols_timing_hrd_idx[i]` specifies the index of the `ols_timing_hrd_parameters()` syntax structure in the list of `ols_timing_hrd_parameters()` syntax structures in the VPS that applies to the `i`-th tier OLS. The value of `vps_ols_timing_hrd_idx[i]` should be in the range of 0 to `vps_num_ols_timing_hrd_params_minus1` (inclusive).

[0159] When vps_ols_timing_hrd_idx[i] does not exist, the following inference is made: - If vps_num_ols_timing_hrd_params_minus1 equals 0, then the value of vps_ols_timing_hrd_idx[i] is inferred to be equal to 0.

[0160] - Otherwise (vps_num_ols_timing_hrd_params_minus1+1 is greater than 1 and equal to NumMultiLayerOlss), the value of vps_ols_timing_hrd_idx[i] is inferred to be equal to i.

[0161] For a single-layer OLS, the applicable ols_timing_hrd_parameters() syntax structure exists in the SPS referenced by the layer in the OLS.

[0162] Each ols_timing_hrd_parameters() syntax structure in the VPS should be referenced by at least one value of vps_ols_timing_hrd_idx[i], where i is in the range of 1 to NumMultiLayerOlss-1 (inclusive of endpoints).

[0163] A `vps_extension_flag` value of 0 indicates that the `vps_extension_data_flag` syntax element does not exist in the VPS RBSP syntax structure. A `vps_extension_flag` value of 1 indicates that the `vps_extension_data_flag` syntax element may exist in the VPS RBSP syntax structure. In bitstreams conforming to this specification, `vps_extension_flag` should be equal to 0. However, some uses of `vps_extension_flag` being equal to 1 may be specified in some future versions of this specification, and decoders conforming to this specification should allow a `vps_extension_flag` value equal to 1 to appear in the syntax.

[0164] The `vps_extension_data_flag` flag can have any value. Its presence and value do not affect the decoding process specified in this version of the specification. Decoders conforming to this version of the specification should ignore all `vps_extension_data_flag` syntax elements.

[0165] A multi-tier decoder needs to be able to decode several layers per frame. For each image in a layer, the encoder's complete toolset can be used. Therefore, the processing power required by the decoder can be the sum of the processing power required to decode each layer. PTL can set constraints for each OLS, but not for each layer. In practice, the constraint will be roughly the worst case of all decoded layers (e.g., layer L1 in the example below) multiplied by the number of layers, which may be overestimated, as shown in the example below. In VVC, tiers, levels, and grades are defined for a given OLS and optionally for sublayers, but not for each layer. Sublayers refer to temporal scalability and temporal layers. Classic GOP structures can be temporally scalable, with some sublayers that can be dropped to reduce the frame rate. Sublayer definitions assume that each layer is based on the same GOP structure and encoder configuration. The number of images that can be stored in the DPB and the number of frames that can be decoded per second depends on the maximum size of the images and is independent of the multi-tier configuration. This is not optimal in spatially scalable scenarios when lower layers have reduced sizes. This lack of per-layer granularity leads to over-designing decoders for some multi-tier configurations.

[0166] In this example, consider a spatially scalable bitstream with two layers and a scaling factor of 2. Layer 0 (L0), the first or base layer (which is an independent layer), is HD (High Definition) color video (1920×1080 pixels) with 4:2:0 color sampling. Layer 1 (L1), the second or enhancement layer, is UHD (Ultra High Definition) color video (3840×2160) with 4:2:0 color sampling. This enhancement layer depends on layer L0 and uses a scaling factor of 2 in both directions. Table 6 shows the DPB size and frame rate PTL constraints for multi-layer decoding used in this example.

[0167]

[0168] (1) Let numDecPics be the number of images in the AU (Access Unit). Set the variable AuSizeMaxInSamplesY to be equal to PicSizeMaxInSamplesY. numDecPics.

[0169] The HD value indicates only the base layer, while the UHD value is given as an example of standalone UHD video. According to the PTL definition in VVC (using PicSizeMaxInSamplesY), using level 5.1 will only allow a frame rate of 32 frames per second (fps) (calculated as MaxLumaSr / AuSizeMaxInSamplesY), and an 8-picture DPB size that will be shared between the two layers. For higher frame rates, the level must be increased to level 5.2 to make the bitstream compliant. If the number of pictures in the DPB is larger (e.g., 6 pictures per layer, which would require storing 12 pictures in the DPB to decode 2 layers), then level 6 must be chosen. In practice, in this case, the DPB capacity is calculated based on the PicSizeMaxInSamplesY value associated with the UHD layer L1, regardless of the fact that layer L0 has a lower resolution. In other words, using these properties—two layers, a frame rate higher than 32 fps, and HD / UHD resolution—the bitstream will be labeled a level 5.2 bitstream, and the decoder will need to conform to at least level 5.2 to decode it.

[0170] The last column shows the technical requirements for adding two layers, taking into account the reduced resolution of the base layer. In this case, its constraints will be lower than those of the enhancement layer. Based on the sample-per-second decoding capability, a feasible frame rate of 52 frames per second (e.g., calculated) can be obtained for decoding the complete OLS (L0+L1), instead of 32 frames per second. In fact, in this case, PicSizeMaxInSamplesY equals the number of samples in the two images L0 and L1, thus achieving a frame rate of 52 fps (calculated as MaxLumaSr / sum(PicSizeMaxInSamplesPerLayer), where PicSizeMaxInSamplesPerLayer is the maximum pixel size of each layer). Furthermore, if the memory sizes of each layer are decoupled from each other, more frames can be allocated in the DPB, specifically {2,8} (2 for the base layer, 8 for the enhancement layer), or {6,7}, or {10,6}.

[0171] This disclosure describes how tiers and constraints can be adapted to multi-layer coding constraints. Typically, memory and processing power constraints can be derived differently for each layer, or the syntax can be adapted for each (sub)layer. In particular, defining constraints for a specific access unit or a specific layer allows for the avoidance of over-designing the DPB.

[0172] Figure 5A A flowchart illustrating a method for checking bitstream consistency, based on an example, is provided. This method can be implemented in an encoder or decoder.

[0173] In S500, information is obtained representing the size (e.g., in luminance samples) of the images associated with the current OLS in the multi-layer bitstream, which are contained in the same access unit (i.e., these images are associated with the same time used for output from the DPB). In the example, this information is the value of the size. The OLS is defined over the entire duration of the video. In this paper, we consider images associated with the same time used for output from the DPB in the OLS, i.e., images belonging to the same AU. The size is, for example, the sum of the number of luminance samples of the images associated with the current OLS in the same access unit. As an example, in the example above, the access unit for the complete OLS (L0+L1) includes an HD image from layer L0 and a UHD image from layer L1. Therefore, the luminance sample size of the images associated with the current OLS is equal to 10,368,000 samples, i.e., 2,073,600 + 8,294,400.

[0174] This information can be obtained from newly defined syntax elements that are signaled in the bitstream, such as... Figure 6A Further, it can be shown that, alternatively, it can be derived from existing syntax elements signaled in the bitstream and defined, for example, in the VVC, which are transferred from their classical use. In S502, the DPB size (e.g., maximum size) for the level associated with the current OLS (e.g., the indicated level, i.e., the level indicated in the bitstream) is determined from the obtained information. The DPB size is, for example, the size in units of picture storage buffers, such as the sum of picture storage buffers for an AU. In a particular example, the DPB size is the maximum size of the DPB in units of picture storage buffers, where, taking into account the size of the pictures in the current OLS (obtained in S500), each picture storage buffer can store all the pictures of the same AU.

[0175] In S504, bitstream consistency is checked for the current OLS by comparing the DPB size determined for the level (e.g., indicated in the bitstream) with the DPB size required to decode the current OLS (e.g., the maximum size). The DPB size required to decode the current OLS can be obtained from the bitstream (e.g., parsed). The bitstream is consistent if the DPB size required to decode the current OLS is less than or equal to the DPB size determined for the level in S502. In practice, the DPB (decode picture buffer) that holds the picture before output must be large enough to handle multiple layers. The DPB size required to decode the current OLS can be obtained from the syntax elements signaled in the bitstream. A decoder conforming to the specification at a level higher than or equal to the level indicated by the current OLS should be able to decode the current OLS. If the decoder cannot decode the current OLS, it can signal it, request a new stream, or decode a different OLS, etc. Therefore, various actions can be taken in this situation.

[0176] Optionally, in S506, the feasibility of real-time decoding of the current OLS is determined (e.g., possibly) by comparing the luminance sampling rate of the indicated level with the obtained information. In the example, real-time decoding of the current OLS is feasible if the luminance sampling rate is greater than the size obtained in S500 multiplied by the frame rate (i.e., the number of images per second). The luminance sampling rate is obtained, for example, from the level indicated in the bitstream. A decoder conforming to a level higher than or equal to the indicated level should be able to decode the current OLS in real time. Referring to Table 3, a decoder conforming to level 5.0 is able to decode 267,386,880 samples per second.

[0177] The corresponding encoding method is in Figure 5B It is publicly available in China.

[0178] In S510, information representing the size of the image contained in the same access unit and associated with the current output layer set is obtained.

[0179] In S512, the acquired information is encoded in a syntax structure associated with the current output layer set.

[0180] Figure 6A A flowchart illustrating a method for checking bitstream consistency based on a specific example is provided. In this example, constraints are defined per access unit for the current OLS, which contains the pictures associated with the same time for output across all decoding layers. Variables PicWidthMaxInSamplesY and PicHeightMaxInSamplesY are defined as in VVC, and the variable AuSizeMaxInSamplesY is signaled for multi-layer bitstreams.

[0181] In S600, syntax elements (e.g., VPS syntax elements) are parsed (e.g., decoded) from the bitstream. This syntax is a newly defined syntax element (e.g., denoted as vps_ols_dpb_au_size[i]) that specifies the size (e.g., in luminance samples) of the image associated with the i-th multi-layer OLS and contained within the same access unit. For example, it is the sum of the number of luminance samples of the image associated with the i-th OLS and contained within the same access unit.

[0182] Therefore, a new VPS syntax structure can be defined in Table 7 below, where the newly defined syntax elements (indicated in italics and bold) are added to the existing VVC syntax elements in Table 5.

[0183]

[0184] From the parsed syntax elements, obtain information representing the size of the image associated with the current OLS in the multi-level stream and contained in the same access unit (AuSizeMaxInSamplesY).

[0185] For an OLS with an OLS index TargetOlsIdx, the derivation of the variable AuSizeMaxInSamplesY and the applicable dpb_parameters() syntax structure is as follows: If NumLayersInOls[TargetOlsIdx] equals 1, or if layer i is independent (and there exists an OLS containing only layer i), then AuSizeMaxInSamplesY is set to equal to sps_pic_width_max_in_luma_samples. sps_pic_height_max_in_luma_samples (as in VVC), where sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are found in the SPS referenced by the layer in OLS, and the applicable dpb_parameters() syntax structure is also found in that SPS.

[0186] Otherwise, AuSizeMaxInSamplesY is set to equal vps_ols_dpb_au_size[MultiLayerOlsIdx[TargetOlsIdx]], and the applicable dpb_parameters() syntax structure is identified by vps_layer_dpb_params_idx[MultiLayerOlsIdx[TargetOlsIdx]] found in the VPS.

[0187] In S602, in response to the obtained information, the size of the DPB associated with the current OLS (e.g., the level indicated in the bitstream, also known as the indicator level) is determined, denoted as MaxDpbSize. In a specific example, MaxDpbSize is the maximum size, in image storage buffers, of the DPB used for the indicator level given AuSizeMaxInSamplesY.

[0188] MaxDpbSize can be derived as follows:

[0189] MaxLumaPs is specified in Table 2 according to the indicator level, maxDpbPicBuf is equal to a constant value, such as 8, and dpb_max_dec_pic_buffering_minus1[Htid] is found in or derived from the applicable dpb_parameters() syntax structure (as previously described in VVC).

[0190] In S604, bitstream consistency is checked for the current OLS by comparing MaxDpbSize with the DPB size required to decode the current OLS. The DPB size required to decode the current OLS can be obtained from the value of the dpb_parameter() syntax element dpb_max_dec_pic_buffering_minus1[Htid]. When the specified level is not level 15.5, the bitstream consistency requirement is that the value of dpb_max_dec_pic_buffering_minus1[Htid]+1 is less than or equal to MaxDpbSize. Decoders that meet the requirements of a level higher than or equal to the specified level are able to decode the bitstream.

[0191] Optionally, in S606, the feasibility (e.g., possibility) of real-time decoding for the current OLS is determined by comparing the luminance sampling rate (e.g., the maximum luminance sampling rate) (e.g., rate MaxLumaSr) of the indicated level with the obtained information (AuSizeMaxInSamplesY). Referring to Table 3, a decoder conforming to level 5.0 is capable of decoding 267,386,880 samples per second. Therefore, for the level, the performance limit is defined by MaxLumaSr, where real-time decoding only occurs if MaxLumaSr is greater than or equal to AuSizeMaxInSamplesY. This can only be achieved with FrameRate.

[0192] The new consistency limitations are given in Table 8. This table shows that the spatially scalable examples (HD, UHD) fit at level 5.1 at 50Hz, and the DPB memory allows for 6 images per layer in the DPB. Therefore, the DPB memory allows for 6 images to be reserved for L0 and 6 images for L1, whereas in the previous configuration, the DPB memory could only reserve a total of 8 images at levels 5.x (i.e., 5.0, 5.1, and 5.2), as described in Table 6.

[0193]

[0194] exist Figure 6AIn variations of the described method, the newly defined syntax element `vps_ols_dpb_au_size` is not decoded. In other words, semantics and constraint derivation can be changed without altering the VVC syntax. For example, the VPS syntax elements `vps_ols_dpb_pic_width[i]` and `vps_ols_dpb_pic_height[i]` can be defined as image size parameters, rather than physical image sizes, to ensure their product equals the expected `AuSizeMaxInSamplesY` value, which is the value that would normally be transmitted in the syntax element `vps_ols_dpb_au_size[i]`. In other words, on the encoder side, the values ​​encoded for the syntax elements `vps_ols_dpb_pic_width[i]` and `vps_ols_dpb_pic_height[i]` are pseudo-values ​​(in the sense that they are not physical image sizes), defined to make their product equal to `AuSizeMaxInSamplesY`, such as 10368000 in the example in Table 7.

[0195] The value vps_ols_dpb_pic_height can have a predefined value that depends on the range into which AuSizeMaxInSamplesY falls. vps_ols_dpb_pic_height can take a predefined value from {32, 64, 128, 256, 512, 1080, 1600, 1920, 2048, 2160, 2500}.

[0196] If (NumLayersInOls[TargetOlsIdx] is greater than 1), then AuSizeMaxInSamplesY is set to be equal to vps_ols_dpb_pic_width[MultiLayerOlsIdx[TargetOlsIdx]] × vps_ols_dpb_pic_height[MultiLayerOlsIdx[TargetOlsIdx]], and the applicable dpb_parameters() syntax structure is identified by vps_ols_dpb_params_idx[MultiLayerOlsIdx[TargetOlsIdx]] found in the VPS.

[0197] Therefore, in S600, the syntax elements vps_ols_dpb_pic_width[i] and vps_ols_dpb_pic_height[i] are parsed (e.g., decoded) from the bitstream, and from the parsed syntax elements, information representing the size of the picture associated with the current OLS in the multi-layer stream and contained in the same access unit (AuSizeMaxInSamplesY) is obtained, where AuSizeMaxInSamplesY is equal to the product of the values ​​of the two parsed syntax elements.

[0198] In the example above, the value of AuSizeMaxInSamplesY is either derived from the newly defined syntax element vps_ols_dpb_au_size[i] or from the existing syntax elements vps_ols_dpb_pic_width[MultiLayerOlsIdx[TargetOlsIdx]] and vps_ols_dpb_pic_height[MultiLayerOlsIdx[TargetOlsIdx]].

[0199] Figure 6B A flowchart illustrating a method for checking bitstream consistency based on another specific example is provided. Figure 6A The steps of the described method that are identical are identified by the same numerical notation and will not be further disclosed. Therefore, Figure 6B The method includes steps S602 to S606. In this example, SPS syntax elements are parsed (e.g., decoded) from the bitstream, and the value of AuSizeMaxInSamplesY is derived from these SPS syntax elements.

[0200] More precisely, each layer requires information signaled from the SPS. In the example, the syntax elements of the SPS are the maximum width in luminance samples for each image in the reference sequence parameter set of the current output layer set and the maximum height in luminance samples for each image in the reference sequence parameter set of the current output layer set.

[0201] Therefore, in S610, for each layer of the current OLS, the dimensions of the images contained in that layer are obtained from the sequence parameter set syntax elements. As an example, for layer i, PicWidthMaxInSamplesY[i] is set to equal to sps_pic_width_max_in_luma_samples, PicHeightMaxInSamplesY[i] is set to equal to sps_pic_height_max_in_luma_samples, and PicSizeMaxInSamplesY[i] is set to equal to PicWidthMaxInSamplesY[i]. PicHeightMaxInSamplesY[i], where sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are found in the SPS referenced by layer i. The applicable dpb_parameters() syntax structure for layer i is also found in this SPS.

[0202] In S612, for the current OLS at index k, AuSizeMaxInSamplesY is therefore derived as the sum of PicSizeMaxInSamplesY[i] for each layer i decoded for that OLS. In the example, AuSizeMaxInSamplesY can be calculated as follows:

[0203] Then, AuSizeMaxInSamplesY can be used in S602 to determine the DPB size (e.g., MaxDpbSize) for the current OLS, to check bitstream consistency in S604, and to determine in S606, as previously described, whether real-time decoding of the current OLS is feasible.

[0204] In the following example, the value of AuSizeMaxInSamplesY is derived from per-level information. Per-OLS definitions are useful for overall requirements, but some per-level information will allow specifying lower levels and avoid over-designing the decoder, as explained in the example below. Therefore, to specify constraints more precisely and without overestimating the necessary levels, constraints can be defined per level rather than per OLS-AU. Then, based on the OLS information, per-level requirements are aggregated to derive per-OLS requirements, such as... Figure 7A As described. In the following text, the terms “requirement” and “constraint” are used interchangeably.

[0205] In S700, derivation (e.g., parsing from a bitstream) represents the syntax elements required for each layer. In the example, parameters can be defined, for example, in the VPS (Video Parameter Set) syntax structure, as described in Table 9. Newly defined elements are indicated in bold.

[0206]

[0207] The semantics of the newly defined syntax elements used in Table 9 are specified below.

[0208] vps_num_dpb_params_minus1 (if it exists) incremented by 1 specifies the number of dpb_parameters() syntax structures in the VPS. The value of vps_num_dpb_params_minus1 should be in the range of 0 to vps_max_layers_minus1 (inclusive).

[0209] The variable VpsNumDpbParams specifies the number of dpb_parameters() syntax structures in the VPS, as deduced below:

[0210] In the variant, this parameter is inferred to be equal to vps_max_layers_minus1+1 (the number of layers defined in the VPS) and is not signaled.

[0211] The `vps_sublayer_dpb_params_present_flag` flag controls the presence of the syntax elements `dpb_max_dec_pic_buffering_minus1[j]`, `dpb_max_num_reorder_pics[j]`, and `dpb_max_latency_increase_plus1[j]` in the `dpb_parameters()` syntax structure of the VPS when `vps_dpb_max_tid[i]` is greater than 0. Here, `j` is in the range of 0 to `vps_dpb_max_tid[i]-1` (inclusive). When `j` does not exist, the value of `vps_sublayer_dpb_params_present_flag` is inferred to be equal to 0.

[0212] `vps_dpb_max_tid[i]` specifies the TemporalId of the highest sublayer representation of DPB parameters that can exist in the `i`th `dpb_parameters()` syntax structure of the VPS. The value of `vps_dpb_max_tid[i]` should be in the range of 0 to `vps_max_sublayers_minus1` (inclusive). When it does not exist, the value of `vps_dpb_max_tid[i]` is inferred to be equal to `vps_max_sublayers_minus1`.

[0213] vps_layer_dpb_pic_width[i] specifies the width, in luminance samples, of the image storage buffer used for the i-th layer.

[0214] vps_layer_dpb_pic_height[i] specifies the height in luminance samples for each image storage buffer used in the i-th layer.

[0215] vps_layer_dpb_chroma_format[i] specifies the maximum allowed value of sps_chroma_format_idc for all SPS referenced by CLVS in CVS for layer i.

[0216] vps_layer_dpb_bitdepth_minus8[i] specifies the maximum allowed value of sps_bitdepth_minus8 for all SPS referenced by CLVS in CVS for layer i. The value of vps_layer_dpb_bitdepth_minus8[i] should be in the range of 0 to 2 (inclusive).

[0217] `vps_layer_dpb_params_idx[i]` specifies the index of the `dpb_parameters()` syntax structure in the list of `dpb_parameters()` syntax structures in the VPS that applies to the `i`-th layer. When present, the value of `vps_layer_dpb_params_idx[i]` should be in the range of 0 to `VpsNumDpbParams-1` (inclusive).

[0218] When vps_layer_dpb_params_idx[i] does not exist, the following inference is made: - If VpsNumDpbParams equals 1, then the value of vps_layer_dpb_params_idx[i] equals 0.

[0219] - Otherwise (VpsNumDpbParams is greater than 1 and equal to vps_max_layers_minus1+1), the value of vps_layer_dpb_params_idx[i] is inferred to be equal to i.

[0220] For independent layers, the applicable dpb_parameters() syntax structure can exist in the SPS referenced by that layer.

[0221] Each dpb_parameters() syntax structure in a VPS should be referenced by at least one value of vps_layer_dpb_params_idx[i], where i is in the range from 0 to vps_max_layers_minus1 (inclusive).

[0222] Other semantics remain unchanged.

[0223] In another example, the elements under the loop for all layers (for( i = 0; i<= vps_max_layers_minus1;i++ )) exist only for non-independent layers. For independent layers, if there exists an OLS that only includes that given layer, these elements are signaled in the SPS referenced by that layer. In this case, the index of the element can remain unchanged, the signaling for independent layers can be skipped, or the index can be changed and indirect addressing can be added to retrieve the correct index from the layer number. In the following, it is assumed that the same index is maintained (i.e., the layer index is used to index the signaled elements).

[0224] In S702, OLS information is parsed from the bitstream. This information allows us to determine which layer(s) are being decoded. Examples of this type of information are defined in Table 5, such as: `vps_direct_ref_layer_flag`, which defines the dependencies between layers; `vps_ols_mode_idc`; and `vps_ols_output_layer_flag`, which defines which layers are output. OLS information can be found in SPS, VPS, etc.

[0225] In S704, for the current OLS (e.g., identified by its index), the requirements for each OLS (e.g., in terms of memory size) are derived from the requirements for each layer obtained in S700. Therefore, the verification of the DPB memory size requirements for decoding an OLS containing more than one layer (otherwise, if the OLS contains only one layer, it does not need to be changed compared to the current VVC) involves resolving the layer parameters of a given OLS in S700.

[0226] For the i-th layer (the layer where GeneralLayerIdx[nuh_layer_id]=i), in the OLS with index TargetOlsIdx, the derivation of variables PicWidthMaxInSamplesY[i], PicHeightMaxInSamplesY[i], and PicSizeMaxInSamplesY[i], as well as the applicable dpb_parameters() syntax structure, is as follows: If NumLayersInOls[TargetOlsIdx] equals 1, or if layer i is independent (and there exists an OLS containing only layer i), then PicWidthMaxInSamplesY[i] is set to equals sps_pic_width_max_in_luma_samples, PicHeightMaxInSamplesY[i] is set to equals sps_pic_height_max_in_luma_samples, and PicSizeMaxInSamplesY[i] is set to equals PicWidthMaxInSamplesY[i]. PicHeightMaxInSamplesY[i], where sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are found in the SPS referenced by the layer in OLS, and the applicable dpb_parameters() syntax structure is also found in that SPS.

[0227] Otherwise, PicWidthMaxInSamplesY[i] is set to equal to vps_layer_dpb_pic_width[i], PicHeightMaxInSamplesY[i] is set to equal to vps_layer_dpb_pic_height[i], and PicSizeMaxInSamplesY[i] is set to equal to PicWidthMaxInSamplesY[i]. PicHeightMaxInSamplesY[i], and the applicable dpb_parameters() syntax structure is identified by vps_layer_dpb_params_idx[i] found in the VPS.

[0228] The DPB memory size OLSDPBMemorySize[Htid] required to decode each sub-layer Htid in the OLS is the sum of the memory (also called buffer) sizes required to decode all layers in the k-th OLS, and its derivation is as follows: - If vps_each_layer_is_an_ols_flag is not zero, then each OLS is a layer, and the dpb size requirement is the same as in VVC: OLSDPBMemorySize[Htid]=(dpb_max_dec_pic_buffering_minus1[Htid]+1) × PicSizeMaxInSamplesY[i], where dpb_max_dec_pic_buffering_minus1 is signaled in the dpb_parameters() syntax structure applicable to the k-th layer (i=k).

[0229] - Otherwise, it is the sum of the sizes of all decoding layers (including those in the OLS):

[0230] In S706, in response to the derived per-OLS requirement, bitstream consistency is checked for the current OLS. In the example, the derived DPB memory size OLSDPBMemorySize[Htid] required to decode the OLS is compared with m × MaxLumaPs.

[0231] When the specified level is not level 15.5, the requirement for bitstream consistency is that for any Htid, the value of OLSDPBMemorySize[Htid] is less than or equal to m × MaxLumaPs, where the value of MaxLumaPs is derived from the level signaled in the profile_tier_level syntax structure used for the k-th OLS. For example, in VVC, m=8. Therefore, a decoder that conforms to a level higher than or equal to the signaled level will be able to decode the k-th OLS.

[0232] The level defines the maximum number of image buffers in the DPB relative to the maximum image size used for that level. Given that each layer can have a different size, the DPB's memory can be allocated such that each layer in a given OLS can reserve a portion of the total available DPB memory. Therefore, in a variant, to check in S706 that the OLS of the bitstream conforms to the DPB image buffer memory constraints, when the specified level is not level 15.5, it is determined whether the value of dpbMaxDecPicBuffering[Htid] is less than P, where P is the number of images of the maximum size that can be accommodated in the DPB. For example, in VVC, P = 8. For a given OLSk, dpbMaxDecPicBuffering[Htid] is derived in S704 as follows: The number of image buffers required for one layer is derived as follows:

[0233] It should be noted that any fractional value can be used here. However, a simple fraction that reflects, for example, the scaling ratio between layers in a spatial scalability scene is useful (e.g., ¼ for HD to UHD scalability).

[0234]

[0235] Optionally, in S708, it is determined whether real-time decoding of the current OLS is feasible (e.g., possible). The computational performance required for a given level is represented by the maximum luminance sampling rate (MaxLumaSr). Consistency with a given level is checked by calculating AuSizeMaxInSamplesY. Here, for greater accuracy on multi-layer video, the maximum number of samples to be decoded is the sum of the number of samples in each layer of OLS k. Its derivation is as follows:

[0236] In this way, the value of AuSizeMaxInSamplesY will not be overestimated. Therefore, the performance limit is defined by MaxLumaSr, where real-time decoding only occurs if MaxLumaSr is greater than or equal to AuSizeMaxInSamplesY. This can only be achieved with FrameRate.

[0237] The corresponding encoding method is in Figure 7B It is publicly available in China.

[0238] In S710, the requirements for each layer are obtained from the image size and the image buffer, which depends on the GOP structure.

[0239] In S712, the requirements of each layer are encoded in a syntax structure, such as those mentioned in Table 9.

[0240] exist Figure 8 In another example depicted, the dpb_parameters() syntax is signaled according to OLS, as in current VVC. Therefore, to more tightly adjust constraints to fit multiple bitstreams, each layer's information is extracted (also known as parsing or decoding) if available. All values ​​are initialized, for example, as in current VVC, and then other values ​​can be adjusted if available in the bitstream (e.g., SPS, VPS). Figure 8 As depicted. In this embodiment, with Figure 7A The syntax remains the same, unlike the example above.

[0241] In S800, the first per-OLS requirement is parsed from the bitstream. As an example, these first per-OLS requirements are PicSizeMaxInSamplesY and dpb_max_dec_pic_buffering_minus1. In practice, the requirements (i.e., the values ​​of PicSizeMaxInSamplesY and dpb_max_dec_pic_buffering_minus1) are signaled according to the OLS, and therefore parsed in S800. In S802, the OLS information is parsed from the bitstream in the same manner as in S702.

[0242] In S804, each layer value (also called requirement / constraint) is derived from the value / requirement notified by the OLS signal, such as Layer_PicSizeMaxInSamplesY[i] and layer_DpbMaxDecPicBuffering[i]. layer_PicSizeMaxInSamplesY specifies the maximum size (in luminance samples) of an image used to decode layer i. In the above equation, it replaces PicSizeMaxInSamplesY[i] obtained for layer i.

[0243] `layer_DpbMaxDecPicBuffering[i]` specifies the maximum required size of the DPB (in image storage buffers) for decoding layer i. In the equation above, it replaces `(dpb_max_dec_pic_buffering_minus1[Htid]+1)` obtained for layer i. In the example, the requirement for each layer is derived by pairwise comparisons of OLS, as follows. Layer_PicSizeMaxInSamplesY[0] is set to equal to sps_pic_width_max_in_luma_samples × sps_pic_height_max_in_luma_samples, where sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are found in the SPS referenced by layer 0, and layer_DpbMaxDecPicBuffering[0] is derived from dpb_max_dec_pic_buffering_minus1, which is signaled in the applicable dpb_parameters() syntax structure and is also found in the SPS, as layer_DpbMaxDecPicBuffering[0] = dpb_max_dec_pic_buffering_minus1 + 1. Since the first layer (layer 0) is required to be independent, these values ​​are likely to exist in the SPS. If they do not exist, they are derived as in other layers, as described below.

[0244] The maximum image size of layer i (Layer_PicSizeMaxInSamplesY[i]) is set to the minimum size required for all OLS including this given layer, and OlsMinDpb_max_dec_pic_buffering_minus1[i] is set to the minimum number of DPB images required to decode the OLS containing this layer:

[0245] Next, the number of DPB images required for decoding layer i, layer_DpbMaxDecPicBuffering[i], is derived by subtracting the number of images required for decoding dependency layers from OlsMinDpb_max_dec_pic_buffering_minus1. We use the dependency table dependencyFlag[i][j] defined above:

[0246] Note that for layer i, the dependent layer is required to have a lower ID. This explains the "j" in the second for loop condition. <i”。

[0247] Subsequently, all necessary per-layer requirement values ​​are available, and the embodiment disclosed in the foregoing portion of Figure 7 (where the per-layer constraints are explicitly signaled) is used to precisely derive PTL consistency (PicSizeMaxInSamplesY replaces PicSizeMaxInSamplesY[i] obtained for layer i, and layer_DpbMaxDecPicBuffering[i] replaces dpb_max_dec_pic_buffering_minus1[Htid]+1 obtained for layer i). Performance limits can also be derived using these per-layer constraints. In other words, S806 is the same as S704, S808 is the same as S706, and S810 is the same as S708. Therefore, in S806, the second per-OLS requirement (e.g., OLSDPBMemorySize[Htid]) is derived, which is the same as the per-OLS requirement derived in S704.

[0248] It's important to note that this only applies when defining OLS incrementally. For example, consider three layers and two OLS: one with a base layer and the other with two other layers that depend on the base layer. In this case, the minimum size requirement for the two higher layers will be set to the maximum of the sizes of the two higher layers, and the DPB image requirement for each layer cannot be derived. Therefore, this method is only usable when vps_ols_mode_idc equals 0 or 1.

[0249] vps_ols_mode_idc>=2 is not a common setting for scalable use cases. In this case, per-OLS constraints are used directly because they are likely accurate enough (image sizes can be equal across different layers).

[0250] In yet another variant, per OLS constraint or per layer constraint can also be signaled based on vps_ols_mode_idc.

[0251] vps_ols_mode_idc>=2 is not a common setting for scalable use cases. In this case, per-OLS constraints may be accurate enough.

[0252] In this embodiment, whether constraints in the VPS are signaled by layer or by OLS is determined by the value of vps_ols_mode_idc (in the current VVC syntax, vps_ols_mode_idc is signaled before constraints are applied): - If vps_ols_mode_idc < 2, then the constraints (which include the vps_layer_dpb_pic_width, vps_layer_dpb_pic_height, and dpb_parameters syntax structures) are defined layer by layer as described above, and the consistency constraints are also derived as described above.

[0253] - Otherwise (if vps_ols_mode_idc>=2), the constraints (which include the vps_layer_dpb_pic_width, vps_layer_dpb_pic_height, and dpb_parameters syntax structures) are defined by OLS as in the current VVC.

[0254] This aspect is not limited to ECM, VVC, or HEVC, and can be applied to, for example, other standards and recommendations, as well as any extensions of such standards and recommendations. Unless otherwise stated or technically impractical, the aspects described in this application may be used alone or in combination.

[0255] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0256] It should be noted that the syntax elements (such as terms in equations and algorithms), signals (e.g., flags, patterns), and labels / names (such as MaxDpbSize, AuSizeMaxInSamplesY, MaxLumaSr, vps_ols_dpb_au_size[i], etc.) used in this article are descriptive terms. Therefore, they do not preclude the use of other syntax element names.

[0257] Various implementations involve decoding. As used herein, “decoding” can encompass, for example, all or part of a process performed on a received encoded sequence to produce an output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of various implementations described herein, such as bitstream consistency checks.

[0258] As other examples, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding; and in yet another embodiment, "decoding" refers to the entire image reconstruction process including entropy decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be clear from the specific context of the description and is believed to be fully understood by those skilled in the art.

[0259] Various implementations involve encoding. In a manner similar to the discussion of “decoding” above, “encoding,” as used herein, can encompass, for example, all or part of the process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also, or alternatively, include processes performed by encoders of various implementations described herein, such as bitstream consistency checks.

[0260] As other examples, in one embodiment, “encoding” refers only to entropy encoding; in another embodiment, “encoding” refers only to differential encoding; and in yet another embodiment, “encoding” refers to a combination of differential and entropy encoding. Whether the phrase “encoding process” is intended to specifically refer to a subset of operations or generally to a broader encoding process will be clear from the specific context of the description and is believed to be fully understood by those skilled in the art.

[0261] This disclosure describes various types of information (e.g., syntax) that can be transmitted or stored. This information can be packaged or arranged in various ways, including those common in video standards, such as placing the information in SPS, PPS, NAL units, headers (e.g., NAL unit headers or stripe headers), or SEI messages. Other methods are also available, including those common in system-level or application-level standards, such as placing the information in one or more of the following: a. SDP (Session Description Protocol), a format used to describe multimedia communication sessions for session notification and session invitation, such as as described in the RFC, and used in conjunction with RTP (Real-Time Transport Protocol) transmission.

[0262] b. DASH MPD (Media Presentation Description) descriptors, such as those used in DASH and transmitted via HTTP, are associated with a representation or set of representations to provide additional features to the content representation.

[0263] c. RTP header extensions, such as those used during RTP streaming.

[0264] d. ISO-based media file formats, such as those used in OMAF, and using object-oriented building blocks (also called “atoms” in some specifications) defined by unique type identifiers and lengths.

[0265] e. An HLS (HTTP Live Streaming) manifest transmitted via HTTP. The manifest can be associated, for example, with a version or set of versions of the content to provide characteristics of the version or set of versions.

[0266] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0267] Some implementations may involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often within constraints of given computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function that is a weighted sum of rate and distortion. Different approaches exist for solving rate-distortion optimization problems. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly by computation based on the predicted or approximate distortion of the residual signal rather than the reconstructed signal. A hybrid of these approaches can also be used, for example, using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of encoding costs and associated distortion.

[0268] The embodiments and aspects described herein can be implemented, for example, as methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), implementations of the discussed features can be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. These methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0269] References to "an embodiment" or "an embodiment" or "an implementation" or "implementation", and their other variations, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in one implementation" or "in one implementation", and any other variations appearing throughout various places in this application, do not necessarily refer to the same embodiment.

[0270] Furthermore, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0271] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0272] Furthermore, this application may refer to "receiving" various types of information. "Receiving," like "accessing," is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) from one or more sources. Moreover, "receiving" generally involves, in some way, operations such as, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0273] It should be understood that any use of “ / ”, “and / or”, and “…at least one of…” below, such as in the cases of “A / B”, “A and / or B”, and “at least one of A and B”, is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as listed, as is clear to a person skilled in the art.

[0274] Furthermore, as used herein, the term "signal" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals to indicate the size of the image associated with the current OLS in a multi-layer stream or a specific syntax element required by each layer. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in many embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be done in a variety of ways. For example, in many embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" may also be used as a noun herein.

[0275] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the embodiments described. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted via a variety of different wired or wireless links as known. The signal may be stored on a processor-readable medium.

[0276] This document describes numerous examples. Features of the examples may be provided individually or in any combination across multiple claim classes and types. Furthermore, examples may include one or more features, devices, or aspects described herein, individually or in any combination across multiple claim classes and types. For example, features described herein may be implemented in a bitstream or signal including information generated as described herein. This information may allow a decoder to decode a bitstream, encoder, bitstream, and / or decoder according to any of the described embodiments. For example, features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, features described herein may be implemented by a method, process, apparatus, medium storing instructions, medium storing data, or signal. For example, features described herein may be implemented by a TV, set-top box, cellular phone, tablet computer, or other electronic device performing decoding. The TV, set-top box, cellular phone, tablet computer, or other electronic device may display (e.g., using a monitor, screen, or other type of display) a resulting image (e.g., an image reconstructed from the residual of a video bitstream). The TV, set-top box, cellular phone, tablet computer, or other electronic device may receive a signal including an encoded image and perform decoding.

[0277] Many embodiments have been described above. The features of these embodiments may be provided individually or in any combination across a variety of claim classes and types.

[0278] A method for performing bitstream consistency checks on multi-layered streams is disclosed. The method includes: For each layer of the current output layer set of the multi-layer stream, obtain the size of the image contained in that layer from the syntax elements of the sequence parameter set referenced by that layer; For the current output layer set, information representing the size of the image contained in the layers decoded for the current output layer set and associated with the current output layer set is obtained by summing the sizes of the images contained in the layers decoded for the current output layer set; For each level that is signaled to the current output layer, the size of the decoded image buffer is determined in response to the information obtained; and The bitstream consistency for the current output layer set is checked by comparing the determined size of the decoded image buffer with the size required to decode the current output layer set.

[0279] An apparatus is disclosed comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform: For each layer of the current output layer set of the multi-layer stream, obtain the size of the image contained in that layer from the syntax elements of the sequence parameter set referenced by that layer; For the current output layer set, information representing the size of the image contained in the layers decoded for the current output layer set and associated with the current output layer set is obtained by summing the sizes of the images contained in the layers decoded for the current output layer set; For each level that is signaled to the current output layer, the size of the decoded image buffer is determined in response to the information obtained; and The bitstream consistency for the current output layer set is checked by comparing the determined size of the decoded image buffer with the size required to decode the current output layer set.

[0280] In the example, the syntax elements of the sequence parameter set are the maximum width in luminance samples for each image in the reference sequence parameter set of the current output layer set and the maximum height in luminance samples for each image in the reference sequence parameter set of the current output layer set.

[0281] In the example, checking bitstream consistency for the current output layer set includes determining that the multi-layer streams for the current output layer set are consistent if the size of the decoded image buffer required to decode the current output layer set is less than or equal to the determined size of the decoded image buffer.

[0282] In the example, the method includes (or one or more processors configured to implement) determining whether real-time decoding of the current output layer set is possible by comparing the luminance sampling rate of the level notified by the signal with the obtained information.

[0283] In the example, determining whether real-time decoding of the current output layer set is possible by comparing the luminance sampling rate of the signaled level with the obtained information includes: determining that real-time decoding of the current output layer set is possible if the luminance sampling rate of the signaled level is higher than the obtained information multiplied by the frame rate.

[0284] In the example, the size required to decode the current output layer set of the decoded image buffer is parsed from the decoded image buffer syntax structure.

[0285] A computer program comprising program code instructions, which, when executed by a processor, implements a method according to any one example.

Claims

1. A method for performing bitstream consistency checks on multi-layer streams, comprising: For each layer of the current output layer set of the multi-layer stream, obtain (S610) the maximum size of the image contained in the layer from the syntax elements of the sequence parameter set referenced by the layer; For the current output layer set, information representing the size of the images contained in the same access unit and associated with the current output layer set is obtained (S612) by summing the maximum sizes of the images contained in the layers decoded for the current output layer set; For the level that is signaled for the current output layer set, the size of the decoded image buffer is determined (S602) based on the information representing the size of the image contained in the same access unit and associated with the current output layer set; as well as The bitstream consistency of the current output layer set is checked (S604) by comparing the determined size of the decoded image buffer with the size of the decoded image buffer required to decode the current output layer set.

2. The method of claim 1, wherein the syntax elements of the sequence parameter set are the maximum width in units of luminance samples for each image of the current output layer set referencing the sequence parameter set and the maximum height in units of luminance samples for each image of the current output layer set referencing the sequence parameter set.

3. The method of claim 1 or 2, wherein checking bitstream consistency for the current output layer set includes: If the size required for decoding the current output layer set in the decoded image buffer is less than or equal to the determined size of the decoded image buffer, it is determined that the multi-layer streams for the current output layer set are consistent.

4. The method of any one of claims 1 to 3, wherein the method comprises: The possibility of real-time decoding of the current output layer set is determined by comparing the luminance sampling rate of the signal notification level with the obtained information representing the image size.

5. The method of claim 4, wherein determining whether real-time decoding of the current output layer set is possible by comparing the luminance sampling rate of the signal-notified level with the obtained information representing the image size includes: If the luminance sampling rate of the level notified by the signal is higher than the obtained information representing the image size multiplied by the frame rate, it is possible to determine the real-time decoding of the current output layer set.

6. The method of any one of claims 1 to 5, wherein the size required for decoding the current output layer set of the decoded image buffer is parsed from the decoded image buffer syntax structure.

7. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform: For each layer of the current output layer set of the multi-layer stream, obtain the maximum size of the image contained in that layer from the syntax elements of the sequence parameter set referenced by that layer; For the current output layer set, information representing the size of the image contained in the same access unit and associated with the current output layer set is obtained by summing the maximum sizes of the images contained in the layers decoded for the current output layer set; For the level that is signaled for the current output layer set, the size of the decoded image buffer is determined based on the information representing the size of the image contained in the same access unit and associated with the current output layer set; as well as The bitstream consistency of the current output layer set is checked by comparing the determined size of the decoded image buffer with the size of the decoded image buffer required to decode the current output layer set.

8. The apparatus of claim 7, wherein the syntax elements of the sequence parameter set are the maximum width in units of luminance samples for each image of the current output layer set referencing the sequence parameter set and the maximum height in units of luminance samples for each image of the current output layer set referencing the sequence parameter set.

9. The apparatus of claim 7 or 8, wherein checking bitstream consistency for the current output layer set comprises: If the size required for decoding the current output layer set in the decoded image buffer is less than or equal to the determined size of the decoded image buffer, it is determined that the multi-layer streams for the current output layer set are consistent.

10. The apparatus of any one of claims 7 to 9, wherein the one or more processors are configured to perform: determining whether real-time decoding of the current output layer set is possible by comparing the luminance sampling rate of the level notified by the signal with information representing the image size obtained.

11. The apparatus of claim 10, wherein determining whether real-time decoding of the current output layer set is possible by comparing the luminance sampling rate of the signal-notified level with the obtained information representing the image size includes: If the luminance sampling rate of the level notified by the signal is higher than the obtained information representing the image size multiplied by the frame rate, it is possible to determine the real-time decoding of the current output layer set.

12. The apparatus of any one of claims 7 to 11, wherein the size required for decoding the current output layer set of the decoded image buffer is parsed from the decoded image buffer syntax structure.

13. A computer program comprising program code instructions for implementing the method as described in any one of claims 1 to 6 when executed by a processor.

Images