Signaling of non-scalable-nested hypothetical reference decoder information

Abstract

Examples of video encoding methods and apparatuses and video decoding methods and apparatuses are described. An example method of video processing includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable nested supplemental enhancement information (SEI) message that includes information about a hypothetical reference decoder (HRD) is applicable to all output layer sets that include a same video layer as the bitstream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of International Patent Application No. PCT / US2021 / 036484, filed June 8, 2021, which claims priority to and benefit of U.S. Provisional Patent Application No. 63 / 036,808, filed June 9, 2020. The entire disclosure of all of the aforementioned patent applications is hereby incorporated by reference. TECHNICAL FIELD

[0003] This patent document relates to image and video coding and decoding. BACKGROUND

[0004] Digital video accounts for the largest bandwidth use on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that bandwidth demands for digital video usage will continue to grow. SUMMARY

[0005] This document discloses techniques that video encoders and decoders can use to perform video encoding or decoding.

[0006] In one example aspect, a method of video processing is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested supplemental enhancement information (SEI) message that includes information about a hypothetical reference decoder (HRD) is applicable to all output layer sets that include the same video layers as the bitstream.

[0007] In another example aspect, another method of video processing is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested hypothetical reference decoder (HRD)-related supplemental enhancement information (SEI) message is omitted in response to a condition that an output layer set that includes the same layer set as the bitstream is not present.

[0008] In another example aspect, another method of video processing is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a value of a layer identifier for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit that includes a non-scalable-nested SEI message is not constrained.

[0009] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a particular payload type value corresponding to subpicture level information is disallowed in a list comprising allowed supplemental enhancement information (SEI) payload type values for non-hypothetical reference decoder (HRD) related supplemental enhancement information (SEI) messages.

[0010] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that non-scalable nested supplemental enhancement information (SEI) messages comprising information unrelated to hypothetical reference decoder (HRD) are applicable to all layers in the bitstream.

[0011] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video according to a rule, wherein the rule specifies that no supplemental enhancement information (SEI) network abstraction layer (NAL) units containing scalable nested SEI messages carrying picture timing information are included due to a same picture timing being used in all output layer sets in the bitstream.

[0012] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more supplemental enhancement information (SEI) network abstraction layer (NAL) units according to a rule, wherein the rule specifies that, responsive to the SEI NAL unit including a non-scalable nested SEI message of a first payload type, the SEI NAL unit is disallowed from including another SEI message of a second payload type.

[0013] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video according to a rule, wherein the bitstream comprises one or more supplemental enhancement information (SEI) network abstraction layer (NAL) units according to the rule, wherein the rule specifies that, responsive to the SEI NAL unit including a scalable nested SEI message of a first payload type, the SEI NAL unit is disallowed from including another SEI message of a second payload type.

[0014] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more supplemental enhancement information, SEI, network abstraction layer (NAL) units according to a rule, wherein the rule specifies that, responsive to a SEI NAL unit comprising a SEI message of a first payload type, the SEI NAL unit is disallowed to comprise another SEI message that is not equal to the first payload type or a second payload type.

[0015] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a rule, wherein the rule specifies a particular decoding order between a subpicture level information (SLI) supplemental enhancement information (SEI) message and a buffering period (BP) SEI message applicable to a particular output layer set, responsive to a condition that the SLI SEI message and the BP SEI message are included in an access unit.

[0016] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video and a bitstream of the video according to a rule, wherein, responsive to a particular value of a second syntax field indicating a maximum number of temporal sub-layers for which an initial coded picture buffer (CPB) removal delay is indicated in a buffering period supplemental enhancement information (SEI) message, the rule specifies omitting a first syntax field indicating a sub-layer representation information for which syntax elements related to the initial CPB removal delay are present.

[0017] In yet another example aspect, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement a method recited above.

[0018] In yet another example aspect, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement a method recited above.

[0019] In yet another example aspect, a computer readable medium having code stored thereon is disclosed. The code embodies one of the methods described herein in the form of processor-executable code.

[0020] These and other features are described throughout this document. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a block diagram illustrating a video coding system in accordance with some embodiments of the disclosed technology.

[0022] Figure 2 is a block diagram of an example hardware platform for video processing.

[0023] Figure 3is a flowchart of an example method of video processing.

[0024] Figure 4 is a block diagram illustrating an example video coding system.

[0025] Figure 5 is a block diagram illustrating an encoder in accordance with some embodiments of the disclosed technology.

[0026] Figure 6 is a block diagram illustrating a decoder in accordance with some embodiments of the disclosed technology.

[0027] Figures 7A to 7E is a flowchart of an example method of video processing based on some embodiments of the disclosed technology.

[0028] Figure 8 is a flowchart of an example method of video processing based on some embodiments of the disclosed technology.

[0029] Figures 9A to 9C is a flowchart of an example method of video processing based on some embodiments of the disclosed technology.

[0030] Figure 10A and Figure 10B is a flowchart of an example method of video processing based on some embodiments of the disclosed technology. DETAILED DESCRIPTION

[0031] The use of section headings in this document is for convenience only and does not limit the applicability of techniques and embodiments disclosed in each section to the section alone. Also, the use of H.266 terminology in some descriptions is for convenience only and is not intended to limit the scope of the disclosed technology. Thus, the technology described herein is applicable to other video codec protocols and designs as well.

[0032] 1. INTRODUCTION

[0033] This document relates to video coding technology. In particular, it is about defining levels and bitstream conformance for a video codec that supports single-layer video coding and multi-layer video coding. It can be applied to any video coding standard or non-standard video codec that supports single-layer video coding and multi-layer video coding, such as the Versatile Video Coding (VVC) that is under development.

[0034] 2. ABBREVIATIONS

[0035] APS adaptation parameter set

[0036] AU access unit

[0037] AUD access unit delimiter

[0038] AVC advanced video coding

[0039] BP buffering period

[0040] CLVS coded layer video sequence

[0041] CPB coded picture buffer

[0042] CRA clean random access

[0043] CTU coding tree unit

[0044] CVS coded video sequence

[0045] DPB decoded picture buffer

[0046] DPS decoding parameter set

[0047] DUI decoding unit information

[0048] EOB end of bitstream

[0049] EOS end of sequence

[0050] GCI general constraint information

[0051] GDR gradual decoding refresh

[0052] HEVC high efficiency video coding

[0053] HDR hypothetical reference decoder

[0054] IDR instantaneous decoding refresh

[0055] JEM joint exploration model

[0056] MCTS motion-constrained tile set

[0057] NAL network abstraction layer

[0058] OLS output layer set

[0059] PH picture header

[0060] PPS picture parameter set

[0061] PT picture timing

[0062] PTL profile, tier, and level

[0063] PU picture unit

[0064] RRP reference picture resampling

[0065] RBSP raw byte sequence payload

[0066] SEI supplemental enhancement information

[0067] SH slice header

[0068] SLI sub-picture level information

[0069] SPS sequence parameter set

[0070] SVC scalable video coding

[0071] VCL video coding layer

[0072] VPS video parameter set

[0073] VTM VVC test model

[0074] VUI video usability information

[0075] VVC versatile video coding

[0076] 3. Preliminary discussion

[0077] Video coding standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. The ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, the video coding standards are based on the hybrid video coding structure where temporal prediction plus transform coding are used. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was founded by VCEG and MPEG jointly in 2015. Since then, many new methods have been adopted by the JVET and put into the reference software named Joint Exploration Model (JEM). The JVET meeting is held every quarter, and the new coding standard targets at 50% bit rate reduction compared to HEVC. The new video coding standard was officially named as Versatile Video Coding (VVC) in the April 2018 JVET meeting, when the first version of VVC test model (VTM) was released. As constant effort is made towards VVC standardization, new coding technologies are adopted into the VVC standard in every JVET meeting. The VVC working draft and test model VTM are then updated after every meeting. The VVC project now targets at the technical completion (FDIS) in the July 2020 meeting.

[0078] 3.1. Parameter sets

[0079] AVC, HEVC and VVC specify parameter sets. The types of parameter sets include SPS, PPS, APS and VPS. All of AVC, HEVC and VVC support SPS and PPS. VPS is introduced from HEVC and included in HEVC and VVC. APS is not included in AVC or HEVC, but included in the latest VVC draft text.

[0080] SPS is designed to carry sequence-level header information, while PPS is designed to carry picture-level header information that changes infrequently. With SPS and PPS, there is no need to repeat the infrequently changing information for each sequence or picture, thus the redundant signaling of this information can be avoided. In addition, the use of SPS and PPS enables the out-of-band transmission of important header information, thus not only avoiding the need for redundant transmission, but also improving the error recovery capability.

[0081] VPS is introduced to carry sequence-level header information that is common to all layers in a multi-layer bitstream.

[0082] APS is introduced to carry such picture-level or slice-level information that requires a considerable number of bits to code, can be shared by multiple pictures, and can have quite a number of different variations in one sequence.

[0083] 3.2. Picture resolution change within a sequence

[0084] In AVC and HEVC, the spatial resolution of a picture cannot change unless a new sequence with a new SPS starts with an IRAP picture. VVC is able to change the picture resolution at a certain location within a sequence without the need to encode an IRAP picture, while an IRAP picture is always intra-coded. This feature is sometimes referred to as reference picture resampling (RPR) because it requires resampling of a reference picture for inter prediction when the reference picture has a different resolution from the current picture being decoded.

[0085] The scaling ratio is limited to be greater than or equal to 1 / 2 (2x downsampling from the reference picture to the current picture) and less than or equal to 8 (8x upsampling). Three sets of resampling filter are specified with different cut-off frequencies to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filter are applied to the scaling ratios of 1 / 2 to 1 / 1.75, 1 / 1.75 to 1 / 1.25 and 1 / 1.25 to 8, respectively. Each set of resampling filter has 16 phases for luma and 32 phases for chroma, which is the same as the case of motion compensation interpolation filter. The usual MC interpolation process is actually a special case of the resampling process with scaling ratios from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived from the picture width and height and the left, right, top and bottom scaling offsets specified for the reference picture and the current picture.

[0086] Aspects of the VVC design that support this feature that differ from HEVC include: i) picture resolution and corresponding conformance window are signaled in PPS instead of SPS, while maximum picture resolution is signaled in SPS. ii) For single-layer bitstreams, each picture store (a time slot in the DPB for storing one decoded picture) occupies the buffer size required to store a decoded picture with the maximum picture resolution.

[0087] 3.3. Scalable video coding (SVC) in general and in VVC

[0088] Scalable video coding (SVC, sometimes just referred to as scalability in video coding) refers to video coding in which a base layer (BL) (sometimes referred to as a reference layer (RL)) and one or more scalable enhancement layers (ELs) are used. In SVC, the base layer can carry video data with a base level of quality. The one or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise ratio (SNR) levels. An enhancement layer can be defined with respect to a previously coded layer. For example, a bottom layer can serve as a BL, while a top layer can serve as an EL. An intermediate layer can serve as an EL or an RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest layer nor the highest layer) can be an EL of the layers below the intermediate layer, such as the base layer or any intermediate enhancement layers, and at the same time serve as an RL of the one or more enhancement layers above the intermediate layer. Similarly, in the multiview or 3D extension of the HEVC standard, there can be multiple views, and information of one view can be utilized to code (e.g., encode or decode) information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancies).

[0089] In SVC, parameters used by an encoder or decoder are grouped into parameter sets based on the coding level (e.g., video level, sequence level, picture level, slice level, etc.) at which they can be used. For example, parameters that can be used by one or more coded video sequences of different layers in a bitstream can be included in a video parameter set (VPS), and parameters used by one or more pictures in a coded video sequence can be included in a sequence parameter set (SPS). Similarly, parameters used by one or more slices in a picture can be included in a picture parameter set (PPS), and other parameters specific to a single slice can be included in a slice header. Similarly, an indication of which parameter set(s) a particular layer uses at a given time can be provided at various coding levels.

[0090] Since VVC supports reference picture resampling (RPR), there is no need for any additional signal processing stage coding tool to design support for bitstreams containing multiple layers (e.g., two layers with SD and HD resolutions in VVC) as upsampling needed for supporting spatial scalability only requires using RPR upsampling filters. However, to support scalability, high level syntax changes (compared to not supporting scalability) are needed. Scalability support is specified in VVC version 1. Unlike scalability support in any earlier video coding standards, including extensions of AVC and HEVC, the design of VVC scalability has been made as friendly as possible to single layer decoder design. The decoding capability for multi-layer bitstreams is specified in a way as if there is only a single layer in the bitstream. For example, decoding capability such as DPB size is specified in a way independent of the number of layers in the bitstream to be decoded. Essentially, a decoder designed for single layer bitstreams does not need much change to be able to decode multi-layer bitstreams. Compared to the design of multi-layer extensions of AVC and HEVC, the HLS aspect is significantly simplified at the expense of some flexibility. For example, an IRAP AU needs to contain pictures of every layer present in the CVS.

[0091] 3.4. SEI messages and general SEI semantics and constraints

[0092] Annex D of VVC specifies the syntax and semantics of SEI message payload for certain SEI messages and specifies the usage of SEI messages and VUI parameters whose syntax and semantics are specified in ITU-T H.265 | ISO / IEC 23000-7.

[0093] SEI messages are helpful for processes related to decoding, display, or other purposes. However, SEI messages are not needed for constructing luma or chroma samples by the decoding process. Consistent decoders are not required to process this information for output order conformance. Some SEI messages are needed for bitstream conformance and output timing decoder conformance checking. Other SEI messages are not needed for bitstream conformance checking.

[0094] In the latest VVC draft text, general SEI is needed.

[0095] The syntax and semantics of scalable nesting SEI messages in the latest VVC draft text are as follows.

[0096] D.2.1 Syntax of general SEI messages

[0097]

[0098]

[0099]

[0100] D.2.2 General SEI payload semantics

[0101] reserved_payload_extension_data shall not occur in bitstreams conforming to this version of this Specification. However, decoders conforming to this version of this Specification shall ignore the presence and value of reserved_payload_extension_data. When present, the length of reserved_payload_extension_data, in bits, is equal to 8*payloadSize - nEarlierBits - nPayloadZeroBits - 1, where nEarlierBits is the number of bits in the sei_payload( ) syntax structure preceding the reserved_payload_extension_data syntax element, and nPayloadZeroBits is the number of payload_bit_equal_to_zero syntax elements at the end of the sei_payload( ) syntax structure.

[0102] payload_bit_equal_to_one shall be equal to 1.

[0103] payload_bit_equal_to_zero shall be equal to 0.

[0104] NOTE 1 - SEI messages with the same payloadType value are conceptually the same SEI message, regardless of whether they are contained in a prefix or suffix SEI NAL unit.

[0105] NOTE 2 - For SEI messages with payloadType values in the range 0 to 47, inclusive, as specified in this Specification, the payloadType values are consistent with similar SEI messages specified in Rec. ITU-T H.264 | ISO / IEC 14496-10.

[0106] The semantics and persistence range of each SEI message is specified in the semantics specification of each particular SEI message.

[0107] NOTE 3 - The persistence information for SEI messages is informatively summarized in Table D.1.

[0108] Table D.1 - Persistence range of SEI messages (informative)

[0109]

[0110] The list VclAssociatedSeiList is set to consist of the payloadType values 3, 19, 45, 129, 132, 137, 144, 145, 147 to 150, inclusive, 153 to 156, inclusive, 168, 203, and 204.

[0111] The list PicUnitRepConSeiList is set to consist of the payloadType values 0, 1, 19, 45, 129, 132, 133, 137, 147 to 150, inclusive, 153 to 156, inclusive, 168, 203, and 204.

[0112] NOTE 4 - VclAssociatedSeiList consists of the payloadType values of SEI messages that, when non-scalable-nested and contained in a SEI NAL unit, infer constraints on the NAL unit header of the SEI NAL unit based on the NAL unit header of the associated VCL NAL unit. PicUnitRepConSeiList consists of the payloadType values of SEI messages that are restricted to 4 repetitions per PU.

[0113] The requirement of bitstream conformance is that the following restrictions apply when SEI messages are contained in a SEI NAL unit:

[0114] - When a SEI NAL unit contains a non-scalable-nested BP SEI message, a non-scalable-nested PT SEI message, or a non-scalable-nested DUI SEI message, the SEI NAL unit shall not contain any other SEI message with a payloadType not equal to 0 (BP), 1 (PT), or 130 (DUI).

[0115] - When a SEI NAL unit contains a scalable-nested BP SEI message, a scalable-nested PT SEI message, or a scalable-nested DUI SEI message, the SEI NAL unit shall not contain any other SEI message with a payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable-nested).

[0116] The following applies to applicable OLSs or non-scalable-nested SEI message layers:

[0117] - For non-scalable-nested SEI messages, when the payloadType is equal to 0 (BP), 1 (PT), or 130 (DUI), the non-scalable-nested SEI message only applies to the 0th OLS.

[0118] - When the payloadType of a non-scalable-nested SEI message is equal to any value in VclAssociatedSeiList, the non-scalable-nested SEI message only applies to layers with nuh layer id equal to the nuh layer id of the SEI NAL unit containing the SEI message.

[0119] It is a conformance requirement of the bitstream that the following restrictions apply to the value of nuh layer id of SEI NAL units:

[0120] - When the payloadType of a non-scalable-nested SEI message is equal to 0 (BP), 1 (PT), or 130 (DUI), the nuh layer id of the SEI NAL unit containing the non-scalable-nested SEI message shall be equal to vps layer id[0].

[0121] - When the payloadType of a non-scalable-nested SEI message is equal to any value in VclAssociatedSeiList, the nuh layer id of the SEI NAL unit containing the non-scalable-nested SEI message shall be equal to the value of nuh layer id of the VCL NAL unit associated with the SEI NAL unit.

[0122] - The nuh layer id of the SEI NAL unit containing a scalable-nested SEI message shall be equal to the lowest value of nuh layer id of all layers to which the scalable-nested SEI message applies (when sn_ols_flag of the scalable-nested SEI message is equal to 0) or the lowest value of nuh layer id of all layers in the OLS to which the scalable-nested SEI message applies (when sn_ols_flag of the scalable-nested SEI message is equal to 1).

[0123] It is a conformance requirement of the bitstream that the following restrictions apply to the repetition of SEI messages:

[0124] - For each payloadType value contained in PicUnitRepConSeiList, there shall be less than or equal to 4 identical sei_payload() syntax structures in one PU.

[0125] - In one DU, there shall be less than or equal to 4 identical sei_payload() syntax structures with payloadType equal to 130.

[0126] The following applies to the order of BP, PT, and DUI SEI messages:

[0127] - When BP SEI messages and PT SEI messages applicable to a particular OP exist within an AU, the BP SEI messages shall precede the PT SEI messages in decoding order.

[0128] - When BP SEI messages and DUI SEI messages applicable to a particular OP exist within an AU, the BP SEI messages shall precede the DUI SEI messages in decoding order.

[0129] - When PT SEI messages and DUI SEI messages applicable to a particular OP exist within an AU, the PT SEI messages shall precede the DUI SEI messages in decoding order.

[0130] 4. Technical problems solved by the disclosed technical solutions

[0131] The existing general SEI payload semantics (including general SEI constraints) have the following problems:

[0132] 1) It should be specified that non-scalable-nested HRD-related SEI messages are applicable to OLSs that include the same set of layers as the entire bitstream (instead of being applicable only to the 0th OLS).

[0133] 2) An entire bitstream can include multiple layers, without an OLS being specified that includes all layers. In this case, there cannot be any non-scalable-nested HRD-related SEI messages, as they would be applicable to OLSs that include the same set of layers as the entire bitstream.

[0134] 3) Currently specified, non-scalable-nested non-HRD-related SEI messages are applicable only to layers with nuh layer id equal to the nuh layer id of the SEI NAL unit. However, to be consistent with non-scalable-nested HRD-related SEI messages, it should be specified that non-scalable-nested non-HRD-related SEI messages are applicable to all layers in the entire bitstream.

[0135] 4) The current specification states that the value of nuh layer id of SEI NAL units containing non-scalable-nested HRD-related SEI messages shall be equal to vps layer id[0] and the value of nuh layer id of SEI NAL units containing non-scalable-nested non-HRD-related SEI messages shall be equal to the nuh layer id of the VCL NAL units associated with the SEI NAL units. However, when non-scalable-nested SEI messages apply to the entire bitstream, these constraints on the value of nuh layer id shall be removed such that the value of nuh layer id of SEI NAL units containing non-scalable-nested SEI messages is unconstrained, the same as for DCI, VPS, AUD and EOS NAL units.

[0136] 5) The list variable VclAssociatedSeiList is currently composed of SEI payloadType values of non-HRD-related SEI messages. However, for SLI SEI messages, the value 203 is also a HRD-related SEI message. Therefore, the payloadType value 203 (SLI) shall be removed from the list.

[0137] 6) The constraint is missing that when general_same_pic_timing_in_all_ols_flag is equal to 1, there shall be no SEI NAL units containing scalable-nested SEI messages with payloadType equal to 1 (PT). This is because when general_same_pic_timing_in_all_ols_flag is equal to 1, there is no need to include PT SEI messages in scalable-nested SEI messages.

[0138] 7) The constraint is missing that when a SEI NAL unit contains non-scalable-nested SEI messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI) or 203 (SLI), the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 0, 1, 130 or 203. Only then, in the sub-bitstream extraction process, it is possible to perform the removal of the four HRD-related SEI messages from the output bitstream by simply removing the SEI NAL units containing one or more of these SEI messages.

[0139] 8) The lack of a constraint that when an SEI NAL unit contains a scalable-nested SEI message with payloadType equal to 0 (BP), 1 (PT), 130 (DUI) or 203 (SLI), the SEI NAL unit shall not contain any other SEI message (scalable-nested) with payloadType not equal to 0, 1, 130, 203 or 133. Only then, it is possible to set the sn_ols_flag value of the scalable-nested SEI message without problems for scalable-nested HRD-related and non-HRD-related SEI messages.

[0140] 9) The lack of a constraint that when an SEI NAL unit contains an SEI message with payloadType equal to 3 (padding payload), the SEI NAL unit shall not contain any other SEI message with payloadType not equal to 3. Only then, it is possible to perform the removal of padding payload SEI messages from the output bitstream by simply removing the SEI NAL units containing one or more padding payload SEI messages during the sub-bitstream extraction process.

[0141] 10) The lack of a constraint that when a SLI SEI message and a BP SEI message applicable to a particular OLS exist within an AU, the SLI SEI message shall precede the BP SEI message in decoding order. This is necessary because SLI SEI messages provide sequence-level information like VPS and SPS, which also precede BP SEI messages when present in the same AU as the BP SEI messages.

[0142] 11) In BP SEI messages, bp_sublayer_initial_cpb_removal_delay_present_flag is signaled even when bp_max_sublayers_minus1 is equal to 0. However, when bp_max_sublayers_minus1 is equal to 0, the value of bp_sublayer_initial_cpb_removal_delay_present_flag is known to be 0.

[0143] 5. Examples of solutions and embodiments

[0144] To solve the above problems and others, the following methods summarized below are disclosed. These solution items should be considered as examples to explain the general concepts and should not be interpreted in a narrow way. Moreover, these items can be applied individually or in any combination.

[0145] 1) To solve the first problem, specify that non-scalable-nested HRD-related SEI messages apply to OLSs with the same set of layers as the entire bitstream (instead of only the 0th OLS).

[0146] a. In one example, HRD-related SEI messages refer to SEI messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0147] 2) To solve the second problem, add a constraint such that when there is no OLS with the same set of layers as the entire bitstream, there shall be no non-scalable-nested HRD-related SEI messages.

[0148] a. In one example, HRD-related SEI messages refer to SEI messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0149] 3) To solve the third problem, specify that non-scalable-nested non-HRD-related SEI messages apply to all layers in the entire bitstream, consistent with non-scalable-nested HRD-related SEI messages.

[0150] a. In one example, non-HRD-related SEI messages refer to SEI messages with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0151] 4) To solve the fourth problem, remove the constraint on the value of nuh_layer_id of SEI NAL units containing non-scalable-nested SEI messages, such that the value of nuh_layer_id of SEI NAL units containing non-scalable-nested SEI messages is not constrained, same as nuh_layer_id of DCI, VPS, AUD, and EOS NAL units.

[0152] 5) To solve the fifth problem, rename VclAssociatedSeiList to NestingForLayersSeiList and remove the payloadType value 203 (SLI) from the list.

[0153] 6) To solve the sixth problem, add a constraint such that when general_same_pic_timing_in_all_ols_flag is equal to 1, there shall be no SEI NAL units containing scalable-nested SEI messages with payloadType equal to 1 (PT).

[0154] a. In addition, in one example, it is specified that when general_same_pic_timing_in_all_ols_flag is equal to 1, the non-scalable-nested PT SEI message applies to all OLSs and subpicture sequences extractable from the bitstream of the OLS.

[0155] b. In one example, instead, a constraint is added so that when general_same_pic_timing_in_all_ols_flag is equal to 1, there shall be no SEI NAL unit containing a scalable-nested SEI message with payloadType equal to 1 (PT) and its sn_subpic_flag equal to 0.

[0156] 7) To address the seventh problem, it is specified that when a SEI NAL unit contains a non-scalable-nested SEI message with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain any other SEI message with payloadType not equal to 0, 1, 130, or 203.

[0157] 8) To address the eighth problem, it is specified that when a SEI NAL unit contains a scalable-nested SEI message with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain any other SEI message with payloadType not equal to 0, 1, 130, 203, or 133 (scalable-nested).

[0158] 9) To address the ninth problem, a constraint is added so that when a SEI NAL unit contains a SEI message with payloadType equal to 3 (padding payload), the SEI NAL unit shall not contain any other SEI message with payloadType not equal to 3.

[0159] a. In addition, in one example, it is specified that the padding data SEI message shall not be scalable-nested, i.e., it shall not be contained in a scalable-nested SEI message.

[0160] b. In one example, instead, a constraint is added so that when a SEI NAL unit contains a SEI message with payloadType equal to 3 (padding payload), the SEI NAL unit shall not contain any other SEI message with payloadType not equal to 3 or 133 (scalable-nested).

[0161] 10) To address the tenth issue, add a constraint such that when SLI SEI messages and BP SEI messages applicable to a particular OLS exist within an AU, the SLI SEI messages shall precede the BP SEI messages in decoding order.

[0162] 11) To address the eleventh issue, specify that when bp_max_sublayers_minus1 is equal to 0, the bp_sublayer_initial_cpb_removal_delay_present_flag is skipped (i.e., not signaled in the BP SEI message).

[0163] a. In addition, in one example, when bp_max_sublayers_minus1 is equal to 0, the value of bp_sublayer_initial_cpb_removal_delay_present_flag is inferred to be equal to 0.

[0164] 6. Embodiments

[0165] Below are some example embodiments of some aspects of the invention summarized above in this section, which can be applied to the VVC specification. The changed text is based on the latest VVC text in JVET-S0152-v5. Most of the relevant parts that have been added or modified are in bold and italic, and some deleted parts are marked with double brackets (e.g., [[a]] means the deleted character “a”).

[0166] 6.1. Embodiment 1

[0167] This embodiment addresses items 1 to 11 and some of their sub-items.

[0168] D.2.2 General SEI payload semantics ...

[0170] The list [[VclAssociated]] NestingForLayersSeiList is set to consist of the payloadType values 3, 19, 45, 129, 132, 137, 144, 145, 147 to 150, inclusive, 153 to 156, inclusive, 168, [

[203] ], and 204.

[0171] The list PicUnitRepConSeiList is set to consist of the payloadType values 0, 1, 19, 45, 129, 132, 133, 137, 147 to 150, inclusive, 153 to 156, inclusive, 168, 203, and 204.

[0172] NOTE 4 - [[VclAssociatedSeiList consists of payloadType values of SEI messages that, when non-scalable-nested and contained in an SEI NAL unit, infer constraints on the NAL unit header of the SEI NAL unit based on the NAL unit header of the associated VCL NAL unit. PicUnitRepConSeiList consists of payloadType values of SEI messages that are restricted to 4 repetitions per PU.

[0173] The following restrictions apply when SEI messages are contained in an SEI NAL unit:

[0174]

[0175] - When an SEI NAL unit contains a non-scalable-nested SEI message with payloadType equal to 0 (BP), 1 (PT), [[or]] 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 0, 1, [[or]] 130, or 203.

[0176] - When an SEI NAL unit contains a scalable-nested SEI message with payloadType equal to 0 (BP), 1 (PT), [[or]] 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 0, 1, 130, 203, or 133 (scalable-nested).

[0177] - When an SEI NAL unit contains an SEI message with payloadType equal to 3 (padding payload), the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 3.

[0178] The following applies to the applicable OLS or non-scalable-nested SEI message layer:

[0179] - For non-scalable-nested SEI messages, when payloadType is equal to 0 (BP), 1 (PT), [[or]] 130 (DUI), or 203 (SLI), the non-scalable-nested SEI message applies to When present, the OLS includes the same set of layers as in the entire bitstream [[only applicable to the 0th OLS]].

[0180] - When the payloadType of a non-scalable-nested SEI message is equal to any value in VclAssociatedSeiList, the non-scalable-nested SEI message applies to all layers in the entire bitstream [[only to layers with nuh layer id equal to the nuh layer id of the SEI NAL unit containing the SEI message]].

[0181] It is a conformance requirement of the bitstream that the following restrictions apply to the value of nuh layer id of SEI NAL units:

[0182] - When the payloadType of a non-scalable-nested SEI message is equal to 0 (BP), 1 (PT), or 130 (DUI), the nuh layer id of the SEI NAL unit containing the non-scalable-nested SEI message shall be equal to vps layer id[0].

[0183] - When the payloadType of a non-scalable-nested SEI message is equal to any value in VclAssociatedSeiList, the nuh layer id of the SEI NAL unit containing the non-scalable-nested SEI message shall be equal to the value of the nuh layer id of the VCL NAL unit associated with the SEI NAL unit.

[0184] - The nuh layer id of the SEI NAL unit containing a scalable-nested SEI message shall be equal to the lowest value of nuh layer id of all layers to which the scalable-nested SEI message applies (when sn_ols_flag of the scalable-nested SEI message is equal to 0) or the lowest value of nuh layer id of all layers in the OLS to which the scalable-nested SEI message applies (when sn_ols_flag of the scalable-nested SEI message is equal to 1).

[0185]

[0186] It is a conformance requirement of the bitstream that the following restrictions apply to the repetition of SEI messages:

[0187] - For each payloadType value contained in PicUnitRepConSeiList, there shall be less than or equal to 4 identical sei_payload( ) syntax structures in one PU.

[0188] ​- In one DU, there shall be less than or equal to 4 identical sei_payload( ) syntax structures with payloadType equal to 130.

[0189] The following applies to the order of SLI, BP, PT, and DUI SEI messages:

[0190]

[0191] - When BP SEI messages and PT SEI messages applicable to a particular [[OP]] are present within an AU, the BP SEI messages shall precede the PT SEI messages in decoding order.

[0192] - When BP SEI messages and DUI SEI messages applicable to a particular [[OP]] are present within an AU, the BP SEI messages shall precede the DUI SEI messages in decoding order.

[0193] - When PT SEI messages and DUI SEI messages applicable to a particular [[OP]] are present within an AU, the PT SEI messages shall precede the DUI SEI messages in decoding order.

[0194] D.3.1 Buffer period SEI message syntax

[0195]

[0196] D.3.2 Buffer period SEI message semantics ...

[0198] bp_sublayer_initial_cpb_removal_delay_present_flag equal to 1 specifies that initial CPB removal delay related syntax elements are present in the sublayer indicated by bp_sublayer_id. bp_sublayer_initial_cpb_removal_delay_present_flag equal to 0 specifies that initial CPB removal delay related syntax elements are not present in the sublayer indicated by bp_sublayer_id. ...

[0200] Figure 1This is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0201] System 1900 may include codec component 1904, which may implement the various codec or encoding methods described in this document. Codec component 1904 may reduce the average bit rate of video from input 1902 to the output of codec component 1904 to produce a codec representation of video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1904 may be stored or transmitted via communication through the connection represented by component 1906. Component 1908 may use the stored or communicated bitstream (or codec) representation of the video received at input 1902 to generate pixel values ​​or displayable video to be sent to display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder, and the decoder will perform a corresponding decoding tool or operation that reverses the result of the codec.

[0202] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be found in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0203] Figure 2is a block diagram of a video processing device 3600. The device 3600 can be used to implement one or more of the methods described herein. The device 3600 can be embodied in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The device 3600 can include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The processor(s) 3602 can be configured to implement one or more methods described in the present document. The memory (memories) 3604 can be used for storing data and code used during operation of the present techniques described herein. The video processing hardware 3606 can be used to implement, in hardware circuitry, some of the techniques described in the present document.

[0204] Figure 4 is a block diagram illustrating an example video coding system 100 that can utilize the techniques of this disclosure.

[0205] As shown in Figure 4 , the video coding system 100 can include a source device 110 and a destination device 120. The source device 110 generates encoded video data, and the source device 110 can be referred to as a video encoding device. The destination device 120 can decode the encoded video data generated by the source device 110, and the destination device 120 can be referred to as a video decoding device.

[0206] The source device 110 can include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0207] The video source 112 can include a source such as a video capture device, an interface to receive video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. The video data can comprise one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream can include a sequence of bits that forms a coded representation of the video data. The bitstream can include coded pictures and associated data. A coded picture is a coded representation of a picture. The associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be transmitted directly to the destination device 120 by the I / O interface 116 through the network 130a. The encoded video data can also be stored onto a storage medium / server 130b for access by the destination device 120.

[0208] The destination device 120 can include an I / O interface 126, a video decoder 124, and a display device 122.

[0209] The I / O interface 126 can include a receiver and / or a modem. The I / O interface 126 can obtain encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 can decode the encoded video data. The display device 122 can display the decoded video data to a user. The display device 122 can be integrated with the destination device 120, or can be external to the destination device 120 which is configured to interface with an external display device.

[0210] The video encoder 114 and the video decoder 124 can operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard, and other current and / or further standards.

[0211] Figure 5 is shown to illustrate that the video encoder 114 can be a component of the source device 110, the destination device 120, or a separate device. Figure 4 A block diagram of an example of a video encoder 200 of the video encoder 114 in the system 100 shown.

[0212] The video encoder 200 can be configured to perform any or all of the techniques of this disclosure. In Figure 5 In an example, the video encoder 200 includes a plurality of functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the techniques described in this disclosure can be performed by a processor configured to perform such techniques.

[0213] The functional components of the video encoder 200 can include a partition unit 201, a prediction unit 202, a prediction unit 1602 can include a mode select unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra-prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy encoding unit 214.

[0214] In other examples, the video encoder 200 can include more, less, or different functional components. In one example, the prediction unit 202 can include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0215] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, can be highly integrated, but are represented separately in Figure 5 for the sake of explanation.

[0216] The partition unit 201 can partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0217] The mode selection unit 203 can select a type of coding mode (intra or inter), for example, based on the error results, and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 can select a combined intra and inter prediction (CIIP) mode in which the prediction is based on both an inter prediction signal and an intra prediction signal. The mode selection unit 203 can also select a resolution of a motion vector (e.g., sub-pixel or integer pixel precision) for the block in the case of inter prediction.

[0218] To perform inter prediction for a current video block, the motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from the buffer 213 to the current video block. The motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 (except for the picture associated with the current video block).

[0219] For example, the motion estimation unit 204 and the motion compensation unit 205 can perform different operations depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0220] In some examples, the motion estimation unit 204 can perform single directional prediction for the current video block, and the motion estimation unit 204 can search for a reference video block for the current video block in a reference picture of List 0 or List 1. The motion estimation unit 204 can then generate a reference index indicating the reference picture of List 0 or List 1 that contains the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 204 can output the reference index, a prediction direction indicator, and the motion vector as the motion information for the current video block. The motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0221] In other examples, the motion estimation unit 204 can perform bi-prediction for the current video block, the motion estimation unit 204 can search for a reference video block for the current video block in a reference picture in list 0 and also search for another reference video block for the current video block in a reference picture in list 1. The motion estimation unit 204 can then generate a reference index that indicates the reference pictures in list 0 and list 1 that contain the reference video blocks and a motion vector that indicates a spatial displacement between the reference video blocks and the current video block. The motion estimation unit 204 can output the reference index and the motion vector for the current video block as motion information for the current video block. The motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information for the current video block.

[0222] In some examples, the motion estimation unit 204 can output a full set of motion information for a decoding process of a decoder.

[0223] In some examples, the motion estimation unit 204 can not output a full set of motion information for a current video. Instead, the motion estimation unit 204 can signal motion information for a current video block with reference to motion information for another video block. For example, the motion estimation unit 204 can determine that the motion information for the current video block is sufficiently similar to the motion information for a neighboring video block.

[0224] In one example, the motion estimation unit 204 can indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0225] In another example, the motion estimation unit 204 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates a difference between a motion vector for the current video block and a motion vector for the indicated video block. The video decoder 300 can use the motion vector for the indicated video block and the motion vector difference to determine the motion vector for the current video block.

[0226] As described above, the video encoder 200 can predictively signal motion vectors. Two examples of predictively signaling techniques that can be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0227] The intra prediction unit 206 can perform intra prediction for the current video block. When the intra prediction unit 206 performs intra prediction for the current video block, the intra prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0228] Residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by the minus sign) the prediction video block(s) for the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of samples in the current video block.

[0229] In other examples, such as in a skip mode, there can be no residual data for the current video block, and residual generation unit 207 can not perform a subtraction operation.

[0230] Transform processing unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0231] After transform processing unit 208 generates the transform coefficient video blocks associated with the current video block, quantization unit 209 can quantize the transform coefficient video blocks associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0232] Inverse quantization unit 210 and inverse transform unit 211 can apply inverse quantization and inverse transform, respectively, to the transform coefficient video blocks to reconstruct the residual video blocks from the transform coefficient video blocks. Reconstruction unit 212 can add the reconstructed residual video blocks to corresponding samples from the prediction video block(s) generated by prediction unit 202 to produce a reconstructed video block associated with the current block for storage in buffer 213.

[0233] After reconstruction unit 212 reconstructs the video block, loop filtering operations can be performed to reduce video block artifacts in the video block.

[0234] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, entropy encoding unit 214 can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.

[0235] Figure 6 is shown to illustrate that the video decoder 300 of the video decoder 114 in the system 100 can be Figure 4 A block diagram of an example of a video decoder 300 of the video decoder 114 in the system 100 shown.

[0236] The video decoder 300 can be configured to perform any or all of the techniques of the present disclosure. In Figure 5 In examples, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0237] In Figure 6 In the example of FIG. 3, video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-prediction unit 303, an inverse quantization unit 304, an inverse transformation unit 305, a reconstruction unit 306, and a buffer 307. Video decoder 300 may, in some examples, perform a decoding process generally reciprocal to the encoding process described with respect to video encoder 200. Figure 5

[0238] Entropy decoding unit 301 can retrieve an encoded bitstream. The encoded bitstream can include entropy encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy encoded video data and motion compensation unit 302 can determine, from the entropy decoded video data, motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. For example, motion compensation unit 302 can determine such information by performing AMVP and merge mode.

[0239] Motion compensation unit 302 can generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter used at sub-pixel precision can be included in syntax elements.

[0240] Motion compensation unit 302 can use an interpolation filter used by video encoder 20 during video block encoding to calculate interpolated values for sub-integer pixels of a reference block. Motion compensation unit 302 can determine the interpolation filter used by video encoder 200 from received syntax information and use the interpolation filter to generate a prediction block.

[0241] Motion compensation unit 302 can use some syntax information to determine the size of blocks used to encode frame(s) and / or slice(s) of a coded video sequence, partition information describing how to partition each macroblock of a picture of the coded video sequence, modes indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-coded block, and other information used to decode the coded video sequence.

[0242] Intra-prediction unit 303 can form a prediction block from spatial neighboring blocks using, for example, intra-prediction modes received in the bitstream. Inverse quantization unit 303 inverse quantizes (i.e., de-quantizes) quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transformation unit 303 applies an inverse transform.

[0243] ​The reconstruction unit 306 can add the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If desired, a deblocking filter can also be applied to the decoded block to remove blocking artifacts. The decoded video block is then stored in the buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also produces decoded video for presentation on a display device.

[0244] The list of solutions describes some embodiments of the disclosed technology.

[0245] A first set of solutions is provided below. The following solutions show example embodiments of the techniques discussed in the previous section (e.g., items 1-3).

[0246] 1. A method of video processing (e.g., Figure 3 The method 600 shown), comprising performing (602) a conversion between a video comprising one or more video layers and a coded representation of the video comprising one or more output layer sets, wherein the coded representation conforms to a format rule related to whether and how one or more syntax elements are related to a non-scalable-nested hypothetical reference decoder (HRD) related supplemental enhancement information (SEI).

[0247] 2. The method of any of solution 1, wherein the format rule specifies that a message related to the non-scalable-nested HRD related SEI applies to output layer sets that include the same layer set as the entire coded representation.

[0248] 3. The method of any of solutions 1-2, wherein the format rule specifies that the one or more syntax elements are omitted in the absence of output layer sets that have the same layer set as the entire coded representation.

[0249] 4. The method of solution 1, wherein the format rule specifies that the one or more syntax elements apply to all layers in the coded representation.

[0250] The following solutions show example embodiments of the techniques discussed in the previous section (e.g., items 4-10).

[0251] 5. A method of video processing, comprising performing a conversion between a video comprising one or more video layers and a coded representation of the video comprising one or more output layer sets, wherein the coded representation conforms to a format rule related to whether and how one or more syntax elements are included in supplemental enhancement information (SEI) network abstraction layer (NAL) units.

[0252] 6. The method according to solution 5, wherein the format rule specifies that the value of the layer identifier is not constrained in case the SEI NAL unit includes non-scalable-nested SEI messages.

[0253] 7. The method according to any of solutions 5-6, wherein the format rule prohibits SEI NAL units including scalable-nested SEI messages containing a particular payload type due to the use of the same picture timing signaling in all output layer sets in the coded representation.

[0254] 8. The method according to any of solutions 5-7, wherein the format rule specifies that SEI NAL units not allowed to include non-scalable-nested SEI messages of a first particular payload type include another SEI message of a second particular type.

[0255] 9. The method according to solution 8, wherein the first particular payload type is equal to 0, 1, 130 or 203.

[0256] 10. The method according to solution 8 or 9, wherein the second particular payload type is equal to 0, 1, 130, 203 or 133.

[0257] 11. The method according to solutions 8-10, wherein the first particular payload type and the second particular payload type are 3.

[0258] 12. The method according to any of solutions 1-11, wherein performing the conversion comprises encoding the video to generate the coded representation.

[0259] 13. The method according to any of solutions 1-11, wherein performing the conversion comprises parsing and decoding the coded representation to generate the video.

[0260] 14. A video decoding apparatus comprising a processor configured to implement the method recited in any one or more of solutions 1-13.

[0261] 15. A video encoding apparatus comprising a processor configured to implement the method recited in any one or more of solutions 1-13.

[0262] 16. A computer program product having computer code stored thereon, the computer code, when executed by a processor, causing the processor to implement the method recited in any one of solutions 1-13.

[0263] 17. A method, apparatus or system described in this document.

[0264] The second set of solutions shows example embodiments of the techniques discussed in the previous section (e.g., items 1-5).

[0265] 1. A method of processing video data (e.g., method 700 as shown in Figure 7A FIG. 7), comprising performing 702 a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested supplemental enhancement information (SEI) message that includes information about a hypothetical reference decoder (HRD) is applicable to all output layer sets that comprise the same video layers as the bitstream.

[0266] 2. The method according to solution 1, wherein the non-scalable-nested SEI message is a SEI message that is not contained in a scalable-nested SEI message.

[0267] 3. The method according to solution 1 or 2, wherein the non-scalable-nested SEI message that includes information about the HRD is a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0268] 4. The method according to solution 1 or 2, wherein the non-scalable-nested SEI message that includes information about the HRD has a payload type equal to 0, 1, 130, or 203.

[0269] 5. A method of processing video data (e.g., method 710 as shown in Figure 7B FIG. 7), comprising performing 712 a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested hypothetical reference decoder (HRD)-related supplemental enhancement information (SEI) message is omitted in response to a condition that an output layer set comprising the same layer set as the bitstream is not present.

[0270] 6. The method according to solution 5, wherein the non-scalable-nested SEI message is a SEI message that is not contained in a scalable-nested SEI message.

[0271] 7. The method according to solution 5 or 6, wherein the non-scalable-nested HRD-related SEI message is a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0272] 8. The method of solution 5 or 6, wherein the non-scalable-nested HRD-related SEI message has a payload type equal to 0, 1, 130, or 203.

[0273] 9. A method of processing video data (e.g., method 720 as shown in Figure 7C FIG. 7B), comprising performing 722 a conversion between video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a value of a layer identifier for a supplemental enhancement information (SEI) network abstraction layer (NAL) unit including a non-scalable-nested SEI message is unconstrained.

[0274] 10. The method of solution 9, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message.

[0275] 11. The method of solution 9, wherein the non-scalable-nested SEI message is a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0276] 12. The method of solution 9, wherein the non-scalable-nested SEI message has a payload type equal to 0, 1, 130, or 203.

[0277] 13. A method of processing video data (e.g., method 730 as shown in Figure 7D FIG. 7C), comprising performing 732 a conversion between video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a particular payload type value corresponding to subpicture level information is not allowed in a list of allowed supplemental enhancement information (SEI) payload type values for non-hypothetical reference decoder (HRD)-related SEI messages.

[0278] 14. The method of solution 13, wherein the particular payload type value is 203.

[0279] 15. The method of solution 13, wherein the allowed SEI payload type values include a filler payload, film grain characteristics, frame packing arrangement, parameter set inclusion indication, master display color volume, content light level information, dependent rap indication, alternative transfer characteristics, ambient viewing environment, content color volume, equirectangular projection, generalized cubemap projection, sphere rotation, region packing, omnidirectional viewport, frame field information, and sample aspect ratio information.

[0280] 16. The method of solution 13, wherein the allowed SEI payload type values include 3, 19, 45, 129, 137, 144, 145, 147-150, 153-156, 168, and 204.

[0281] 17. A method of processing video data, comprising performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested supplemental enhancement information (SEI) message that includes information independent of a hypothetical reference decoder (HRD) is applicable to all layers in the bitstream.

[0282] 18. The method of solution 17, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message.

[0283] 19. The method of solution 17 or 18, wherein the non-scalable-nested SEI message that includes information independent of an HRD has a payload type not equal to 0, 1, 130, or 203.

[0284] 20. The method of solution 17 or 18, wherein the non-scalable-nested SEI message that includes information independent of an HRD does not correspond to a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0285] 21. The method of any of solutions 1 to 20, wherein the conversion comprises encoding the video as the bitstream.

[0286] 22. The method of any of solutions 1 to 20, wherein the conversion comprises decoding the video from the bitstream.

[0287] 23. The method of any of solutions 1 to 20, wherein the conversion comprises generating the bitstream from the video, and the method further comprises storing the bitstream in a non-transitory computer-readable recording medium.

[0288] 24. A video processing apparatus comprising a processor configured to implement a method recited in one or more of solutions 1 to 23.

[0289] 25. A method of storing a bitstream of a video, comprising a method recited in any of solutions 1 to 23, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0290] 26. A computer-readable medium storing program code that, when executed, causes a processor to implement a method recited in any one or more of solutions 1 to 23.

[0291] 27. A computer-readable medium storing a bitstream generated according to any of the methods above.

[0292] 28. A video processing apparatus for storing a bitstream, wherein the video processing apparatus is configured to implement a method recited in any one or more of solutions 1 to 23.

[0293] The third set of solutions shows example embodiments of the techniques discussed in the previous section (e.g., item 6).

[0294] 1. A method of processing video data (e.g., method 800 as shown in Figure 8 FIG. 8), comprising performing 802 a conversion between a video and a bitstream of the video comprising one or more output layer sets according to a rule, wherein the rule specifies that no supplemental enhancement information (SEI) messages containing a scalable-nested SEI message carrying picture timing information are included in SEI network abstraction layer (NAL) units due to a same picture timing being used in all output layer sets in the bitstream.

[0295] 2. The method of solution 1, wherein the scalable-nested SEI message is an SEI message contained in a scalable-nested SEI message.

[0296] 3. The method of solution 1 or 2, wherein the scalable-nested SEI message carrying picture timing information corresponds to a picture timing (PT) SEI message.

[0297] 4. The method of solution 1, wherein the scalable-nested SEI message carrying picture timing information has a payload type equal to 1.

[0298] 5. The method of any of solutions 1 to 4, wherein the rule further specifies that non-scalable-nested SEI messages carrying picture timing information are applicable to all output layer sets and subpicture sequences allowed to be extracted from the bitstream.

[0299] 6. The method of solution 5, wherein the non-scalable-nested SEI message is an SEI message not contained in a scalable-nested SEI message.

[0300] 7. The method according to any of solutions 1 to 4, wherein the rule further specifies that, in case the syntax field has a value that specifies that the scalable-nested SEI message that applies to a particular output layer set or layer applies to all sub-pictures of the specified output layer set or layer, SEI NAL units that include scalable-nested SEI messages carrying picture timing information are not included.

[0301] 8. The method according to any of solutions 1 to 7, wherein the conversion comprises encoding the video into a bitstream.

[0302] 9. The method according to any of solutions 1 to 7, wherein the conversion comprises decoding the video from a bitstream.

[0303] 10. The method according to any of solutions 1 to 7, wherein the conversion comprises generating a bitstream from the video, and the method further comprises: storing the bitstream in a non-transitory computer-readable recording medium.

[0304] 11. A video processing apparatus comprising a processor configured to implement a method recited in any one or more of solutions 1 to 10.

[0305] 12. A method of storing a bitstream of a video, comprising a method recited in any one of solutions 1 to 10, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0306] 13. A computer-readable medium storing program code that, when executed, causes a processor to implement a method recited in any one or more of solutions 1 to 10.

[0307] 14. A computer-readable medium storing a bitstream generated according to any of the methods described above.

[0308] 15. A video processing apparatus for storing a bitstream, wherein the video processing apparatus is configured to implement a method recited in any one or more of solutions 1 to 10.

[0309] The fourth set of solutions shows example embodiments of the techniques discussed in the previous section (e.g., items 7-9).

[0310] 1. A method of processing video data (e.g., a method 900 as shown in Figure 9A FIG. 1), comprising performing 902 a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more supplemental enhancement information (SEI) network abstraction layer (NAL) units according to a rule, wherein the rule specifies that, in response to the SEI NAL unit including a non-scalable-nested SEI message of a first payload type, the SEI NAL unit is not allowed to include another SEI message of a second payload type.

[0311] 2. The method of solution 1, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message.

[0312] 3. The method of solution 1 or 2, wherein the non-scalable-nested SEI message of the first payload type corresponds to a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0313] 4. The method of solution 1 or 2, wherein the first payload type is equal to 0, 1, 130, or 203.

[0314] 5. The method of any of solutions 1 to 4, wherein the non-scalable-nested SEI message of the second payload type does not correspond to any of a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, and a subpicture level information (SLI) SEI message.

[0315] 6. The method of any of solutions 1 to 4, wherein the second payload type is not equal to any of 0, 1, 130, and 203.

[0316] 7. A method of processing video data (e.g., a method 910 as shown in Figure 9B FIG. 10), comprising performing 912 a conversion between a video and a bitstream of the video according to a rule, wherein the bitstream comprises one or more supplemental enhancement information, SEI, network abstraction layer (NAL) units according to the rule, wherein the rule specifies that, responsive to a SEI NAL unit including a scalable-nested SEI message of a first payload type, the SEI NAL unit is disallowed to include another SEI message of a second payload type.

[0317] 8. The method of solution 7, wherein the scalable-nested SEI message is an SEI message that is contained in a scalable-nested SEI message.

[0318] 9. The method of solution 7 or 8, wherein the scalable-nested SEI message of the first payload type corresponds to a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message.

[0319] 10. The method of solution 7 or 8, wherein the first payload type is equal to 0, 1, 130, or 203.

[0320] 11. The method of any of solutions 7 to 10, wherein the scalable-nested SEI messages of the second payload type do not correspond to any of a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, a subpicture level information (SLI) SEI message, and a scalable nesting information SEI message.

[0321] 12. The method of any of solutions 7 to 10, wherein the second payload type is not equal to any of 0, 1, 130, 203, and 133.

[0322] 13. A method of processing video data (e.g., a method 920 as shown in Figure 9C FIG. 21), comprising performing 922 a conversion between a video and a bitstream of the video, wherein the bitstream comprises one or more supplemental enhancement information, SEI, network abstraction layer (NAL) units according to a rule, wherein the rule specifies that, responsive to an SEI NAL unit including an SEI message of a first payload type, the SEI NAL unit is not allowed to include another SEI message that is not equal to the first payload type or a second payload type.

[0323] 14. The method of solution 13, wherein the SEI message of the first payload type corresponds to a filler payload SEI message.

[0324] 15. The method of solution 13 or 14, wherein the first payload type is equal to 3.

[0325] 16. The method of any of solutions 13 to 15, wherein the rule further specifies that the SEI message of the first payload type is not allowed to be included in a scalable nesting SEI message.

[0326] 17. The method of any of solutions 13 to 16, wherein the rule further specifies that the SEI message of the second payload type is a scalable nesting SEI message.

[0327] 18. The method of any of solutions 13 to 16, wherein the second payload type is equal to 133.

[0328] 19. The method of any of solutions 1 to 18, wherein the conversion comprises encoding the video as the bitstream.

[0329] 20. The method of any of solutions 1 to 18, wherein the conversion comprises decoding the video from the bitstream.

[0330] 21. The method of any one of solutions 1 to 18, wherein the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0331] 22. A video processing apparatus, comprising a processor configured to implement the method described in any one or more of solutions 1 to 21.

[0332] 23. A method for storing a bitstream of video, comprising the method of any one of solutions 1 to 21, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0333] 24. A computer-readable medium storing program code that, when executed, causes a processor to perform the method described in any one or more of solutions 1 to 21.

[0334] 25. A computer-readable medium storing a bit stream generated according to any of the methods described above.

[0335] 26. A video processing apparatus for storing bit streams, wherein the video processing apparatus is configured to implement the method described in any one or more of solutions 1 to 21.

[0336] The fifth set of solutions illustrates example embodiments of the techniques discussed in the previous section (e.g., items 10 and 11).

[0337] 1. A method for processing video data (e.g., such as...) Figure 10A The method 1000 shown includes: performing a conversion between a video and a video bitstream, wherein the bitstream includes one or more output layer sets according to rules, the output layer sets including one or more video layers, wherein in response to conditions in which Sub-Picture Level Information (SLI) Supplemental Enhancement Information (SEI) messages and Buffer Period (BP) SEI messages are included in an access unit, the rules specify a specific decoding order between SLI SEI messages and BP SEI messages applicable to a particular output layer set.

[0338] 2. According to the method of Solution 1, the specific decoding order is that the SLI SEI message precedes the BP SEI message.

[0339] 3. A method for processing video data (e.g., such as...) Figure 10BThe illustrated method 1010) includes performing 1012 a conversion between a video and a bitstream of the video according to a rule, wherein, in response to a particular value of a second syntax field indicating a maximum number of temporal sub-layers for which an initial coded picture buffer (CPB) removal delay is indicated in a buffering period supplemental enhancement information (SEI) message, the rule specifies omitting a first syntax field indicating sub-layer representation information for which syntax elements related to the initial CPB removal delay are present.

[0340] 4. The method of solution 3, wherein the particular value is 0.

[0341] 5. The method of solution 3 or 4, wherein the second syntax field having the particular value specifies that the maximum number of temporal sub-layers is 1.

[0342] 6. The method of any of solutions 3 to 5, wherein the first syntax field equal to another particular value specifies that the syntax elements related to the initial CPB removal delay are present for sub-layer representations in a range of 0 to the second syntax field, inclusive.

[0343] 7. The method of solution 6, wherein the another particular value is 1.

[0344] 8. The method of any of solutions 3 to 7, wherein the syntax field equal to another particular value specifies that the syntax elements related to the initial CPB removal delay are present for sub-layer representations corresponding to the particular value of the second syntax field.

[0345] 9. The method of solution 8, wherein the another particular value is 0.

[0346] 10. The method of any of solutions 3 to 9, wherein the rule further specifies inferring a value of the first syntax field equal to 0 in response to the particular value of the second syntax field.

[0347] 11. The method of any of solutions 1 to 10, wherein the conversion includes encoding the video into the bitstream.

[0348] 12. The method of any of solutions 1 to 10, wherein the conversion includes decoding the video from the bitstream.

[0349] 13. The method of any of solutions 1 to 10, wherein the conversion includes generating the bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0350] 14. A video processing apparatus comprising a processor configured to implement a method recited in one or more of solutions 1 to 13.

[0351] 15. A method of storing a bitstream of a video, comprising the method of any of solutions 1 to 13, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0352] 16. A computer-readable medium storing program code that, when executed, causes a processor to implement the method of any one or more of solutions 1 to 13.

[0353] 17. A computer-readable medium storing a bitstream generated according to any of the methods.

[0354] 18. A video processing apparatus for storing a bitstream, wherein the video processing apparatus is configured to implement the method of any one or more of solutions 1 to 13.

[0355] The disclosures and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in combinations of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine- readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated for encoding information to transmit to a suitable receiver apparatus.

[0356] A computer program, which can also be referred to or referred to as a program, software, a software application, an app, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program is not necessarily corresponding to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to execute on one computer or on a plurality of computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0357] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, and that apparatus can also be implemented as special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0358] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0359] While this patent document contains many details, these should not be construed as limiting the scope of any subject matter or potentially patentable content in any way, but as a description of features that can be particular to certain embodiments of the particular technology. Certain features described in the context of separate embodiments can also be implemented in combination with each other. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination. Moreover, although the features can be described as acting in certain combinations and even initially claimed as such, in some cases the features from a claimed combination can be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0360] Similarly, while operations are described in a particular order in the drawings, this should not be understood as requiring such order or sequence of operations, or that all illustrated operations be performed, to achieve desirable results. Further, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0361] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method of processing video data, comprising: performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets according to a format rule, the output layer sets including one or more video layers, wherein the format rule specifies that a particular payload type value corresponding to subpicture level information is disallowed in a list including allowed supplemental enhancement information, SEI, payload type values.

2. The method of claim 1, wherein, a supplemental enhancement information network abstraction layer, NAL, unit includes non-scalable-nested SEI messages, wherein the non-scalable-nested SEI messages are SEI messages that are not contained in scalable-nested SEI messages.

3. The method of claim 1, wherein, the format rule further specifies that non-scalable-nested SEI messages including information about a hypothetical reference decoder, HRD, apply to all output layer sets as the bitstream includes the video layers.

4. The method of claim 3, wherein, the format rule further specifies that a non-scalable-nested hypothetical reference decoder, HRD-related supplemental enhancement information, SEI, message is omitted in response to a condition that an output layer set as the bitstream includes the video layers is not present.

5. The method of claim 4, wherein, the non-scalable-nested SEI messages are buffering period, BP, SEI messages, picture timing, PT, SEI messages, decoding unit information, DUI, SEI messages, or subpicture level information, SLI, SEI messages.

6. The method of claim 5, wherein, the non-scalable-nested SEI messages have a payload type equal to 0, 1, 130, or 203.

7. The method of claim 1, wherein, the particular payload type value is 203.

8. The method of claim 7, wherein, the allowed SEI payload type values include a filler payload, a film grain characteristics, a frame packing arrangement, a parameter set inclusion indication, a master display color volume, a content light level information, a dependent knock indication, an alternative transfer characteristics, an ambient viewing environment, a content color volume, an equirectangular projection, a generalized cubemap projection, a sphere rotation, a region packing, an omnidirectional viewport, a frame field information, and a sample aspect ratio information.

9. The method of claim 8, wherein, the allowed SEI payload type values include 3, 19, 45, 129, 137, 144, 145, 147-150, 153-156, 168, and 204.

10. The method of claim 8, wherein, when non-scalable-nested, the SEI messages corresponding to the allowed SEI payload type values in the list have constraints on the NAL unit header of the SEI NAL unit inferred based on the NAL unit header of an associated video coding layer, VCL, NAL unit.

11. The method of claim 1, wherein, the conversion includes encoding the video into the bitstream.

12. The method of claim 1, wherein, the conversion includes decoding the video from the bitstream.

13. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, the instructions, when executed by the processor, cause the processor to: perform a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets according to a format rule, the output layer sets including one or more video layers, wherein the format rule specifies that a particular payload type value corresponding to subpicture level information is disallowed in a list including allowed supplemental enhancement information, SEI, payload type values.

14. The apparatus of claim 13, wherein, The supplemental enhancement information network abstraction layer (NAL) unit includes a non-scalable-nested SEI message, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message.

15. The apparatus of claim 13, wherein, The format rule further specifies that a non-scalable-nested SEI message including information about a hypothetical reference decoder (HRD) applies to all output layer sets as the bitstream includes a video layer, wherein the format rule further specifies that a non-scalable-nested hypothetical reference decoder (HRD) related supplemental enhancement information (SEI) message is omitted in response to an absence of a condition that the bitstream includes an output layer set of the video layer, wherein the non-scalable-nested SEI message is a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message, wherein the non-scalable-nested SEI message has a payload type equal to 0, 1, 130, or 203, wherein the particular payload type value is 203, wherein the allowed SEI payload type values include a filler payload, a film grain characteristics, a frame packing arrangement, a parameter set inclusion indication, a master display color volume, a content light level information, a dependent knock indication, an alternative transfer characteristics, an ambient viewing environment, a content color volume, an equirectangular projection, a generalized cubemap projection, a sphere rotation, a region packing, an omnidirectional viewport, a frame field information, and a sample aspect ratio information.

16. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets including one or more video layers in accordance with a format rule, wherein the format rule specifies that a particular payload type value corresponding to subpicture level information is disallowed in a list of allowed supplemental enhancement information (SEI) payload type values.

17. The non-transitory computer-readable storage medium of claim 16, wherein, The supplemental enhancement information network abstraction layer (NAL) unit includes a non-scalable-nested SEI message, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message. The format rule further specifies that a non-scalable-nested SEI message including information about a hypothetical reference decoder (HRD) applies to all output layer sets as the bitstream includes a video layer, wherein the format rule further specifies that a non-scalable-nested hypothetical reference decoder (HRD) related supplemental enhancement information (SEI) message is omitted in response to an absence of a condition that the bitstream includes an output layer set of the video layer, wherein the non-scalable-nested SEI message is a buffering period (BP) SEI message, a picture timing (PT) SEI message, a decoding unit information (DUI) SEI message, or a subpicture level information (SLI) SEI message, wherein the non-scalable-nested SEI message has a payload type equal to 0, 1, 130, or 203, wherein the particular payload type value is 203, wherein the allowed SEI payload type values include a filler payload, a film grain characteristics, a frame packing arrangement, a parameter set inclusion indication, a master display color volume, a content light level information, a dependent knock indication, an alternative transfer characteristics, an ambient viewing environment, a content color volume, an equirectangular projection, a generalized cubemap projection, a sphere rotation, a region packing, an omnidirectional viewport, a frame field information, and a sample aspect ratio information. The allowed SEI payload type values include a filler payload, film grain characteristics, frame packing arrangement, parameter set inclusion indication, master display color volume, content light level information, dependent clip info, alternative transfer characteristics, surround view, content color volume, equirectangular projection, generalized cubemap projection, sphere rotation, region enclosure, omnidirectional viewport, frame field information, and sample aspect ratio information.

18. A non-transitory computer-readable recording medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing apparatus, wherein, The method includes: generating the bitstream of the video, wherein the bitstream includes one or more output layer sets according to a format rule, the output layer set including one or more video layers, wherein the format rule specifies that a particular payload type value corresponding to sub-picture level information is not allowed in a list of allowed supplemental enhancement information, SEI, payload type values. 19.The non-transitory computer-readable recording medium of claim 18, wherein, A supplemental enhancement information network abstraction layer, NAL, unit includes a non-scalable-nested SEI message, wherein the non-scalable-nested SEI message is an SEI message that is not contained in a scalable-nested SEI message; wherein the format rule further specifies that the non-scalable-nested SEI message including information about a hypothetical reference decoder, HRD, applies to all output layer sets as the bitstream includes a video layer, wherein the format rule further specifies that in response to a condition that an output layer set as the bitstream includes the video layer is not present, a non-scalable-nested hypothetical reference decoder, HRD, related supplemental enhancement information, SEI, message is omitted, wherein the non-scalable-nested SEI message is a buffering period, BP, SEI message, a picture timing, PT, SEI message, a decoding unit information, DUI, SEI message, or a sub-picture level information, SLI, SEI message, wherein the non-scalable-nested SEI message has a payload type equal to 0, 1, 130, or 203, wherein the particular payload type value is 203, wherein the allowed SEI payload type values include a filler payload, film grain characteristics, frame packing arrangement, parameter set inclusion indication, master display color volume, content light level information, dependent clip info, alternative transfer characteristics, surround view, content color volume, equirectangular projection, generalized cubemap projection, sphere rotation, region enclosure, omnidirectional viewport, frame field information, and sample aspect ratio information.

20. A method for storing a bitstream of a video, comprising: generating the bitstream of the video, storing the bitstream in a non-transitory computer-readable recording medium, wherein the bitstream includes one or more output layer sets according to a format rule, the output layer set including one or more video layers, wherein the format rule specifies that a particular payload type value corresponding to sub-picture level information is not allowed in a list of allowed supplemental enhancement information, SEI, payload type values.

Images