Technique for generating applicable output sub-bitstream

Abstract

To provide a video encoding method and device including a plurality of layers, and a video decoding method and device.SOLUTION: A video processing method performs conversion between a video having a plurality of layers and a bitstream of the video according to a rule defining a maximum allowable value of a value of an identifier of a temporal layer of a sub-bitstream extracted from the bitstream.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS Under applicable patent laws and / or regulations under the Paris Convention, this application is intended to timely claim priority to and the benefit of U.S. Provisional Patent Application No. 63 / 029308, filed May 22, 2020. For all purposes under law, the entire disclosure of said application is incorporated by reference as part of the disclosure herein.

[0002] This patent specification relates to image and video encoding and decoding. [Background technology]

[0003] Digital video accounts for the largest bandwidth usage on the Internet and other digital communications networks, and the bandwidth demands for digital video use are expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases. Summary of the Invention

[0004] This specification discloses techniques that may be used by video encoders and decoders to perform video encoding or decoding.

[0005] In one exemplary aspect, a video processing method is disclosed, the method including performing a conversion between a video including multiple layers and a video bitstream according to rules, the rules specifying maximum allowable values ​​of temporal layer identifiers of sub-bitstreams extracted from the bitstream.

[0006] In another exemplary aspect, a video processing method is disclosed, the method includes performing a conversion between video and a video bitstream according to rules, the rules defining Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream, and providing for deriving the sub-bitstream based on whether a list of values ​​of NAL unit header identifiers in an Output Layer Set (OLS) having an OLS index does not include all values ​​of NAL unit header identifiers in all Video Coding Layer (VCL) NAL units in the bitstream input to the sub-bitstream extraction process.

[0007] In another exemplary aspect, a video processing method is disclosed, the method including performing a conversion between video and a video bitstream according to rules, the rules defining Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream, the rules specifying, corresponding to a payload type of a first Supplemental Enhancement Information (SEI) message included in the SEI NAL unit, that the SEI NAL unit is not allowed to include an SEI message having a particular payload type.

[0008] In another exemplary aspect, a video processing method is disclosed, the method including performing a conversion between video and a video bitstream according to a format rule, the format rule specifying that the bitstream includes a flag specifying whether one or more non-scalable nested Supplemental Enhancement Information (SEI) messages having one or more particular payload types are applied to all output layer sets referenced by a coding layer.

[0009] In another exemplary aspect, a video processing method is disclosed that includes performing a conversion between video and a video bitstream according to rules, where the rules specify a sub-bitstream extraction process that does not remove Network Abstraction Layer (NAL) units of a particular type and extracts an output sub-bitstream from the bitstream having a particular NAL unit header identifier value, where the particular type includes Access Unit Delimiter (AUD) NAL units.

[0010] In another exemplary aspect, a video processing method is disclosed, the method including converting between a video including one or more layers and a video bitstream according to a rule, the rule specifying removing a Network Abstraction Layer (NAL) unit including a scalable nesting Supplementary Enhancement Information (SEI) message that applies to a layer that is not included in a target Output Layer Set (OLS) in the process of sub-bitstream extraction.

[0011] In another exemplary aspect, a video processing method is disclosed. The method includes performing conversion between a video including one or more layers and a video bitstream according to a rule, in which in a process of sub-bitstream extraction, a non-scalable nested SEI message is generated by extracting a scalable nested SEI message from a scalable nesting SEI message based on a first flag indicating whether the SEI message applies to a specific Output Layer Set (OLS) and a second flag indicating whether the SEI message applies to all subpictures or only to a specific subpicture.

[0012] In another exemplary aspect, a video processing method is disclosed, the method includes performing conversion between a video including one or more layers and a video bitstream according to a rule, the rule specifying that in a process of sub-bitstream extraction, a non-scalable nested SEI message is generated by extracting a plurality of scalable nested SEI messages from a first Supplemental Enhancement Information (SEI) Network Abstraction Layer (NAL) unit in a picture unit.

[0013] In another exemplary aspect, a video processing method is disclosed, the method including performing a conversion between a video including one or more layers and a video bitstream according to rules, the rules specifying a sub-bitstream extraction process to generate an output bitstream, the rules specifying handling of one or more Supplemental Enhancement Information (SEI) Network Abstraction Layer (NAL) units during the sub-bitstream extraction process.

[0014] In yet another exemplary aspect, a video encoder apparatus is disclosed, the video encoder comprising a processor configured to implement the above-described method.

[0015] In yet another exemplary aspect, a video decoder apparatus is disclosed, the video decoder comprising a processor configured to implement the above-described method.

[0016] In yet another exemplary aspect, a computer readable medium is disclosed having code stored thereon, the code performing one of the methods described herein in the form of processor executable code.

[0017] These and other features are described throughout the specification. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating a video coding system in accordance with some embodiments of the disclosed techniques. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware platform used for video processing. [Diagram 3] FIG. 3 is a flowchart showing an example of a video processing method. [Figure 4] FIG. 4 is a block diagram illustrating an example video coding system. [Diagram 5] FIG. 5 is a block diagram illustrating an encoder in accordance with some embodiments of the disclosed techniques. [Figure 6] FIG. 6 is a block diagram illustrating a decoder in accordance with some embodiments of the disclosed techniques. [Figure 7A] FIG. 7A shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 7B] FIG. 7B shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 7C] FIG. 7C shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 7D] FIG. 7D shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 8] FIG. 8 illustrates a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 9] FIG. 9 illustrates a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 10A] FIG. 10A shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 10B]FIG. 10B shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. [Figure 10C] FIG. 10C shows a flowchart of an exemplary method of video processing according to some implementations of the disclosed technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Section headings are used herein for ease of understanding and are not intended to limit the applicability of the technology and embodiments described in each section to that section alone. Additionally, the term H.266 is used in certain descriptions for ease of understanding only and is not intended to limit the scope of the disclosed technology. Thus, the technology described herein is applicable to other video codec protocols and designs.

[0020] 1. Introduction This specification relates to video coding technology. In particular, some improvements are made to a general sub-bitstream extraction process, signaling of picture-level HRD parameters, and including SEI messages in SEI NAL units. The ideas may be applied individually or in various combinations to any video coding standard or non-standard video codec that supports multi-layer video coding, such as the currently developed Versatile Video Coding (VVC).

[0021] 2. Abbreviation APS Adaptation Parameter Set AU Access Unit AUD Access Unit Delimiter AVC Advanced Video Coding CLVS Coded Layer Video Sequence CPB Coded Picture Buffer CRA Clean Random Access CTU Coding Tree Unit CVS Coded Video Sequence DCI Decoding Capability Information DPB Decoded Picture Buffer EOB End Of Bitstream EOS End Of Sequence GDR Gradual Decoding Refresh HEVC High Efficiency Video Coding HRD Hypothetical Reference Decoder IDR Instantaneous Decoding Refresh ILP Inter-Layer Prediction ILRP Inter-Layer Reference Picture JEM Joint Exploration Model LTRP Long-Term Reference Picture MCTS Motion-Constrained Tile Sets NAL Network Abstraction Layer OLS Output Layer Set PH Picture Header PPS Picture Parameter Set PTL Profile,Tier and Level PU Picture Unit RAP Random Access Point RBSP Raw Byte Sequence Payload SEI Supplemental Enhancement Information SPS Sequence Parameter Set STRP Short-Term Reference Picture SVC Scalable Video Coding VCL Video Coding Layer VPS Video Parameter Set VTM VVC Test Model VUI Video Usability Information VVC Versatile Video Coding

[0022] 3. Initial Negotiation Video coding standards have evolved primarily through the development of well-known ITU-T and ISO / IEC standards. ITU-T created H.261 and H.263, ISO / IEC created MPEG-1 and MPEG-4 Visual, and the two organizations jointly created H.262 / MPEG-2Video, H.264 / MPEG-4AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on hybrid video coding structures in which temporal prediction and transform coding are utilized. In order to explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, many new methods have been adopted by JVET and incorporated into reference software called the Joint Exploration Model (JEM). JVET meets quarterly, and the new coding standard aims for a 50% bitrate reduction compared to HEVC. At the April 2018 JVET meeting, the new video coding standard was officially named VVC (Versatile Video Coding) and the first version of VTM (VCC Test Model) was released at that time. As efforts to contribute to the standardization of VVC continue, new coding techniques are adopted for the VVC standard at every JVET meeting. After every meeting, the VVC working draft and test model VTM are updated. The VVC project is currently aiming for technical completion (FDIS) at the July 2020 meeting.

[0023] 3.1 Changing picture resolution in a sequence In AVC and HEVC, the spatial resolution of a picture cannot be changed unless a new sequence using the new SPS starts with an IRAP picture. VVC allows changing the resolution of a picture within a sequence at a position without coding an IRAP picture, which is always intra-coded. This feature is called Reference Picture Resampling (RPR), because it is necessary to resample the reference picture used for inter prediction if the reference picture has a different resolution than the current picture being decoded.

[0024] The scaling ratio is limited to be greater than or equal to 1 / 2 (2x downsampling from the reference picture to the current picture) and less than or equal to 8 (8x upsampling). To address various scaling ratios between the reference picture and the current picture, we define three sets of resampling filters with different frequency cutoffs. The three sets of resampling filters apply to scaling ratios ranging from 1 / 2 to 1 / 1.75, 1 / 1.75 to 1 / 1.25, and 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luma and 32 phases for chroma, similar to the case of the motion compensation interpolation filters. In practice, the normal MC interpolation process is a special case of the resampling process with scaling ratios ranging from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height, as well as the left, right, top and bottom scaling offsets defined for the reference picture and the current picture.

[0025] Other aspects of the VVC design to support this feature that differ from HEVC include: i) signaling picture resolution and corresponding compatibility window in the PPS instead of the SPS, signaling maximum picture resolution in the SPS; ii) in the case of a single-layer bitstream, each picture store (one slot in the DPB for storing one decoded picture) occupies the buffer size required to store a decoded picture with the maximum picture resolution.

[0026] 3.2. SVC (Scalable Video Coding) in General and VVC Scalable Video Coding (SVC, sometimes also referred to as scalability in video coding) refers to video coding in which a Base Layer (BL) (sometimes referred to as a Reference Layer (RL)) and one or more scalable Enhancement Layers (EL) are used. In SVC, a base layer can carry video data of a basic level of quality. One or more enhancement layers can carry additional video data, for example, to support higher spatial, temporal, and / or signal-to-noise (SNR) levels. An enhancement layer may be defined relative to a previous, coded layer. For example, a lower layer may function as a BL and an upper layer may function as an EL. An intermediate layer may function as either an EL or an RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the top layer) may be an EL for a layer below the intermediate layer, e.g., a base layer or any intervening enhancement layer, and at the same time serve as an RL for one or more enhancement layers above the intermediate layer. Similarly, in multiview or 3D extensions of the HEVC standard, there may be multiple views and information of one view may be utilized to code (e.g., encode or decode) information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0027] In SVC, parameters used in an encoder or decoder are grouped into parameter sets based on the coding level at which they can be utilized (e.g., video level, sequence level, picture level, slice level, etc.). For example, parameters that can be utilized by one or more coded video sequences of different layers in the bitstream may be included in a Video Parameter Set (VPS), and parameters utilized by one or more pictures in a coded video sequence may be included in a Sequence Parameter Set (SPS). Similarly, parameters utilized by one or more slices of a picture may be included in a Picture Parameter Set (PPS), and other parameters specific to a single slice may be included in a slice header. Similarly, an indication of which parameter set a particular layer is using at a given time may be provided at various coding levels.

[0028] Thanks to the support of Reference Picture Resampling (RPR) in VVC, the upsampling required for spatial scalability support only requires the use of an RPR upsampling filter, so that a bitstream containing multiple layers, e.g., two layers of SD and HD resolution in VVC, can be designed to support without requiring additional signal processing level coding tools. Nevertheless, a high level of syntax change (compared to not supporting scalability) is required for scalability support. Scalability support is specified in VVC version 1. Unlike scalability support in any previous video coding standard, including the extensions of AVC and HEVC, the design of scalability in VVC has been made as suitable as possible for single-layer decoder design. The decoding capabilities of a multi-layer bitstream are specified as if there was only a single layer in the bitstream. For example, the decoding capabilities, such as DPB size, are specified to be independent of the number of layers of the bitstream being decoded. Essentially, a decoder designed for a single-layer bitstream does not require many modifications to be able to decode a multi-layer bitstream. Compared to the multi-layer extension designs of AVC and HEVC, aspects of HLS have been significantly simplified at the expense of some flexibility: for example, an IRAP AU is required to contain a picture in each of the layers present in the CVS.

[0029] 3.3 Parameter Set AVC, HEVC, and VVC specify parameter sets. Parameter set types are SPS, PPS, APS, VPS, etc. SPS and PPS are supported by all of AVC, HEVC, and VVC. VPS was introduced by HEVC and is included in both HEVC and VVC. APS was not included in AVC or HEVC, but is included in the recent VVC draft text.

[0030] The SPS was designed to carry sequence-level header information, and the PPS was designed to carry picture-level header information that does not change frequently. With the SPS and PPS, redundant signaling of frequently changing information can be avoided because there is no need to repeat this information for every sequence or picture. Furthermore, using the SPS and PPS enables out-of-band transmission of important header information, thereby improving error resilience as well as avoiding the need for redundant transmission.

[0031] The VPS was introduced to carry sequence-level header information that is common to all layers of a multi-layer bitstream.

[0032] APS was introduced to carry such picture-level or slice-level information that requires significant bits to code, is shared by multiple pictures, and can exist in many different variations in a sequence.

[0033] 3.4. General Sub-Bitstream Extraction Process Section C.6 of the latest VVC text specifies the general sub-bitstream extraction process as follows:

[0034] C.6 Sub-bitstream extraction process The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the target's highest TemporalId value tIdTarget. The output of this process is the sub-bitstream outBitstream. The requirement for bitstream conformance with respect to an input bitstream is that any output sub-bitstream that satisfies all of the following conditions is a conforming bitstream: The output sub-bitstream is the output of the processing specified in this section on inputs the bitstream, targetOlsIdx equal to the index of the list of OLSs specified in the VPS, and tIdTarget, which may be any value in the range from 0 to 6. - The output sub-bitstream contains at least one VCL NAL unit with nuh_layer_id equal to each nuh_layer_id value in LayerIdInOls[targetolsIdx]. The output sub-bitstream contains at least one VCL NAL unit with TemporalId equal to tIdTarget. NOTE - A conforming bitstream contains one or more coded slice NAL units with TemporalId equal to 0, but is not required to contain a coded slice NAL unit with nuh_layer_id equal to 0.

[0035] The output sub-bitstream outBitstream is derived as follows. 1. The bitstream outBitstream is set to be identical to the bitstream inBitstream. 2. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream. 3. Remove all NAL units from outBitstream whose nal_unit_type is not equal to any of VPS_NUT, DCI_NUT, and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx]. 4. Remove from outBitstream all VCL NAL units for which all of the following conditions are true and non-VCL NAL units with associated PayloadType not equal to 0, 1, or 130 and nal_unit_type equal to PH_NUT, FD_NUT, SUFFIX_SEI_NUT, and PREFIX_SEI_NUT: - nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type is equal to GDR_NUT and the associated ph_recovery_poc_cnt is not equal to 0. - nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j] for values ​​of j in the range from 0 to NumLayersInOls[targetOlsIdx]-1. -TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0036] 5. Remove all SEI NAL units from outBitstream, including scalable nesting SEI messages with sn_ols_flag equal to 1 and NestingOlsIdx[i] equal to targetOlsIdx, for values ​​of i not in the range from 0 to sn_num_olss_minus1. 6. If LayerIdInOls[targetOlsIdx] does not contain all values ​​of nuh_layer_id for all NAL units in the bitstream, the following applies: a. Remove all SEI NAL units, including non-scalable nested SEI messages, with payloadType equal to 0 (BP) or 130 (DUI) from outBitstream. b. If general_same_pic_timing_in_all_ols_flag is equal to 0, remove all SEI NAL units, including non-scalable nested SEI messages with payloadType equal to 1(PT), from outBitstream. c. If outBitstream contains a scalable nesting SEI message with sn_ols_flag equal to 1 and contains a SEI NAL unit applicable to outBitstream (NestingOlsIdx[i] equal to targetOlsIdx), the following applies: -If general_same_pic_timing_in_all_ols_flag is equal to 0, extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (BP), 1 (PT) or 130 (DUI) from the scalable nesting SEI message and include those SEI messages in the outBitstream. - Otherwise (if general_same_pic_timing_in_all_ols_flag is equal to 1), extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI) from the scalable nesting SEI message and include those SEI messages in the outBitstream.

[0037] 4. Technical problem solved by the disclosed technical solution The existing design of the general sub-bitstream extraction process and other related parts in the latest VVC text (JVET-R2001-vA / v10) has the following problems: 1) In the conditions where an output sub-bitstream is required to be a conforming bitstream, the value of tIdTarget is said to be in the range of 0 to 6. However, in many bitstreams the highest TemporalId value is less than 6, and this value is specified by the syntax element vps_max_sublayers_minus1. 2) AUD (Access Unit Delimiter) NAL units, if present, can have any nuh_layer_id value, however, step 3 of the sub-bitstream extraction process removes AUD NAL units whose nuh_layer_id value is not included in the list LayerIdInOls[targetOlsIdx]. 3) Some SEI NAL units contain a scalable nesting SEI message with sn_ols_flag equal to 0, but the applicable layers indicated in the scalable nesting SEI message do not include any layers in the target OLS, i.e., the nuh_layer_id values ​​of the applicable layers are not included in the list LayerIdInOls[targetOlsIdx]. These SEI NAL units should also be removed. 4) The condition in step 6, i.e., "if LayerIdInOls[targetOlsIdx] does not contain all values ​​of nuh_layer_id of all NAL units in the bitstream", has the following two problems: a. If a DCI, VPS, AUD, or EOB NAL unit is present and has a nuh_layer_id that is not equal to any of the nuh_layer_id values ​​of the VCL NAL unit, then this condition does not work. b. The term "bitstream" is unclear in some contexts as there are two bitstreams: inBitstream and outBitstream.

[0038] 5) Step 6.c extracts scalable nested SEI messages from scalable nesting SEI messages that have sn_ols_flag equal to 1 and have sn_subpic_flag equal to 1 to generate non-scalable nested SEI messages, where such scalable nested SEI messages apply only to a particular subpicture and should not be extracted. 6) In step 6.c, if multiple scalable nested SEI messages are extracted from one SEI NAL unit seiNalUnitA into non-scalable nested SEI messages, they SHOULD be contained in one SEI NAL unit seiNalUnitB, and SEI NAL unit seiNalUnitB SHOULD be contained in the same PU that contains SEI NAL unit seiNalUnitA, although this is not specified. 7) Step 6.c removes from outBitstream all SEI NAL units that were included as non-scalable nested SEI messages when any SEI messages were extracted, although this is unspecified. 8) There is no constraint that if an SEI NAL unit contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL unit does not contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting). This makes the removal of SEI messages in step 4 more inclusive than simply removing SEI NAL units. 9) The flag general_same_pic_timing_in_all_ols_flag only specifies whether the non-scalable nested PT SEI message applies to all OLS. However, the information carried in the DUI SEI message is for the same purpose as the PT SEI message.

[0039] 5. List of technical solutions and embodiments In order to solve the above problems, the following methods are disclosed. These items are examples for explaining the general concept, and should not be interpreted in a narrow sense. Furthermore, these items may be applied individually or may be combined in any manner. 1) To solve issue 1, the condition under which an output sub-bitstream is required to be a conforming bitstream is specified such that the value of tIdTarget is in the range from 0 to vps_max_sublayers_minus1. a. Alternatively, the condition for an output sub-bitstream to be a conforming bitstream is specified as the value of tIdTarget being in the range of 0 to vps_max_sublayers_minus1 if the input bitstream has multiple layers, and in the range of 0 to sps_max_sublayers_minus1 if the input bitstream has only one layer. 2) To solve issue 2, a general sub-bitstream extraction process is defined to treat AUD NAL units the same as NAL units with nal_unit_type equal to VPS_NUT, DCI_NUT or EOB_NUT, i.e., do not remove AUD NAL units from the output bitstream outBitstream according to the nuh_layer_id value.

[0040] 3) To solve problem 3, the general sub-bitstream extraction process removes from the output bitstream outBitstream SEI NAL units that contain scalable nesting SEI messages with sn_ols_flag equal to 0, while the applicable layers indicated in the scalable nesting SEI messages are specified not to include any layers in the target OLS. a. In one example, it is specified to remove from outBitstream all SEI NAL units, including scalable nesting SEI messages, that have sn_ols_flag equal to 0 and that have no values ​​in the list LayerIdInOls[targetOlsIdx] in the list nestingLayerId[i] for i in the range from 0 to nestingNumLayers-1. 4) To solve issue 4, the condition “List LayerIdInOls[targetOlsIdx] does not contain all values ​​of nuh_layer_id of all NAL units in the Bitstream” is changed to “List LayerIdInOls[targetOlsIdx] does not contain all values ​​of nuh_layer_id of all VCL NAL units in the Bitstream inBitstream”.

[0041] 5) To solve problem 5, a general sub-bitstream extraction process is defined to extract only scalable nested SEI messages from scalable nesting SEI messages having sn_ols_flag equal to 1 and sn_subpic_flag equal to 0 to generate non-scalable nested SEI messages. 6) To solve problem 6, a general sub-bitstream extraction process is specified such that when multiple scalable nested SEI messages are extracted from one SEI NAL unit seiNalUnitA to become non-scalable nested SEI messages, they are included in one SEI NAL unit seiNalUnitB in the output bitstream outBitstream, and SEI NAL unit seiNalUnitB is included in the PU that contained SEI NAL unit seiNalUnitA.

[0042] 7) To solve problem 7, a general sub-bitstream extraction process is defined in which, from the output bitstream outBitstream, some SEI messages are extracted and all SEI NAL units that are included as non-scalable nested SEI messages are removed. a. Alternatively, if the scalable nested SEI message in such SEI NAL unit applies only to the target OLS (i.e., the targetOlsIdx-th OLS specified by the VPS), remove that SEI NAL unit from outBitstream. b. Alternatively, if there is no OLS other than the target OLS that contains layers that are all included in the list LayerIdInOls[targetOlsIdx] within the OLS to which the scalable nested SEI message in such SEI NAL unit applies, remove that SEI NAL unit from outBitstream. 8) To solve issue 8, a constraint is added such that if an SEI NAL unit contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL unit must not contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting).

[0043] 9) To solve issue 9, the flag general_same_pic_timing_in_all_ols_flag specifies whether non-scalable nested PT and DUI SEI messages apply to all OLS. a. Alternatively, the flag general_same_pic_timing_in_all_ols_flag specifies whether non-scalable nested BP, PT, and DUI SEI messages apply to all OLS. i. In one example, the flag general_same_pic_timing_in_all_ols_flag is renamed to flag general_same_pic_level_hrd_info_in_all_ols_flag to specify whether non-scalable nested BP, PT, DUI SEI messages apply to all OLS. b. Alternatively, a new flag, for example general_same_dui_in_all_ols_flag, is added to specify whether the non-scalable nested DUI SEI message applies to all OLS. c. Alternatively, a new flag, for example general_same_bp_in_all_ols_flag, may be added to specify whether the non-scalable nested BP SEI message applies to all OLSs.

[0044] 6. Embodiment Below are some exemplary embodiments for some of the inventive aspects summarized in Chapter 5 above, applicable to the VVC specification. The modified text is based on the latest VVC text in JVET-R2001-vA / v10. The most relevant parts that have already been added or modified are highlighted in bold and italic letters, and some of the parts that have been deleted are marked with double brackets (e.g., [[a]] indicates the deletion of the letter "a"). There may be some other changes that are not highlighted, as they are editable in nature.

[0045] 6.1. First embodiment This embodiment is for items 1, 2, 3, 3.a., 4, 5, 6, 7.b, and 8.

[0046] [ka]

[0047] [ka]

[0048] [ka]

[0049] [ka]

[0050] [ka]

[0051] [ka]

[0052] FIG. 1 is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of the system 1900. The system 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format, e.g., 8 or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. The input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0053] The system 1900 may include a coding component 1904 that may implement various coding or encoding methods described herein. The coding component 1904 may reduce the average bit rate of the video from the input 1902 to the output of the coding component 1904, generating a coded representation of the video. Thus, this coding technique may be referred to as a video compression or video transcoding technique. The output of the coding component 1904 may be stored or transmitted via a connected communication, as represented by component 1906. The bitstream (or coded) representation of the video received at the input 1902, stored or communicated, may be used by component 1908 to generate pixel values ​​or displayable video that are transmitted to a display interface 1910. The process of generating a user-viewable video from the bitstream representation may be referred to as video unfolding. Additionally, although certain video processing operations are referred to as "coding" operations or tools, it will be understood that the coding tools or operations are performed by an encoder and corresponding decoding tools or operations that reverse the results of the coding, performed by a decoder.

[0054] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described herein may be implemented in various electronic devices such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0055] FIG. 2 is a block diagram of a video processing device 3600. The device 3600 may be used to implement one or more methods described herein. The device 3600 may be implemented in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. The device 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The one or more processors 3602 may be configured to implement one or more methods described herein. The one or more memories 3604 may be used to store data and codes used to implement the methods and techniques described herein. The video processing hardware 3606 may be used to implement the techniques described herein in a hardware circuit.

[0056] FIG. 4 is a block diagram illustrating an example video coding system 100 that can utilize techniques of this disclosure.

[0057] 4, video coding system 100 may include source device 110 and destination device 120. Source device 110 generates encoded video data, which may also be referred to as a video encoding device. Destination device 120 may decode the encoded video data generated by source device 110, and may also be referred to as a video decoding device.

[0058] Source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0059] The video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 and generates a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include a coded picture and associated data. A coded picture is a coded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be transmitted directly to the destination device 120 via the I / O interface 116 over the network 130a. The coded video data may be stored on a recording medium / server 130b for access by the destination device 120.

[0060] Destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0061] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 configured to interface with an external display device.

[0062] Video encoder 114 and video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVM) standard, and other current and / or future standards.

[0063] FIG. 5 is a block diagram of an example video encoder 200, which may be the video encoder 114 in the system 100 shown in FIG.

[0064] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 5, video encoder 200 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0065] The functional components of the video encoder 200 may include a splitting unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214, and the prediction unit 202 includes a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206.

[0066] In other examples, video encoder 200 may include more, fewer, or different functional components. In one example, prediction unit 202 may include an Intra Block Copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0067] Furthermore, some components, such as the motion estimator 204 and the motion compensator 205, may be highly integrated, but are depicted separately in the example of FIG. 5 for purposes of illustration.

[0068] The divider 201 may divide a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support a variety of video block sizes.

[0069] The mode selection unit 203 may select one of intra or inter coding modes, for example based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 207 for generating residual block data, and to the reconstruction unit 212 for reconstructing a coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a Combination of Intra and Inter Prediction (CIIP) mode that performs prediction based on an inter prediction signal and an intra prediction signal. The mode selection unit 203 may select a resolution of a motion vector (e.g., sub-pixel or integer pixel precision) for the block in the case of inter prediction.

[0070] To perform inter prediction on the current video block, motion estimation unit 204 may generate motion information for the current video block by comparing the current video block to one or more reference frames from buffer 213. Motion compensation unit 205 may determine a prediction video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0071] Motion estimation unit 204 and motion compensation unit 205 may, for example, perform different operations on the current video block based on whether the current video block is an I-slice, a P-slice, or a B-slice.

[0072] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search for a reference picture in list 0 or list 1 for a reference video block for the current video block. The motion estimation unit 204 then generates a reference index indicating a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. The motion compensation unit 205 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0073] In another example, motion estimation unit 204 may bidirectionally predict the current video block, and motion estimation unit 204 may search for reference pictures from list 0 for a reference video block for the current video block, and may also search for reference pictures in list 1 for another reference video block for the current video block. Motion estimation unit 204 may then generate reference indices indicating reference pictures in lists 0 and 1 that contain the reference video blocks, and motion vectors indicating spatial displacements between the reference video blocks and the current video block. Motion estimation unit 204 may output the reference index and the motion vector for the current video block as motion information for the current video block. Motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0074] In some examples, the motion estimation unit 204 may output a full set of motion information for the decoding process of the decoder.

[0075] In some examples, the motion estimator 204 may not output a full set of motion information for the current picture. Rather, the motion estimator 204 may signal the motion information of the current video block by reference to motion information of another video block. For example, the motion estimator 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0076] In one example, motion estimator 204 may indicate, in a syntax structure associated with the current video block, a value that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0077] In another example, motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates a difference between the motion vector of the current video block and the motion vector of the designated video block. Video decoder 300 may use the motion vector of the designated video block and the motion vector difference to determine the motion vector of the current video block.

[0078] As mentioned above, video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0079] The intra predictor 206 may perform intra prediction on the current video block. When the intra predictor 206 intra predicts the current video block, the intra predictor 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0080] Residual generator 207 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0081] In other examples, for example in skip mode, there may be no residual data for the current video block, and residual generator 207 may not perform the subtraction operation.

[0082] Transform processor 208 may generate one or more transform coefficient image blocks for the current video block by applying one or more transforms to a residual video block associated with the current video block.

[0083] After the transform processing unit 208 generates a transform coefficient image block associated with the current image block, the quantization unit 209 may quantize the transform coefficient image block associated with the current image block based on one or more quantization parameter (QP) values ​​associated with the current image block.

[0084] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient image block to reconstruct a residual video block from the transform coefficient image block. The reconstruction unit 212 may add the reconstructed residual video block to samples corresponding to one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0085] After reconstructor 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0086] The entropy encoder 214 may receive data from other functional components of the video encoder 200. If the entropy encoder 214 receives data, the entropy encoder 214 may perform one or more entropy encoding operations to generate entropy coded data, and output a bitstream that includes the entropy coded data.

[0087] FIG. 6 is a block diagram of an example video decoder 300, which may be the video decoder 114 in the system 100 shown in FIG.

[0088] The video decoder 300 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 5, the video decoder 300 includes multiple functional components. The techniques described in this disclosure may be shared among various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0089] 6, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306, as well as a buffer 307. The video decoder 300 may, in some examples, perform a decoding path that is generally the reverse of the encoding path described with respect to the video encoder 200 (FIG. 5).

[0090] The entropy decoding unit 301 retrieves an encoded bitstream. The encoded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 301 decodes the entropy coded video data, and from the entropy decoded video data, the motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 may determine such information by, for example, performing AMVP and merge mode.

[0091] The motion compensation unit 302 may generate motion compensated blocks, possibly performing the interpolation based on an interpolation filter. The syntax element may include an identifier for the interpolation filter to be used with sub-pixel accuracy.

[0092] The motion compensation unit 302 may calculate interpolated values ​​for the sub-integer pixels of the reference block using an interpolation filter as used by video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filter used by video encoder 200 based on the received syntax information and use the interpolation filter to generate the predictive block.

[0093] The motion compensation unit 302 may use the syntax information to determine the size of blocks used to code frames and / or slices of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is divided, a mode indicating how each division is coded, one or more reference frames (and reference frame lists) for each inter-coded block, and any of the other information to decode the coded video sequence.

[0094] The intra prediction unit 303 may form a prediction block from spatially neighboring blocks, for example, using an intra prediction mode received in the bitstream. The inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0095] A reconstruction unit 306 may sum the residual block with a corresponding prediction block generated by the motion compensation unit 202 or intra prediction unit 303 to form a decoded block. If desired, a deblocking filter may be applied to filter the decoded block to remove block artifacts. The decoded video block is stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction to generate decoded video for display on a display device.

[0096] The list of solutions describes some examples of the disclosed technology.

[0097] A first set of solutions is provided below. The following solutions represent exemplary embodiments of the techniques discussed in the previous chapter (eg, items 1-9).

[0098] 1. A video processing method (e.g., method 600 of FIG. 3), comprising converting (602) between video including one or more video layers including one or more video pictures and a coded representation of the video, the coded representation conforming to format rules relating to extraction of a sub-bitstream from the coded representation.

[0099] 2. The method according to solution 1, further comprising extracting a sub-bitstream from the coded representation according to a format rule.

[0100] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 1).

[0101] 3. A method according to any one of solutions 1 to 2, wherein during extraction of the sub-bitstream, the target id used for the extraction is allowed to range between 0 and the value of a syntax field indicated in the video parameter set for the coded representation.

[0102] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 2).

[0103] 4. The method according to any of solutions 1 to 3, extracting sub-bitstreams according to layer ids without removing the AUD NAL (Access Unit Delimiter Network Abstraction Layer) from the output bitstream.

[0104] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 3).

[0105] 5. A method according to any one of solutions 1 to 4, extracting a sub-bitstream by selectively removing network abstraction layer units that contain scalable nesting supplemental enhancement information messages that are not applicable to the output layer being extracted.

[0106] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 5).

[0107] 6. A method according to any one of Solutions 1 to 5, wherein the sub-bitstream is extracted by constraining the extraction to generate a non-scalable nested Supplemental Enhancement Information (SEI) from the scalable nesting SEI message with a flag for the output layer set being set and a flag for the sub-picture being disabled.

[0108] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 6).

[0109] 7. A method according to any one of solutions 1 to 6, extracting sub-bitstreams according to rules governing the extraction of multiple scalable nested SEI (Supplemental Enhancement Information) messages from a single SEI network abstraction layer unit.

[0110] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 7).

[0111] 8. A method according to any one of Solutions 1 to 7, wherein some SEI messages are extracted and the sub-bitstream is extracted according to a rule that removes from the coded representation all Supplemental Enhancement Information Network Abstraction Layer SEI NAL units that are included as non-scalable nested SEI messages.

[0112] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 8).

[0113] 9. The method of any one of Solutions 1 to 8, wherein the format rule specifies that if an SEI NAL (Supplemental Enhancement Information Network Abstraction Layer) unit contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL unit is not allowed to contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting).

[0114] 10. The method according to any one of solutions 1 to 9, wherein performing the conversion includes encoding the video into a coded representation.

[0115] 11. A method according to any of solutions 1 to 9, wherein performing the conversion includes parsing and decoding the coded representation to generate the image.

[0116] 12. A video decoding device comprising a processor configured to perform the methods described in one or more of solutions 1 to 11.

[0117] 13. A video encoding device comprising a processor configured to implement the methods according to one or more of solutions 1 to 11.

[0118] 14. A computer program product having computer code stored thereon, the code causing the processor to implement a method according to any of solutions 1 to 11 when executed by the processor.

[0119] 15. Any method, apparatus or system described herein.

[0120] The second set of solutions illustrates exemplary implementations of the techniques discussed in the previous chapter (eg, items 1, 4, 8, and 9).

[0121] 1. A video processing method (e.g., method 700 shown in FIG. 7A), comprising performing (702) a conversion between a video including multiple layers and a video bitstream in accordance with a rule, the rule specifying a maximum allowable value of a temporal layer identifier value of a sub-bitstream extracted from the bitstream.

[0122] 2. The method according to solution 1, wherein the rule is responsive to the number of layers in the bitstream.

[0123] 3. The method according to solution 1 or 2, wherein if the number of layers is greater than 1, the rules specify that the maximum allowed value of the temporal layer identifier is in the range from 0 to a value based on the first syntax element in the video parameter set referenced by the bitstream.

[0124] 4. The method according to solution 1 or 2, wherein if the number of layers is equal to 1, the rule specifies that the maximum allowed value of the temporal layer identifier is in the range from 0 to a value based on the second syntax element in the sequence parameter set referenced by the bitstream.

[0125] 5. The method according to solution 3, wherein the first syntax element specifies the maximum number of temporal sub-layers allowed to be present in a layer specified by the video parameter set minus one.

[0126] 6. The method according to solution 4, wherein the second syntax element specifies the maximum number of temporal sublayers allowed to be present in a layer specified by a value of the sequence parameter set minus one.

[0127] 7. The method according to Solution 3, wherein the first syntax element in the video parameter set is vps_max_sublayers_minus1.

[0128] 8. The method according to Solution 4, wherein the second syntax element in the sequence parameter set is sps_max_sublayers_minus1.

[0129] 9. The method according to solution 3 or 4, wherein the value is equal to the value of the first syntax element or the second syntax element.

[0130] 10. A video processing method (e.g., method 710 shown in FIG. 7B), comprising performing (712) a conversion between video and a video bitstream in accordance with rules, the rules defining Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream, the rules providing for deriving the sub-bitstream based on whether a list of NAL unit header identifier values ​​in an Output Layer Set (OLS) having a target OLS index does not include all values ​​of NAL unit header identifiers in all Video Coding Layer (VCL) NAL units in the bitstream input to the sub-bitstream extraction process.

[0131] 11. The method according to solution 10, wherein the rule specifies that if the list of NAL unit header identifier values ​​in the output layer set does not include all NAL unit header identifier values ​​of all VCL NAL units in the bitstream, then all SEI (Supplemental Enhancement Information) NAL units, including non-scalable nested SEI messages with payload type equal to 0 or 130, are removed from the sub-bitstream.

[0132] 12. A video processing method (e.g., method 720 shown in FIG. 7C ), comprising performing (722) a conversion between video and a video bitstream in accordance with rules, the rules defining Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream, and the rules specifying that in response to a payload type of a first Supplemental Enhancement Information (SEI) message included in the SEI Network Abstraction Layer (SEI) NAL unit, not allowing the SEI NAL unit to include an SEI message having a particular payload type.

[0133] 13. The method according to solution 12, wherein the payload type of the first SEI message is 0, 1, or 130, and the rules specify that an SEI NAL unit is not allowed to contain an SEI message having a specific payload type not equal to 0, 1, 130, or 133.

[0134] 14. A video processing method (e.g., method 730 shown in FIG. 7D), comprising performing (732) a conversion between video and a video bitstream in accordance with a format rule, the format rule specifying that the bitstream includes a flag specifying whether one or more non-scalable nested Supplemental Enhancement Information (SEI) messages having one or more particular payload types are applied to all output layer sets referenced by the coding layer.

[0135] 15. The method according to solution 14, wherein one or more specific payload types are equal to one and 130.

[0136] 16. The method according to solution 14, wherein the one or more non-scalable SEI messages are a Picture Timing (PT) SEI message and a Decoding Unit Information (DUI) SEI message.

[0137] 17. The method according to solution 14, wherein the one or more specific payload types are equal to 0, 1, and 130.

[0138] 18. The method according to solution 14, wherein the one or more non-scalable SEI messages are a Buffering Period (BP) SEI message and a Decoding Unit Information (DUI) SEI message.

[0139] 19.1 or more specific payload types are equal to 130.

[0140] 20. The method according to solution 14, wherein the one or more non-scalable SEI messages are Decoding Unit Information (DUI) SEI messages.

[0141] 21.1 The method according to solution 14, wherein the specific payload type or types are equal to 0.

[0142] 22. The method according to solution 1, wherein one or more of the non-scalable SEI messages are BP (Buffering Period) SEI messages.

[0143] 23. The method according to any of solutions 1 to 22, wherein the conversion includes encoding the video into a bitstream.

[0144] 24. The method according to any of solutions 1 to 22, wherein the conversion includes decoding the video from a bitstream.

[0145] 25. A method according to any of Solutions 1 to 22, wherein the conversion includes generating a bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0146] 26. A video processing device comprising a processor configured to implement the methods according to one or more of solutions 1 to 25.

[0147] 27. A method for storing a video bitstream comprising the method of any one of solutions 1 to 25 and further comprising storing the bitstream in a non-transitory computer-readable recording medium.

[0148] 28. A computer-readable medium storing program code that, when executed, causes a processor to implement a method according to one or more of solutions 1 to 25.

[0149] 29. A computer readable medium storing a bitstream generated according to any of the methods described above.

[0150] 30. A video processing device for storing a bitstream representation, configured to implement a method according to any one of solutions 1 to 25.

[0151] The third set of solutions illustrates exemplary implementations of the techniques discussed in the previous section (eg, item 2).

[0152] 1. A video processing method (e.g., method 800 shown in FIG. 8), comprising performing (802) a conversion between video and a video bitstream in accordance with rules, the rules defining a sub-bitstream extraction process that does not remove Network Abstraction Layer (NAL) units of a particular type and extracts an output sub-bitstream from the bitstream having a particular NAL unit header identifier value, the particular type including Access Unit Delimiter (AUD) NAL units.

[0153] 2. The method according to solution 1, wherein the specific type includes a Video Parameter Set (VPS) NAL unit.

[0154] 3. The method according to solution 1 or 2, wherein the specific type includes a decoding capability information NAL unit.

[0155] 4. The method according to any one of solutions 1 to 3, wherein the specific type includes a bitstream end NAL unit.

[0156] 5. The method according to any one of Solutions 1 to 4, wherein the specific type includes a supplemental enhancement information NAL unit including a non-scalable nested SEI message having a payload type equal to 0, 1, 130, or 203.

[0157] 6. A method according to any one of solutions 1 to 6, wherein the particular NAL unit header identifier value comprises a layer identifier value included in a list of layer values ​​for the output sub-bitstream.

[0158] 7. The method according to solution 1, wherein the rule specifies that NAL units of a particular type shall not be removed from the output bitstream, subject to exceptions, regardless of the NAL unit header identifier value associated with the NAL units of that type.

[0159] 8. The method according to any one of solutions 1 to 7, wherein the conversion comprises encoding the video into a bitstream.

[0160] 9. The method according to any of solutions 1 to 7, wherein the conversion includes decoding the video from a bitstream.

[0161] 10. A method according to any of Solutions 1 to 7, wherein the conversion includes generating a bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0162] 11. A video processing device comprising a processing device configured to implement the method according to any one or more of solutions 1 to 10.

[0163] 12. A method for storing a video bitstream, comprising the method according to any one of solutions 1 to 10 and further comprising storing the bitstream in a non-transitory computer-readable recording medium.

[0164] 13. A computer-readable medium storing program code that, when executed, causes a processor to implement a method according to any one or more of solutions 1 to 10.

[0165] 14. A computer readable medium storing a bitstream generated according to any of the methods described above.

[0166] 15. A video processing device for storing a bitstream representation, configured to implement a method according to any one or more of solutions 1 to 10.

[0167] The fourth set of solutions illustrates exemplary implementations of the techniques discussed in the previous chapter (eg, item 3).

[0168] 1. A video processing method (e.g., method 900 shown in FIG. 9 ), comprising: performing (902) a conversion between a video including one or more layers and a video bitstream in accordance with rules, the rules providing for removing, in a sub-bitstream extraction process, Network Abstraction Layer (NAL) units including scalable nesting Supplementary Enhancement Information (SEI) messages that apply to layers that are not included in a target Output Layer Set (OLS).

[0169] 2. The method according to solution 1, wherein the scalable nesting SEI message is associated with a flag having a value equal to a particular value that specifies the scalable nesting SEI message that applies to the layer.

[0170] 3. The method according to solution 2, where the specific value is 0.

[0171] 4. The rule further specifies removing NAL units containing SEI messages that do not have a value in the first list (NestingLayerID[i]) in a second list (LayerIdInOls[targeOlsIdx]), where the first list specifies NAL unit header identifier values ​​of layers to which the scalable nesting SEI message applies, and i is in the range of 0 to NumLayers-1, where NumLayer indicates the number of layers to which the scalable nesting SEI message applies, and the second list specifies NAL unit header identifier values ​​at the target output layer set by the target output layer index.

[0172] 5. The method according to any of solutions 1 to 4, wherein the conversion comprises encoding the video into a bitstream.

[0173] 6. The method according to any of solutions 1 to 4, wherein the conversion includes decoding the video from a bitstream.

[0174] 7. A method according to any of Solutions 1 to 4, wherein the conversion includes generating a bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0175] 8. A video processing device comprising a processor configured to implement the method according to any one or more of solutions 1 to 7.

[0176] 9. A method for storing a video bitstream comprising the method according to any one of solutions 1 to 7 and further comprising storing the bitstream in a non-transitory computer-readable recording medium.

[0177] 10. A computer-readable medium storing program code that, when executed, causes a processor to implement a method according to any one or more of solutions 1 to 7.

[0178] 11. A computer readable medium storing a bitstream generated according to any of the methods described above.

[0179] 12. A video processing device for storing a bitstream representation, configured to implement a method according to any one or more of solutions 1 to 7.

[0180] The fifth solution set illustrates exemplary implementations of the techniques discussed in the previous chapter (eg, items 5-7).

[0181] 1. A video processing method (e.g., method 1000 shown in FIG. 10A), comprising performing conversion between a video including one or more layers and a video bitstream according to rules, the rules specifying that in the process of sub-bitstream extraction, a non-scalable nested SEI message is generated by extracting a scalable nested SEI message from a scalable nesting SEI message based on a first flag indicating whether the SEI (Supplementary Enhancement Information) message applies to a particular OLS (Output Layer Set) and a second flag indicating whether the SEI message applies to all subpictures or only to a particular subpicture.

[0182] 2. The method of Solution 1, wherein the rule specifies generating a non-scalable nested SEI message in response to the following conditions being satisfied: i) a first flag has a first value specifying that the SEI message applies to a particular Output Layer Set (OLS); and ii) a second flag has a second value specifying that the SEI message applies to all sub-pictures of the particular OLS.

[0183] 3. A video processing method (e.g., method 1010 shown in FIG. 10B), comprising performing (1012) a conversion between a video including one or more layers and a video bitstream in accordance with rules, the rules specifying that in the process of sub-bitstream extraction, a non-scalable nested SEI message is generated by extracting multiple scalable nested SEI messages from a first SEI (Supplemental Enhancement Information) NAL (Network Abstraction Layer) unit in a picture unit.

[0184] 4. The method according to solution 3, wherein the multiple scalable nested SEI messages are included in a second SEI NAL unit that is included in the picture unit.

[0185] 5. The second SEI NAL unit immediately follows the first SEI NAL unit; The method described in Solution 4.

[0186] 6. A video processing method (e.g., method 1020 shown in FIG. 10C), comprising performing (1022) a conversion between a video including one or more layers and a video bitstream in accordance with rules, the rules specifying a sub-bitstream extraction process to generate an output bitstream, the rules specifying handling of one or more Supplemental Enhancement Information (SEI) Network Abstraction Layer (NAL) units during the sub-bitstream extraction process.

[0187] 7. The method according to solution 6, wherein the rules specify that some SEI messages are extracted and that one or more SEI NAL units that were included as non-scalable nested SEI messages are removed.

[0188] 8. The method according to solution 6, wherein the rules provide for removing SEI NAL units that contain scalable nested SEI messages that apply only to the target output layer set.

[0189] 9. The method according to solution 6, wherein the rule specifies removing SEI NAL units that contain scalable nested SEI messages that apply to an Output Layer Set (OLS) for which no OLS other than the target OLS exists.

[0190] 10. The method according to any of solutions 1 to 9, wherein the conversion includes encoding the video into a bitstream.

[0191] 11. The method according to any of solutions 1 to 9, wherein the conversion includes decoding the video from a bitstream.

[0192] 12. A method according to any of Solutions 1 to 9, wherein the conversion includes generating a bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0193] 13. A video processing device comprising a processor configured to implement the method according to any one or more of solutions 1 to 12.

[0194] 14. A method for storing a video bitstream comprising the method according to any one of solutions 1 to 12 and further comprising storing the bitstream in a non-transitory computer-readable recording medium.

[0195] 15. A computer-readable medium having stored thereon program code that, when executed, causes a processor to implement a method according to any one or more of solutions 1 to 12.

[0196] 16. A computer readable medium storing a bitstream generated according to any of the methods described above.

[0197] 17. A video processing device for storing a bitstream representation, configured to implement a method according to any one or more of solutions 1 to 12.

[0198] Implementations of the disclosed and other solutions, examples, embodiments, modules, and functional operations described herein, including the structures disclosed herein and their structural equivalents, may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or in one or more combinations thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for implementation by or for controlling the operation of a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors, or computers. In addition to hardware, the apparatus may include code that creates an environment for the execution of the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, for example a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiving device.

[0199] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be recorded as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), may be stored in a single file dedicated to the program, or may be stored in multiple coordinating files (e.g., files that store one or more modules, subprograms, or code portions). A computer program can be deployed to run on one computer, or it can be deployed to run on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0200] The processing and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processing and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).

[0201] Processors suitable for executing computer programs include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer may include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or may be operatively coupled to receive data from or transfer data to these mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, EPROMs, EEPROMs, flash storage devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and semiconductor storage devices such as CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0202] Although this patent specification includes many features, these should not be construed as limiting the scope of any subject matter or the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described in this patent document in the context of separate embodiments may be implemented in combination in one example. Conversely, various features described in the context of one example may be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be extracted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0203] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desired results, nor should the separation of various system components in the embodiments described in this patent specification be understood as requiring such separation in all embodiments.

[0204] Only certain implementations and examples have been described; other embodiments, extensions and variations are possible based on the content described and illustrated in this patent document.

Claims

1. 1. A method of image processing, comprising: performing a conversion between a video having a plurality of layers and a bitstream of said video according to a rule; The method of claim 1, wherein the rules specify maximum allowable values ​​of temporal layer identifiers of sub-bitstreams extracted from the bitstream.

2. The method of claim 1 , wherein the rule corresponds to a number of layers in the bitstream.

3. 3. The method of claim 1, wherein if the number of layers is greater than 1, the rules specify that the maximum allowed value of the temporal layer identifier is in the range from 0 to a value based on a first syntax element in a video parameter set referenced by the bitstream.

4. 3. The method of claim 1 or 2, wherein if the number of layers is equal to 1, the rule specifies that the maximum allowed value of the temporal layer identifier is in the range from 0 to a value based on a second syntax element in a sequence parameter set referenced by the bitstream.

5. The method of claim 3 , wherein the first syntax element defines a maximum number of temporal sub-layers allowed to be present in a layer defined by the video parameter set minus one.

6. The method of claim 4 , wherein the second syntax element specifies a maximum number of temporal sub-layers allowed to be present in a layer specified by the sequence parameter set minus one.

7. The method of claim 3 , wherein the first syntax element in the video parameter set is vps_max_sublayers_minus1.

8. The method of claim 4 , wherein the second syntax element in the sequence parameter set is sps_max_sublayers_minus1.

9. The method of claim 3 or 4, wherein the value is equal to a value of the first syntax element or the second syntax element.

10. 1. A method of image processing, comprising: performing a conversion between a video and a bitstream of said video according to a rule; The rules define Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream; The method specifies that the rule derives the sub-bitstream based on whether a list of NAL unit header identifiers in an Output Layer Set (OLS) having a target OLS index does not include all values ​​of NAL unit header identifiers in all Video Coding Layer (VCL) NAL units of the bitstream input to the sub-bitstream extraction process.

11. 11. The method of claim 10, wherein the rules specify that if the list of NAL unit header identifiers in the output layer set does not include all values ​​of NAL unit header identifiers in all VCL NAL units of the bitstream, then all SEI NAL units, including non-scalable nested Supplemental Enhancement Information (SEI) messages, having a payload type equal to 0 or 130 are removed from the sub-bitstream.

12. 1. A method of image processing, comprising: performing a conversion between a video and a bitstream of said video according to a rule; The rules define Network Abstraction Layer (NAL) units to be extracted from the bitstream during a sub-bitstream extraction process to output the sub-bitstream; 11. The method of claim 1, wherein the rules specify that a Supplemental Enhancement Information (SEI) Network Abstraction Layer (NAL) unit is not permitted to contain an SEI message having a predefined payload type corresponding to a payload type of a first SEI message contained in the SEI NAL unit.

13. the payload type of the first SEI message is 0, 1, or 130; The method of claim 12 , wherein the rules specify that the SEI NAL unit is not allowed to contain the SEI message having the predefined payload type not equal to 0, 1, 130, or 133.

14. 1. A method of image processing, comprising: performing a conversion between a video and a bitstream of said video according to a format rule; The method, wherein the format rules specify that the bitstream includes a flag that specifies whether one or more non-scalable nested Supplemental Enhancement Information (SEI) messages having one or more particular payload types apply to all output layer sets referenced by a coding layer.

15. The method of claim 14 , wherein the one or more specific payload types are equal to 1 and 130.

16. The method of claim 14 , wherein the one or more non-scalable SEI messages are a Picture Timing (PT) SEI message and a Decoding Unit Information (DUI) SEI message.

17. The method of claim 14 , wherein the one or more specific payload types are equal to 0, 1, and 130.

18. The method of claim 14 , wherein the one or more non-scalable SEI messages are a Buffering Period (BP) SEI message and a Decoding Unit Information (DUI) message.

19. The method of claim 14 , wherein the one or more specific payload types are equal to 130.

20. The method of claim 14 , wherein the one or more non-scalable SEI messages are Decoding Unit Information (DUI) SEI messages.

21. The method of claim 14 , wherein the one or more specific payload types are equal to 0.

22. The method of claim 1 , wherein the one or more non-scalable SEI messages are Buffering Period (BP) SEI messages.

23. 23. The method of claim 1, wherein the converting comprises encoding the video into the bitstream.

24. 23. The method of claim 1, wherein the converting comprises decoding the video from the bitstream.

25. the converting includes generating the bitstream from the video; The method of claim 1 , further comprising storing the bitstream on a non-transitory computer-readable recording medium.

26. 26. A video processing device having a processor configured to implement a method according to any one of claims 1 to 25.

27. 1. A method for storing a video bitstream, comprising:

26. The method according to any one of claims 1 to 25, The method further comprises storing the bitstream on a non-transitory computer-readable recording medium.

28. A computer readable medium storing program code which, when executed, causes a processor to implement a method according to any one of claims 1 to 25.

29. A computer readable medium storing a bitstream generated according to any of the methods described above.

30. 1. A video processing device for storing a bitstream representation, comprising: The video processing device is configured to implement a method according to any one of claims 1 to 25.

Images