Video data processing method, video data processing device and medium

Abstract

Examples of video encoding methods and apparatus and video decoding methods and apparatus are described. An example method of video processing includes performing conversion between a video and a bitstream of the video according to a rule, wherein, according to the rule, the bitstream includes network abstraction layer (NAL) units for a plurality of video layers; wherein the rule defines a sub-bitstream extraction process for generating an output bitstream including an output layer set (OLS) including one or more operations selectively performed in response to the following conditions: (1) a list of NAL unit header layer identifier values ​​in the OLS does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the bitstream, and (2) the output bitstream includes a supplemental enhancement information (SEI) NAL unit that includes a scalable nested SEI message.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is an application entering the Chinese national phase of International Patent Application No. PCT / US2021 / 036473 filed on June 8, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 036,865 filed on June 9, 2020. The entire disclosure of the above application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to image and video encoding and decoding. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to perform video encoding or decoding.

[0006] In one example aspect, a video processing method is disclosed. The method includes performing conversion between a video and a bitstream of the video, wherein the bitstream includes network abstraction layer (NAL) units for a plurality of video layers according to a rule; wherein the rule defines a sub-bitstream extraction process by which the NAL units are removed from the bitstream to generate an output bitstream, and wherein the rule provides for removing all supplemental enhancement information (SEI) NAL units containing non-scalable nested SEI messages with a specific payload type in response to a list of NAL unit header layer identifier values ​​in an output layer set (OLS) having a target OLS index, the target OLS index not including all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the bitstream input to the sub-bitstream extraction process.

[0007] In another example aspect, another video processing method is disclosed. The method includes performing conversion between a video and a bitstream of the video, wherein the bitstream is divisible into one or more sub-bitstreams according to a rule that specifies a sub-bitstream extraction process to generate an output bitstream, and wherein the rule specifies whether or how to remove all supplemental enhancement information (SEI) network abstraction layer (NAL) units containing SEI messages that apply to a picture or sub-picture from which the VCL NAL unit is removed during the sub-bitstream extraction process based on a type of the VCL NAL unit and a temporal identifier of the video codec layer associated with the VCL NAL unit.

[0008] In another example aspect, another video processing method is disclosed. The method includes performing conversion between a video and a bitstream of the video according to a rule, wherein, according to the rule, the bitstream includes network abstraction layer (NAL) units for a plurality of video layers; wherein the rule defines a sub-bitstream extraction process for generating an output bitstream including an output layer set (OLS) including one or more operations selectively performed in response to the following conditions: (1) the list of NAL unit header layer identifier values ​​in the OLS does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the bitstream, and (2) the output bitstream includes a supplemental enhancement information (SEI) NAL unit, the SEI NAL unit including a scalable nested SEI message.

[0009] In yet another exemplary aspect, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the above method.

[0010] In yet another exemplary aspect, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement the above method.

[0011] In yet another exemplary aspect, a computer-readable medium having stored thereon code is disclosed. The code is in the form of processor-executable code embodying one of the methods described herein.

[0012] These and other features are described in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram illustrating a video encoding and decoding system according to some implementations of the disclosed technology.

[0014] Figure 2 is a block diagram of an example hardware platform for video processing.

[0015] Figure 3 is a flow chart of an example method of video processing.

[0016] Figure 4 is a block diagram illustrating an example video encoding and decoding system.

[0017] Figure 5 is a block diagram illustrating an encoder according to some implementations of the disclosed technology.

[0018] Figure 6 is a block diagram illustrating a decoder according to some implementations of the disclosed technology.

[0019] Figure 7A and 7B is a flow chart of an example method of video processing based on some implementations of the disclosed technology.

[0020] Figure 8 is a flow chart of an example method of video processing based on some implementations of the disclosed technology. DETAILED DESCRIPTION

[0021] The section headings used in this document are for ease of understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to that section. Furthermore, the use of H.266 terminology in some descriptions is for ease of understanding only and is not intended to limit the scope of the disclosed techniques. Therefore, the techniques described herein are also applicable to other video codec protocols and designs.

[0022] 1. Introduction

[0023] This document is about video codec technology. Specifically, it is about some improvements to the general sub-bitstream extraction process, the signaling of picture-level HRD parameters, and the inclusion of SEI messages in SEI NAL units. These ideas can be applied alone or in various combinations to any video codec standard or non-standard video codec that supports multi-layer video codecs (for example, the Versatile Video Codec (VVC) under development).

[0024] 2. Abbreviation

[0025] APS Adaptive Parameter Set

[0026] AU Access Unit

[0027] AUD Access Unit Delimiter

[0028] AVC Advanced Video Codec

[0029] CLVS codec layer video sequence

[0030] CPB codec picture buffer

[0031] CRA Clean Random Access

[0032] CTU Codec Tree Unit

[0033] CVS codec video sequence

[0034] DCI decoding capability information

[0035] DPB decoded picture buffer

[0036] EOB End of bitstream

[0037] EOS sequence ends

[0038] GDR Gradual Decode Refresh

[0039] HEVC High-Efficiency Video Codec

[0040] HRD Hypothesized Reference Decoder

[0041] IDR Instant Decode Refresh

[0042] ILP inter-layer prediction

[0043] ILRP inter-layer reference image

[0044] JEM Joint Exploration Model

[0045] LTRP Long Term Reference Picture

[0046] MCTS motion constraint set

[0047] NAL Network Abstraction Layer

[0048] OLS output layer set

[0049] PH Image Header

[0050] PPS picture parameter set

[0051] PTL grade, level, and grade

[0052] PU picture unit

[0053] RAP Random Access Point

[0054] RBSP Raw Byte Sequence Payload

[0055] SEI Supplemental Enhancement Information

[0056] SPS sequence parameter set

[0057] STRP Short-Term Reference Picture

[0058] SVC Scalable Video Codec

[0059] VCL video codec layer

[0060] VPS Video Parameter Set

[0061] VTM VVC test model

[0062] VUI Video Availability Information

[0063] VVC multifunctional video codec

[0064] 3. Preliminary Discussion

[0065] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced the H.261 and H.263 standards, ISO / IEC produced the MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly produced the H.262 / MPEG-2 Video standard, the H.264 / MPEG-4 Advanced Video Codec (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards have been based on a hybrid video codec architecture that utilizes temporal prediction and transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). JVET meetings are held concurrently every quarter, with the goal of reducing the bitrate of new codecs by 50% compared to HEVC. The new video codec standard was officially named Versatile Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. As VVC standardization continues, new codec technologies are incorporated into the VVC standard at each JVET meeting. The VVC working draft and test model (VTM) are updated after each meeting. The VVC project is currently aiming for technical completion (FDIS) at the July 2020 meeting.

[0066] 3.1. Variation of image accuracy within a sequence

[0067] In AVC and HEVC, the spatial precision of a picture cannot be changed unless a new sequence using a new SPS starts with an IRAP picture. VVC allows changing picture precision within a sequence without encoding an IRAP picture; IRAP pictures are always intra-coded. This feature is sometimes called reference picture resampling (RPR) because it requires resampling the reference pictures used for inter prediction when the reference pictures have a different precision than the current picture being decoded.

[0068] The scaling ratio is restricted to be greater than or equal to 1 / 2 (2x downsampling from the reference picture to the current picture) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luma and 32 phases for chroma, which is the same as the case of motion compensated interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process, where the scaling ratio ranges from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height and the left, right, top, and bottom scaling offsets specified for the reference picture and the current picture.

[0069] Other aspects of the VVC design that support this feature that differ from HEVC include: i) the picture precision and corresponding consistency window are signaled in the PPS rather than in the SPS, where the maximum picture precision is signaled; ii) for a single-layer bitstream, each picture store (a slot in the DPB used to store one decoded picture) occupies the buffer size required to store the decoded picture with the maximum picture precision.

[0070] Scalable Video Codec (SVC) in General and VVC

[0071] Scalable Video Codec (SVC, sometimes also referred to as scalability in video codecs) refers to video codecs that use a base layer (BL) (sometimes called a reference layer (RL)) and one or more scalable enhancement layers (EL). In SVC, the base layer can carry video data at a base quality level. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise ratio (SNR) levels. Enhancement layers can be defined relative to previously coded layers. For example, the bottom layer can serve as the BL, while the top layer can serve as the EL. Intermediate layers can serve as the EL, the RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can serve as the EL for the layer below the intermediate layer (e.g., the base layer or any intervening enhancement layer) and simultaneously serve as the RL for one or more enhancement layers above the intermediate layer. Similarly, in the Multiview or 3D extension of the HEVC standard, there may be multiple views, and information from one view can be used to encode (e.g., encode or decode) information from another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0072] In SVC, parameters used by an encoder or decoder are grouped into parameter sets based on the codec level (e.g., video level, sequence level, picture level, slice level, etc.) at which they can be used. For example, parameters that can be used by one or more codec video sequences of different layers in a bitstream can be included in a video parameter set (VPS), and parameters that can be used by one or more pictures in a codec video sequence can be included in a sequence parameter set (SPS). Similarly, parameters used by one or more slices in a picture can be included in a picture parameter set (PPS), and other parameters specific to individual slices can be included in a slice header. Similarly, an indication of which parameter set(s) a particular layer is using at a given time can be provided at various codec levels.

[0073] Thanks to support for reference picture resampling (RPR) in VVC, support for bitstreams containing multiple layers can be designed without any additional signaling to process-level codec tools. For example, in VVC, two layers with SD and HD resolutions can be designed, as the upsampling required for spatial scalability support can use only the RPR upsampling filter. However, to support scalability, high-level syntax changes are required (compared to not supporting scalability). Scalability support is specified in VVC version 1. Unlike scalability support in any earlier video codec standards, including the extensions to AVC and HEVC, VVC scalability has been designed to be as friendly to single-layer decoder designs as possible. The decoding capabilities of multi-layer bitstreams are specified as if there is only a single layer in the bitstream. For example, the decoding capabilities such as the DPB size are specified in a way that is independent of the number of layers in the bitstream to be decoded. Essentially, a decoder designed for single-layer bitstreams can decode multi-layer bitstreams without requiring many changes. Compared to the design of the multi-layer extensions of AVC and HEVC, the HLS aspect is significantly simplified at the expense of some flexibility. For example, the IRAP AU needs to contain pictures from each layer present in the CVS.

[0074] Parameter Set

[0075] AVC, HEVC, and VVC specify parameter sets. Parameter set types include SPS, PPS, APS, and VPS. All AVC, HEVC, and VVC versions support SPS and PPS. VPS was introduced in HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0076] The SPS is designed to carry sequence-level header information, and the PPS is designed to carry infrequently changing picture-level header information. Using SPS and PPS eliminates the need to repeat infrequently changing information for each sequence or picture, thus avoiding redundant signaling of this information. Furthermore, using SPS and PPS enables out-of-band transmission of important header information, eliminating the need for redundant transmission and improving error resilience.

[0077] VPS is introduced to carry sequence-level header information common to all layers in a multi-layer bitstream.

[0078] APS was introduced to carry such picture-level or slice-level information, which requires quite a lot of bits to encode and decode, can be shared by multiple pictures, and can have many different variations in a sequence.

[0079] 3.4. General Sub-Bitstream Extraction Process

[0080] Clause C.6 of the latest VVC text specifies the general sub-bitstream extraction process as follows: C.6 Sub-bitstream extraction process

[0081] The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx and the target highest TemporalId value tIdTarget.

[0082] The output of this process is the sub-bitstream outBitstream.

[0083] The bitstream conformance requirement for an input bitstream is that any output sub-bitstream that satisfies all of the following conditions shall be a conforming bitstream:

[0084] – The output sub-bitstream is the output of the process specified in this clause, where as input the bitstream targetOlsIdx is equal to the index of the list of OLSs specified by the VPS, and tIdTarget is equal to any value in the range 0 to 6 (inclusive).

[0085] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0086] – The output sub-bitstream contains at least one VCL NAL unit with TemporalId equal to tIdTarget.

[0087] NOTE – A conforming bitstream contains one or more slice NAL units of a codec with TemporalId equal to 0, but does not necessarily contain a slice NAL unit of a codec with nuh_layer_id equal to 0.

[0088] The output sub-bitstream OutBitstream is derived as follows:

[0089] 1. The bitstream outBitstream is set to be the same as the bitstream inBitstream.

[0090] 2. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream.

[0091] 3. Remove from outBitstream all NAL units whose nal_unit_type is not equal to any of VPS_NUT, DCI_NUT and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx].

[0092] 4. Remove from outBitstream all VCL NAL units and their associated non-VCL NAL units with nal_unit_type equal to PH_NUT, FD_NUT, SUFFIX_SEI_NUT, and PREFIX_SEI_NUT for which PayloadType is not equal to 0, 1, or 130, for which all of the following conditions are true:

[0093] – nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type is equal to GDR_NUT and the associated ph_recovery_poc_cnt is not equal to 0.

[0094] For values ​​of j in the range of 0 to NumLayersInOls[targetOlsIdx]−1 (inclusive), nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j].

[0095] – TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0096] 5. Remove from outBitstream all SEI NAL units containing scalable nesting SEI messages with sn_ols_flag equal to 1 and no value of i in the range of 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] is equal to targetOlsIdx.

[0097] 6. When LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, the following applies:

[0098] a. Remove from outBitstream all SEI NAL units containing non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI).

[0099] b. When general_same_pic_timing_in_all_ols_flag is equal to 0, all SEI NAL units containing non-scalable nested SEI messages (PT) with payloadType equal to 1 are removed from outBitstream.

[0100] c. When outBitstream contains SEI NAL units and for outBitstream (NestingOlsIdx[i] equals targetOlsIdx) these units contain scalable nesting SEI messages with sn_ols_flag equal to 1, the following applies:

[0101] – If general_same_pic_timing_in_all_ols_flag is equal to 0, then appropriate non-scalable nesting SEI messages with payloadType equal to 0 (BP), 1 (PT), or 130 (DUI) are extracted from the scalable nesting SEI messages and included in outBitstream.

[0102] – Otherwise (general_same_pic_timing_in_all_ols_flag is equal to 1), extract the appropriate non-scalable nesting SEI messages with payloadType equal to 0 (BP) or 130 (DUI) from the scalable nesting SEI messages and include these SEI messages in outBitstream.

[0103] 4. Technical problems solved by the disclosed technical solutions

[0104] The existing design of the general sub-bitstream extraction process and other related parts in the latest VVC text (JVET-R2001-vA / v10) has the following problems:

[0105] 1) Under the condition that the output sub-bitstream is required to be a conforming bitstream, the value of tIdTarget is said to be in the range of 0 to 6 (inclusive). However, in many bitstreams, the highest TemporalId value is less than 6, and this value is specified by the syntax element vps_max_sublayers_minus1.

[0106] 2) When present, the access unit delimiter (AUD) NAL unit may have any nuh_layer_id value. However, step 3 of the sub-bitstream extraction process shall remove AUD NAL units whose nuh_layer_id value is not included in the list LayerIdInOls[targetOlsIdx].

[0107] 3) Some SEI NAL units contain scalable nesting SEI messages with sn_ols_flag equal to 0, and the applicable layers indicated in the scalable nesting SEI message do not include any layers in the target OLS, that is, no nuh_layer_id value of an applicable layer is included in the list layeridols[targetOlsIdx]. These SEI NAL units should also be removed.

[0108] 4) The condition of step 6, i.e., "when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream" has the following two problems.

[0109] a. This condition does not apply when DCI, VPS, AUD, or EOB NAL units are present and nuh_layer_id is not equal to any nuh_layer_id value of the VCL NAL unit.

[0110] b. The phrase "bitstream" is unclear because in the context there are two bitstreams involved, inBitstream and outBitstream.

[0111] 5) Step 6.c extracts the scalable nesting SEI message from the scalable nesting SEI message with sn_ols_flag equal to 1 and sn_subpic_flag equal to 1 to generate the non-scalable nesting SEI message, which only applies to a specific sub-picture and should not be extracted.

[0112] 6) In step 6.c, when multiple scalable nested SEI messages are extracted from one SEI NAL unit seiNalUnitA as non-scalable nested SEI messages, they should still be contained in one SEI NAL unit seiNalUnitB, and the SEI NAL unit seiNalUnitB should be contained in the same PU that contains the SEI NAL unit seiNalUnitA. However, this is not specified.

[0113] 7) Step 6.c should remove from outBitstream all SEI NAL units from which some SEI messages were extracted and included as non-scalable nested SEI messages. However, this is not specified.

[0114] 8) There is a lack of constraints such that a SEI NAL unit shall not contain an SEI message with a payloadType other than 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting) when the SEI NAL unit contains an SEI message with a payloadType equal to 0, 1, or 130. This results in the removal of the SEI message in step 4 involving more than just removing the SEI NAL unit.

[0115] 9) The flag general_same_pic_timing_in_all_ols_flag only specifies whether the non-scalable nested PTSEI message is applicable to all OLSs. However, the information carried in the DUI SEI message has a similar purpose to that in the PTSEI message.

[0116] 10) When removing VCL NAL units from the output bitstream where both of the following are true: a) nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type is equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0, and b) TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]], this step also removes associated SEI NAL units containing SEI messages other than BP, PT, or DUI SEI messages. However, some of those removed SEI messages may apply to OLSs or layers containing pictures remaining in the output bitstream.

[0117] 11) The sub-picture level information SEI message (when present) is applicable to OLS, just like other HRD-related SEI messages (i.e., BP, PT, DUI SEI messages). However, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, the SEI NAL unit containing the non-scalable nested SEI message (i.e., sub-picture level information SEI message) with payloadType equal to 203 is not addressed during extraction.

[0118] 12) The last step of making the scalable nested SEI message into a non-scalable nested SEI message has the following problems:

[0119] a. Excluding SEI messages where sn_ols_flag is equal to 0 and sn_subpic_flag is equal to 0.

[0120] b. It is unspecified where the resulting non-scalable nested SEI message should be placed in the output bitstream (in which SEI NAL unit, where the SEI NAL unit should be).

[0121] 5. List of technical solutions and implementation examples

[0122] To address the above and other issues, the methods summarized below are disclosed. These items should be considered as examples to explain the general concept and should not be interpreted in a narrow sense. In addition, these items can be applied alone or in any combination.

[0123] 1) To solve problem 1, the condition requiring the output sub-bitstream to be a conforming bitstream is specified so that the value of tIdTarget is specified to be within the range of 0 to vps_max_sublayers_minus1 (including the end value).

[0124] a. Alternatively, specify the conditions requiring the output sub-bitstream to be a conforming bitstream such that when there is more than one layer in the input bitstream, the value of tIdTarget is specified to be in the range of 0 to vps_max_sublayers_minus1 (inclusive), and when there is only one layer in the input bitstream, the value of tIdTarget is specified to be in the range of 0 to sps_max_sublayers_minus1 (inclusive).

[0125] 2) To address issue 2, a generic sub-bitstream extraction process is specified to treat AUD NAL units in the same way as NAL units with nal_unit_type equal to VPS_NUT, DCI_NUT, or EOB_NUT. In other words, no AUD NAL units are removed from the output bitstream outBitstream depending on the nuh_layer_id value.

[0126] 3) To address issue 3, a generic sub-bitstream extraction process is defined such that it removes SEINAL units from the output bitstream outBitstream that contain scalable nesting SEI messages with sn_ols_flag equal to 0, while the applicable layers indicated in the scalable nesting SEI message do not include any layers in the target OLS.

[0127] a. In one example, it is provided that all SEINAL units containing scalable nesting SEI messages with sn_ols_flag equal to 0 are removed from outBitstream and the list nestingLayerId[i] does not have a value in the list LayerIdInOls[targetOlsIdx] for i in the range of 0 to nestingNumLayers–1 (inclusive).

[0128] 4) To address issue 4, the condition “when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream” was changed to “when the list LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all VCL NAL units in the bitstream inBitstream”.

[0129] 5) To solve problem 5, the general sub-bitstream extraction process is specified so that it extracts scalable nesting SEI messages only from scalable nesting SEI messages with sn_ols_flag equal to 1 and sn_subpic_flag equal to 0 to generate non-scalable nesting SEI messages.

[0130] 6) In order to solve problem 6, a general sub-bitstream extraction process is specified so that when multiple scalable nested SEI messages are extracted from one SEI NAL unit seiNalUnitA as non-scalable nested SEI messages, they are still included in one SEI NAL unit seiNalUnitB in the output bitstream outBitstream, and the SEI NAL unit seiNalUnitB is included in the PU containing the SEI NAL unit seiNalUnitA.

[0131] 7) To address issue 7, a generic sub-bitstream extraction process is specified such that it removes from the output bitstream outBitstream all SEI NAL units from which some SEI messages were extracted and included as non-scalable nested SEI messages.

[0132] a. Alternatively, when the scalable nested SEI message in such SEI NAL unit is only applicable to the target OLS (ie, the targetOlsIdx-th OLS specified by the VPS), remove the SEI NAL unit from the outBitstream.

[0133] b. Alternatively, when there is no OLS containing all layers included in the list LayerIdInOls[targetOlsIdx] except the target OLS in the OLS to which the scalable nesting SEI message in this SEI NAL unit applies, remove the SEI NAL unit from outBitstream.

[0134] 8) To address issue 8, add a constraint such that when an SEI NAL unit contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL unit shall not contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting).

[0135] 9) To address issue 9, the flag general_same_pic_timing_in_all_ols_flag specifies whether the non-scalable nested PT and DUI SEI messages apply to all OLSs.

[0136] a. Alternatively, the flag general_same_pic_timing_in_all_ols_flag specifies whether the non-scalable nested BP, PT and DUI SEI messages apply to all OLSs.

[0137] i. In one example, the flag general_same_pic_timing_in_all_ols_flag is renamed to flag general_same_pic_level_hrd_info_in_all_ols_flag, which specifies whether the non-scalable nested BP, PT, and DUI SEI messages apply to all OLSs.

[0138] b. Alternatively, add a new flag, for example, named general_same_dui_in_all_ols_flag, to specify whether the non-scalable nested DUI SEI message applies to all OLSs.

[0139] c. Alternatively, add a new flag, for example, named general_same_bp_in_all_ols_flag, to specify whether the non-scalable nested BP SEI message applies to all OLSs.

[0140] 10) To address issue 10, in one example, in a general sub-bitstream extraction process, when a VCL NAL unit is removed from the output bitstream for which both of the following conditions are true: a) nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type is equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0, and b) TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]], instead of removing the associated SEI NAL units containing SEI messages other than BP, PT, or DUI SEI messages, all SEI NAL units containing SEI messages are removed, and the SEI messages apply only to the one or more pictures or sub-pictures for which all VCL NAL units are being removed.

[0141] a. In one example, alternatively, when such VCL NAL units are removed, the associated SEI NAL units are retained in the output bitstream.

[0142] 11) To address issue 11, in one example, in a generic sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, all SEI NAL units containing non-scalable nested SEI messages (i.e., sub-picture level information SEI messages) with payloadType equal to 203 are additionally removed.

[0143] 12) To address issue 12, in one example, in a generic sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, and outBitstream contains an SEI NAL unit seiNalUnitA containing a scalable nesting SEI message with sn_subpic_flag equal to 0 that applies to an OLS (when sn_ols_flag is equal to 1) or a layer with the same layer set as in outBitstream (when sn_ols_flag is equal to 0), perform one or more of the following operations:

[0144] a. Generate a new SEI NAL unit seiNalUnitB.

[0145] b. Include seiNalUnitB in the PU containing seiNalUnitA.

[0146] c. Include seiNalUnitB in the PU containing seiNalUnitA, immediately following seiNalUnitA.

[0147] d. Extract the scalable nesting SEI messages from the scalable nesting SEI message and include them directly in seiNalUnitB (as non-scalable nesting SEI messages).

[0148] e. Remove seiNalUnitA from outBitstream.

[0149] 13) In one example, in a generic sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, and outBitstream contains a SEI NAL unit seiNalUnitA that contains a scalable nesting SEI message, retain seiNalUnitA in the output bitstream.

[0150] 6. Examples

[0151] The following are some example embodiments of some of the aspects of the invention outlined in Section 5 above, which can be applied to the VVC specification. The changed text is based on the latest VVC text in JVET-R2001-vA / v10. The most relevant parts that have been added or modified are highlighted in bold and italics, and some of the deleted parts are marked with double brackets (e.g., [[a]] indicates the deletion of the character "a"). There may be other changes that are editorial in nature and are therefore not highlighted.

[0152] 6.1. First embodiment

[0153] This example addresses items 1, 2, 3, 3.a, 4, 5, 6, 7.b, and 8.

[0154] C.6 General sub-bitstream extraction process

[0155] The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx and the target highest TemporalId value tIdTarget.

[0156] The output of this process is the sub-bitstream outBitstream.

[0157] The bitstream conformance requirement for an input bitstream is that any output sub-bitstream that satisfies all of the following conditions shall be a conforming bitstream:

[0158] – The output sub-bitstream is the output of the process specified in this clause, where as input the bitstream has targetOlsIdx equal to the index of the list of OLSs specified by the VPS and tIdTarget equal to 0 to

[0159] Any value within the range of (including the end value).

[0160] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0161] – The output sub-bitstream contains at least one VCL NAL unit with TemporalId equal to tIdTarget.

[0162] NOTE – A conforming bitstream contains one or more slice NAL units of a codec with TemporalId equal to 0, but does not necessarily contain a slice NAL unit of a codec with nuh_layer_id equal to 0.

[0163] The output sub-bitstream OutBitstream is Derivation:

[0164] 1. The bitstream outBitstream is set to be the same as the bitstream inBitstream.

[0165] 2. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream.

[0166] 3. Remove from outBitstream all NAL units whose nal_unit_type is not equal to any of DCI_NUT, VPS_NUT, AUD_NUT and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx].

[0167] 4. Remove from outBitstream all VCL NAL units for which all of the following conditions are true, and The associated non-VCL NAL unit, nal_unit_type equals PH_NUT FD_NUT, SUFFIX_SEI_NUT PREFIX_SEI_NUT Not equal to 0 (BP), 1 (PT), or 130 (DUI)

[0168] – nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type is equal to GDR_NUT and the associated ph_recovery_poc_cnt is not equal to 0.

[0169] –[[For values ​​of j in the range of 0 to NumLayersInOls[targetOlsIdx]–1 (inclusive), nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j]]]

[0170] – TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0171] 5. Remove from outBitstream all SEI NAL units containing scalable nesting SEI messages with sn_ols_flag equal to 1 and no value of i in the range of 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] is equal to targetOlsIdx.

[0172]

[0173] 7. When LayerIdInOls[targetOlsIdx] does not include bitstream All For all values ​​of nuh_layer_id in the NAL unit, the following application:

[0174] a. Remove from outBitstream all SEI NAL units containing non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI).

[0175] b. When general_same_pic_timing_in_all_ols_flag is equal to 0, all SEI NAL units containing non-scalable nested SEI messages with payloadType equal to 1 (PT) are removed from outBitstream.

[0176] c. When outBitstream contains a SEI NAL unit, the SEI NAL unit contains sn_ols_flag equal to 1 The scalable nesting SEI message applies to the targetOlsIdx-th OLS (i.e., there is at least one value of i in the range of 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] is equal to targetOlsIdx), the following application:

[0177] i. For each scalable nested BP or DUI SEI message in such SEI NAL unit seiNalUnitA, In outBitstream

[0178] ii. When general_same_pic_timing_in_all_ols_flag is equal to 0, for each scalable nested PT SEI message in such SEI NAL unit seiNalUnitA, In outBitstream

[0179]

[0180] D.2.2 General SEI payload semantics ...

[0182] A requirement for bitstream conformance is that the following restrictions apply to the inclusion of SEI messages in SEI NAL units:

[0183] – When a SEI NAL unit contains a non-scalable nested BP SEI message, a non-scalable nested PT SEI message, or a non-scalable nested DUI SEI message, the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 0 (BP), 1 (PT), or 130 (DUI).

[0184] – When the SEI NAL unit contains a scalable nested BP SEI message, a scalable nested PT SEI message, or a scalable nested DUI SEI message, the SEI NAL unit shall not contain any other SEI messages with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting).

[0185] ...

[0187] 6.2. Second embodiment

[0188] This embodiment relates to items 10 to 12 (inclusive), and the changes relative to the text of the above embodiment 1 are highlighted.

[0189] C.6 General sub-bitstream extraction process

[0190] The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx and the target highest TemporalId value tIdTarget.

[0191] The output of this process is the sub-bitstream outBitstream.

[0192]

[0193] The bitstream conformance requirement for an input bitstream is that any output sub-bitstream that satisfies all of the following conditions shall be a conforming bitstream:

[0194] – The output sub-bitstream is the output of the process specified in this clause, where as input targetOlsIdx in the bitstream is equal to the index into the list of OLSs specified by the VPS, and tIdTarget is equal to any value in the range 0 to vps_max_sublayers_minus1 (inclusive).

[0195] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0196] – The output sub-bitstream contains at least one VCL NAL unit with TemporalId equal to tIdTarget.

[0197] NOTE – A conforming bitstream contains one or more slice NAL units of the codec with TemporalId equal to 0, but does not necessarily contain slice NAL units of the codec with nuh_layer_id equal to 0. The output sub-bitstream OutBitstream is derived by applying the following ordered steps:

[0198] 1. The bitstream outBitstream is set to be the same as the bitstream inBitstream.

[0199] 2. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream.

[0200] Remove [[nal_unit_type is not equal to DCI_NUT, VPS_NUT,

[0201] Any of AUD_NUT and EOB_NUT and]] nuh_layer_id not included in the list LayerIdInOls[targetOlsIdx]

[0202] All NAL units.

[0203] 4. Remove from outBitstream all VCL NAL units for which all of the following conditions are true, [[and also remove these VCL NAL units from outBitstream]] Associated non-VCL NAL units with nal_unit_type equal to PH_NUT or FD_NUT [[, or with nal_unit_type equal to SUFFIX_SEI_NUT or PREFIX_SEI_NUT and containing a SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (Scalable Nesting)]]:

[0204] –nal_unit_type equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equal to GDR_NUT and associated ph_recovery_poc_cnt [[Not equal to]]0.

[0205] – TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0206]

[0207] – Remove from outBitstream all SEI NAL units containing scalable nesting SEI messages with sn_ols_flag equal to 1 and no value of i in the range of 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] is equal to targetOlsIdx.

[0208] – Remove from outBitstream all SEI NAL units containing scalable nesting SEI messages with sn_ols_flag equal to 0 and with no value in the list LayerIdInOls[targetOlsIdx] in the list nestingLayerId[i] for i in the range 0 to nestingNumLayers–1, inclusive.

[0209]

[0210] 6. When LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all VCL NAL units in the bitstream inBitstream, the following apply in the order listed: a. Remove from outBitstream all layers containing payloadType equal to 0 (BP)

[0211] [[or 130(DUI)]] SEI NAL units of non-scalable nested SEI messages. b. When general_same_pic_timing_in_all_ols_flag is equal to 0, remove from outBitstream all SEI NAL units containing non-scalable nested SEI messages (PT) with payloadType equal to 1.

[0212]

[0213] [[When outBitstream contains a SEI NAL unit, the SEI NAL unit contains a scalable nesting SEI message with sn_ols_flag equal to 1 and sn_subpic_flag equal to 0, which applies to the targetOlsIdx-th OLS (i.e., there is at least one value of i in the range of 0 to sn_num_olss_minus1, inclusive, such that NestingOlsIdx[i] equals targetOlsIdx), the following apply in the order listed:

[0214] i. For each scalable nested BP or DUI SEI message in such SEI NAL unit seiNalUnitA, generate a non-scalable nested SEI message with the same payloadType and SEI payload and include it in the SEI NAL unit in the PU containing seiNalUnitA in outBitstream.

[0215] ii. When general_same_pic_timing_in_all_ols_flag is equal to 0, for each scalable nested PT SEI message in such SEI NAL unit seiNalUnitA, a non-scalable nested SEI message with the same SEI payload is generated and included in the SEI NAL unit in the PU containing seiNalUnitA in outBitstream.

[0216] iii. When a plurality of SEI messages contained in a specific such SEI NAL unit seiNalUnitA are made into non-scalable nested SEI messages, the non-scalable nested SEI messages are thereby included in one SEI NAL unit.

[0217] iv. When there is no OLS containing all the layers included in the list LayerIdInOls[targetOlsIdx] except the target OLS in the OLS to which the scalable nesting SEI message in this SEI NAL unit seiNalUnitA applies, remove the SEI NAL unit seiNalUnitA from outBitstream. ]]

[0218] Figure 1 1 is a block diagram of an example video processing system 1900 that may implement the various techniques disclosed herein. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8- or 10-bit multi-component pixel values), or may be received in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical networks (PONs), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0219] System 1900 may include a codec component 1904 that can implement the various codecs or encoding methods described in this document. Codec component 1904 can reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to produce a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 1904 can be stored or sent via a connected communication, as represented by component 1906. The stored or communicated bitstream (or codec) representation of the video received at input 1902 can be used by component 1908 to generate pixel values ​​or displayable video that is sent to display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tools or operations are used at the encoder, and the corresponding decoding tools or operations that invert the codec results will be performed by the decoder.

[0220] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), DisplayPort, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document can be implemented in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of digital data processing and / or video display.

[0221] Figure 2 36 is a block diagram of a video processing device 3600. Device 3600 can be used to implement one or more of the methods described herein. Device 3600 can be implemented in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Device 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. Processor(s) 3602 can be configured to implement one or more of the methods described herein. Memory(s) 3604 can be used to store data and code used to implement the methods and techniques described herein. Video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuitry.

[0222] Figure 4 is a block diagram illustrating an example video coding system 100 that may utilize the techniques of this disclosure.

[0223] like Figure 4 As shown, the video encoding and decoding system 100 may include a source device 110 and a target device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The target device 120 may decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device.

[0224] Source device 110 may include a video source 112 , a video encoder 114 , and an input / output (I / O) interface 116 .

[0225] The video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of the video data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other syntax elements. The I / O interface 116 includes a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be sent directly to the target device 120 via the I / O interface 116 via the network 130a. The encoded video data may also be stored on a storage medium / server 130b for access by the target device 120.

[0226] Target device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0227] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the target device 120 or may be external to the target device 120 and configured to interface with an external display device.

[0228] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or other standards.

[0229] Figure 5 is a block diagram illustrating an example of a video encoder 200, which may be Figure 4 The video encoder 114 in the system 100 is shown in FIG.

[0230] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. Figure 5 In the example of FIG, video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0231] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 (which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206), a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[0232] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode where at least one reference picture is a picture in which the current video block is located.

[0233] Furthermore, some components such as the motion estimation unit 204 and the motion compensation unit 205 may be highly integrated but are not shown for the purpose of explanation. Figure 5 In the example, they are shown separately.

[0234] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0235] The mode selection unit 203 can, for example, select one of the intra or inter coding modes based on the error result, and provide the resulting intra or inter coding block to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the coding block for use as a reference picture. In some examples, the mode selection unit 203 can select a combined intra and inter prediction (CIIP) mode, where the prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 203 can also select a resolution of motion vectors (e.g., sub-pixel or integer pixel accuracy) for the block in the case of inter prediction.

[0236] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of a picture from the buffer 213 (other than the picture associated with the current video block).

[0237] Motion estimation unit 204 and motion compensation unit 205 may perform different operations for the current video block, for example, performing different operations depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0238] In some examples, motion estimation unit 204 may perform unidirectional prediction of the current video block, and motion estimation unit 204 may search for a reference video block for the current video block in the reference pictures in list 0 or list 1. Motion estimation unit 204 may then generate a reference index indicating the reference video block in the reference pictures in list 0 or list 1, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 may generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0239] In other examples, the motion estimation unit 204 may perform bidirectional prediction for the current video block. The motion estimation unit 204 may search for a reference video block for the current video block in the reference pictures in list 0 and may also search for another reference video block for the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate a reference index indicating the reference video block in the reference pictures in list 0 or list 1 and a motion vector indicating the spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as motion information for the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information for the current video block.

[0240] In some examples, motion estimation unit 204 may output the entire set of motion information for use in a decoding process by a decoder.

[0241] In some examples, motion estimation unit 204 may not output the entire set of motion information for the current video. Instead, motion estimation unit 204 may reference the motion information of another video block to signal the motion information of the current video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of the neighboring video block.

[0242] In one example, motion estimation unit 204 may indicate in a syntax structure associated with the current video block a value that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0243] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0244] As discussed above, the video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0245] The intra-frame prediction unit 206 can perform intra-frame prediction on the current video block. When the intra-frame prediction unit 206 performs intra-frame prediction on the current video block, the intra-frame prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0246] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the predicted video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0247] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0248] Transform unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0249] After transform unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0250] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0251] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0252] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0253] Figure 6 is a block diagram illustrating an example of a video decoder 300, which may be Figure 4 The video decoder 124 in the system 100 is shown in FIG.

[0254] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 6 In the example of FIG, video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0255] exist Figure 6 In the example of FIG, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform the same operation as that of the video encoder 200 ( Figure 5 ) is a decoding process that is the overall inverse of the encoding process described.

[0256] The entropy decoding unit 301 can retrieve a coded bitstream. The coded bitstream can include entropy-coded video data (e.g., coded blocks of video data). The entropy decoding unit 301 can decode the entropy-coded video data, and based on the entropy-decoded video data, the motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 can determine such information, for example, by performing AMVP and merge modes.

[0257] The motion compensation unit 302 may generate a motion compensated block, possibly interpolated based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in a syntax element.

[0258] The motion compensation unit 302 may calculate interpolated values ​​of a sub-integer number of pixels of the reference block using the interpolation filter used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 based on received syntax information and use the interpolation filter to generate a prediction block.

[0259] The motion compensation unit 302 may use some syntax information to determine: the size of the blocks used to encode the frame(s) and / or slice(s) of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the coded video sequence.

[0260] The intra prediction unit 303 can form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0261] Reconstruction unit 306 can sum the residual block with the corresponding prediction block generated by motion compensation unit 302 or intra prediction unit 303 to form a decoded block. If desired, a deblocking filter can also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also produces decoded video for presentation on a display device.

[0262] A list of preferred solutions for some embodiments is provided below.

[0263] A first set of solutions is provided next. The following solutions show example embodiments of the techniques discussed in the previous section (eg, items 1-9).

[0264] 1. A video processing method (e.g., Figure 3 ), comprising: performing (602) a conversion between a video comprising one or more video layers and a codec representation of the video, the one or more video layers comprising one or more video pictures, wherein the codec representation conforms to format rules associated with extracting a sub-bitstream from the codec representation.

[0265] 2. The method according to solution 1 further includes: extracting a sub-bitstream from the codec representation according to a format rule.

[0266] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 1).

[0267] 3. The method according to any of solutions 1-2, wherein during extraction of the sub-bitstream, the target id allowed for the extraction is in the range 0 to the value of the syntax field indicated in the video parameter set for the codec representation.

[0268] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 2).

[0269] 4. The method according to any of solutions 1-3, wherein the sub-bitstream is extracted without removing the access unit delimiter network abstraction layer (AUD NAL) from the output bitstream according to the layer id.

[0270] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 3).

[0271] 5. The method according to any of solutions 1-4, wherein the sub-bitstream is extracted by selectively removing network abstraction layer units that include scalable nested supplementary enhancement information messages that are not applicable to the extracted output layer.

[0272] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 5).

[0273] 6. A method according to any one of solutions 1-5, wherein the sub-bitstream is extracted by constraining the extraction to generate a non-scalable nested supplemental enhancement information (SEI) from a scalable nested SEI message using a flag that sets the output layer set and a flag that disables the sub-picture.

[0274] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 6).

[0275] 7. The method according to any of solutions 1-6, wherein the sub-bitstreams are extracted according to rules that specify the extraction of multiple scalable nested supplemental enhancement information (SEI) messages from a single SEI network abstraction layer unit.

[0276] The following solution shows an example embodiment of the technique discussed in the previous section (eg, item 7).

[0277] 8. A method according to any of solutions 1-7, wherein the sub-bitstream is extracted according to a rule of removing all Supplementary Enhancement Information Network Abstraction Layer SEI NAL units from the codec representation, from which some SEI messages have been extracted and included as non-scalable nested SEI messages.

[0278] The following solutions show example embodiments of the techniques discussed in the previous section (eg, items 8-13).

[0279] 10. A method for video processing, comprising: performing conversion between a video and a codec representation of the video, wherein the codec representation includes one or more sub-bitstreams; wherein the conversion is performed according to rules that specify a relationship between a sub-bitstream extraction process and one or more syntax elements of the codec representation.

[0280] 11. A method according to solution 10, wherein the rule specifies that all supplemental enhancement information (SEI) network abstraction layer units (NALs) containing SEI messages that apply to a picture or sub-picture of a removed video codec layer are removed if one or more of the following two conditions are true: (1) the type of the NAL unit is a specific type, or (2) the temporal identifier of the video codec layer meets a specific condition.

[0281] 12. A method according to any of Solutions 10-11, wherein the rule specifies that if the layer identifier in the target output layer does not include all values ​​of the layer identifier in the codec representation, then the process also removes all SEI NAL units containing non-scalable nested SEI messages with payloadType equal to 203.

[0282] 13. The method of any of solutions 1-12, wherein performing the conversion comprises encoding the video into a codec representation.

[0283] 14. The method of any of solutions 1-12, wherein performing the conversion comprises parsing and decoding the codec representation to generate the video.

[0284] 15. A video decoding device, comprising a processor configured to implement one or more of the methods described in solutions 1 to 14.

[0285] 16. A video encoding device comprising a processor configured to implement one or more of the methods described in solutions 1 to 14.

[0286] 17. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of solutions 1 to 14.

[0287] 18. The methods, apparatus, or systems described in this document.

[0288] The second set of solutions shows example embodiments of the techniques discussed in the previous section (eg, items 10 and 11).

[0289] 1. A video data processing method (e.g., Figure 7AThe method 700 is shown, comprising: performing (702) conversion between a video and a bitstream of the video, wherein the bitstream includes network abstraction layer (NAL) units for a plurality of video layers according to a rule; wherein the rule defines a sub-bitstream extraction process by which the NAL units are removed from the bitstream to generate an output bitstream, and wherein the rule provides for removing all supplemental enhancement information (SEI) NAL units containing non-scalable nested SEI messages with a specific payload type responsive to a list of NAL unit header layer identifier values ​​in an output layer set (OLS) having a target OLS index, the target OLS index not including all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the bitstream input to the sub-bitstream extraction process.

[0290] 2. The method of solution 1, wherein the non-scalable nested SEI message with a specific payload type corresponds to a sub-picture level information SEI message.

[0291] 3. The method of solution 1, wherein the specific payload type is equal to 203.

[0292] 4. The method of solution 1, wherein the non-scalable nested SEI message with a specific payload type corresponds to a decoding unit information message.

[0293] 5. The method of solution 1, wherein the specific payload type is equal to 130.

[0294] 6. A video data processing method (e.g., Figure 7B The method 710 is shown, comprising: (712) performing conversion between a video and a bitstream of the video, wherein the bitstream is divisible into one or more sub-bitstreams according to rules that specify a sub-bitstream extraction process to generate an output bitstream, and wherein the rules specify whether or how to remove all supplemental enhancement information (SEI) network abstraction layer (NAL) units containing SEI messages that apply to a picture or sub-picture from which the VCL NAL units are removed during the sub-bitstream extraction process based on a type of the VCL NAL unit and a temporal identifier of the video codec layer associated with the VCL NAL unit.

[0295] 7. The method of solution 6, wherein the rule provides for removing all SEI NAL units in response to: (1) the type of the VCL NAL unit being equal to TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, or GDR_NUT, and (2) the temporal identifier of the video codec layer being greater than or equal to the number of sublayers in the video codec layer in the output layer set.

[0296] 8. The method of solution 7, wherein for a type of VCL NAL unit equal to GDR_NUT, the syntax field indicating the recovery point of a decoded picture in output order has a value greater than 0.

[0297] 9. The method of solution 6, wherein the rules specify that SEI NAL units are preserved in the output bitstream.

[0298] 10. The method of any one of solutions 1 to 9, wherein converting comprises encoding the video into a bitstream.

[0299] 11. The method of any one of solutions 1 to 9, wherein converting comprises decoding the video from a bitstream.

[0300] 12. The method of any one of solutions 1 to 9, wherein converting comprises generating a bitstream from the video, and the method further comprises storing the bitstream in a non-transitory computer-readable recording medium.

[0301] 13. A video processing device, comprising a processor configured to implement any one or more of the methods described in solutions 1 to 12.

[0302] 14. A method of storing a bitstream of a video, comprising the method according to any one of solutions 1 to 12, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0303] 15. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method according to any one or more of solutions 1 to 12.

[0304] 16. A computer-readable medium storing a codec representation or a bitstream generated according to any one of the above methods.

[0305] 17. A video processing device storing a bitstream, wherein the video processing device is configured to implement the method according to any one or more of solutions 1 to 12.

[0306] The third set of solutions shows example embodiments of the techniques discussed in the previous section (eg, items 12 and 13).

[0307] 1. A video data processing method (e.g., Figure 8The method 800 shown includes: performing (802) conversion between a video and a bitstream of the video according to a rule, wherein, according to the rule, the bitstream includes network abstraction layer (NAL) units for multiple video layers; wherein the rule defines a sub-bitstream extraction process for generating an output bitstream including an output layer set (OLS) including one or more operations selectively performed in response to the following conditions: (1) the list of NAL unit header layer identifier values ​​in the OLS does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the bitstream, and (2) the output bitstream includes a supplemental enhancement information (SEI) NAL unit that includes a scalable nested SEI message.

[0308] 2. The method according to solution 1, wherein the scalable nesting SEI message is associated with a first flag equal to a specific value, the specific value being applied to a specific OLS or a specific layer having the same set of layers as in the output bitstream.

[0309] 3. The method according to solution 2, wherein the first flag equal to 0 specifies that the scalable nesting SEI message applicable to a specific OLS or a specific layer applies to all sub-pictures of the specific OLS or the specific layer.

[0310] 4. The method according to solution 1, wherein the scalable nesting SEI message is associated with a second flag indicating whether the scalable nesting SEI message applies to a specific OLS or a specific layer.

[0311] 5. A method according to any of solutions 1-4, wherein one or more operations include generating additional SEINAL units.

[0312] 6. The method of solution 5, wherein one or more operations include including additional SEI NAL units in picture units containing SEI NAL units.

[0313] 7. The method of solution 6, wherein the additional SEI NAL unit immediately follows the SEI NAL unit.

[0314] 8. The method of any one of solution 6, wherein the one or more operations further comprise extracting the scalable nesting SEI message from the scalable nesting SEI message and including the scalable nesting SEI message in an additional SEI NAL unit.

[0315] 9. The method of solution 1, wherein the one or more operations include removing SEINAL units from the output bitstream.

[0316] 10. The method of solution 1, wherein the one or more operations include preserving SEI NAL units in the output bitstream.

[0317] 11. The method of any one of solutions 1 to 10, wherein converting comprises encoding the video into a bitstream.

[0318] 12. The method of any one of solutions 1 to 10, wherein converting comprises decoding the video from a bitstream.

[0319] 13. The method of any one of solutions 1 to 10, wherein converting comprises generating a bitstream from the video, and the method further comprises storing the bitstream in a non-transitory computer-readable recording medium.

[0320] 14. A video processing device, comprising a processor configured to implement any one or more of the methods described in solutions 1 to 13.

[0321] 15. A method of storing a bitstream of a video, comprising the method according to any one of solutions 1 to 13, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0322] 16. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method described in any one or more of solutions 1 to 13.

[0323] 17. A computer-readable medium storing a bit stream generated according to any one of the above methods.

[0324] 18. A video processing device storing a bitstream, wherein the video processing device is configured to implement any one or more of the methods described in solutions 1 to 13.

[0325] The disclosed and other aspects, examples, embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions, encoded on a computer-readable medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composite material that effects a machine-readable, propagable signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0326] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file preserving other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers, which are located at a site or distributed across multiple sites and interconnected by a communication network.

[0327] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)).

[0328] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to a mass storage device (e.g., magnetic, magneto-optical, or optical disks), or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[0329] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of protection that may be claimed, but rather as descriptions of features specified in specific embodiments of a particular technology. In this patent document, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in various suitable sub-combinations. In addition, although features may be described above as working in certain combinations and even initially claimed in the same manner, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0330] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0331] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video data processing method, comprising: performing conversion between a video comprising one or more layers and a bitstream of said video according to a rule, The rule states that, during a sub-bitstream extraction process of removing a Network Abstraction Layer (NAL) unit from an input bitstream to generate an output sub-bitstream, In response to at least (1) the list of NAL unit header layer identifier values ​​in the target output layer set (OLS) with the OLS index does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the input bitstream and (2) the output sub-bitstream includes a first supplemental enhancement information (SEI) NAL unit, the first SEI NAL unit including a scalable embedding SEI message with sn_subpic_flag equal to 0 and sn_ols_flag equal to 0, A new SEI NAL unit is generated, and a scalable nested SEI message is extracted from the scalable embedded SEI message and included in the new SEI NAL unit as a non-scalable nested SEI message.

2. The method according to claim 1, wherein The new SEINAL unit is included in the picture unit PU including the first SEINAL unit.

3. The method according to claim 2, wherein: The new SEINAL unit is included immediately after the first SEINAL unit.

4. The method according to claim 1, wherein After the scalable nesting SEI message is included in the new SEI NAL unit, the first SEI NAL unit is removed from the output sub-bitstream.

5. The method according to any one of claims 1 to 4, wherein The converting includes encoding the video into the bitstream.

6. The method according to any one of claims 1 to 4, wherein The converting includes decoding the video from the bitstream.

7. A video data processing apparatus comprising a processor and a non-transitory memory having instructions thereon, wherein: The instructions, when executed by the processor, cause the processor to: performing conversion between a video comprising one or more layers and a bitstream of said video according to a rule, The rule states that, during a sub-bitstream extraction process of removing a Network Abstraction Layer (NAL) unit from an input bitstream to generate an output sub-bitstream, In response to at least (1) the list of NAL unit header layer identifier values ​​in the target output layer set (OLS) with the OLS index does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the input bitstream and (2) the output sub-bitstream includes a first supplemental enhancement information (SEI) NAL unit, the first SEI NAL unit including a scalable embedding SEI message with sn_subpic_flag equal to 0 and sn_ols_flag equal to 0, A new SEI NAL unit is generated, and a scalable nested SEI message is extracted from the scalable embedded SEI message and included in the new SEI NAL unit as a non-scalable nested SEI message.

8. The device according to claim 7, wherein The new SEI NAL unit is included in a picture unit (PU) containing the first SEI NAL unit, wherein the new SEI NAL unit is included immediately after the first SEI NAL unit, and wherein the first SEI NAL unit is removed from the output sub-bitstream after the scalable nesting SEI message is included in the new SEI NAL unit.

9. A non-transitory computer-readable storage medium having stored therein instructions that cause a processor to: performing conversion between a video comprising one or more layers and a bitstream of said video according to a rule, in, The rules state that during the sub-bitstream extraction process of removing Network Abstraction Layer (NAL) units from an input bitstream to generate an output sub-bitstream, In response to at least (1) the list of NAL unit header layer identifier values ​​in the OLS with the target output layer set (OLS) index does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the input bitstream and (2) the output sub-bitstream includes a first supplemental enhancement information (SEI) NAL unit, the first SEI NAL unit including a scalable embedded SEI message with sn_subpic_flag equal to 0 and sn_ols_flag equal to 0, generating a new SEI NAL unit, extracting a scalable nested SEI message from the scalable embedded SEI message, and including the scalable nested SEI message in the new SEI NAL unit as a non-scalable nested SEI message.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The new SEINAL unit is included in a picture unit (PU) containing the first SEINAL unit, wherein the new SEINAL unit is included immediately after the first SEINAL unit, and wherein the first SEI NAL unit is removed from the output sub-bitstream after the scalable nesting SEI message is included in the new SEINAL unit.

11. A method for storing a bitstream of a video, comprising: generating a bitstream of the video comprising one or more layers according to a rule, and storing the bitstream in a non-transitory computer-readable recording medium, The rule states that, during a sub-bitstream extraction process of removing a Network Abstraction Layer (NAL) unit from an input bitstream to generate an output sub-bitstream, In response to at least (1) the list of NAL unit header layer identifier values ​​in the target output layer set (OLS) with the OLS index does not include all values ​​of the NAL unit header layer identifier in all video codec layer (VCL) NAL units in the input bitstream and (2) the output sub-bitstream includes a first supplemental enhancement information (SEI) NAL unit, the first SEI NAL unit including a scalable embedding SEI message with sn_subpic_flag equal to 0 and sn_ols_flag equal to 0, A new SEI NAL unit is generated, and a scalable nested SEI message is extracted from the scalable embedded SEI message and included in the new SEI NAL unit as a non-scalable nested SEI message.

Images