Handling different nal types in video sub-bitstream extraction

Abstract

Examples of video encoding methods and apparatuses and video decoding methods and apparatuses are described. An example method of video processing includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes network abstraction layer (NAL) units for a plurality of video layers according to a rule; wherein the rule defines a sub-bitstream extraction process by which NAL units are removed from the bitstream to generate an output bitstream, and wherein the rule specifies that, in response to a list of NAL unit header layer identifier values in a target output layer set (OLS) that does not include all values of NAL unit headers in all video coding layer (VCL) NAL units in the bitstream input to the sub-bitstream extraction process, all supplemental enhancement information (SEI) NAL units containing non-scalably nested SEI messages with a particular payload type are removed.

Description

[0001] Cross-reference to related applications

[0002] This application is an application filed on June 8, 2021, under International Patent Application No. PCT / US2021 / 036471, which entered the Chinese national phase and claims priority to U.S. Provisional Patent Application No. 63 / 036,865, filed on June 9, 2020. The entire disclosure of the above applications is incorporated herein by reference and is part of the disclosure of this application. Technical Field

[0003] This patent document relates to image and video encoding and decoding. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to perform video encoding or decoding.

[0006] In one example aspect, a video processing method is disclosed. The method includes: performing a conversion between a video and a bitstream of the video, wherein the bitstream comprises Network Abstraction Layer (NAL) units for multiple video layers according to a rule; wherein the rule defines a sub-bitstream extraction process by which NAL units are removed from the bitstream to generate an output bitstream; and wherein the rule specifies, in response to a list of NAL unit header identifier values ​​in an OLS having a target Output Layer Set (OLS) index, the removal of all Supplemental Enhancement Information (SEI) NAL units containing non-scalable nested SEI messages of a specific payload type, wherein the target OLS index does not include all values ​​of NAL unit header identifiers in all Video Codec Layer (VCL) NAL units input to the sub-bitstream extraction process.

[0007] In another example, a different video processing method is disclosed. This method includes: performing a conversion between a video and a video bitstream, wherein the bitstream can be divided into one or more sub-bitstreams according to rules governing a sub-bitstream extraction process to generate an output bitstream, and wherein the rules specify whether and how to remove all Supplemental Enhancement Information (SEI) NAL units containing SEI messages applied to images or sub-images of VCL NAL units removed during the sub-bitstream extraction process, based on the type of the Video Codec Layer (VCL) Network Abstraction Layer (NAL) unit and the temporal identifier of the VCL NAL unit associated with it.

[0008] In another example, another video processing method is disclosed. The method includes: performing a conversion between a video and a video bitstream according to rules, wherein, according to the rules, the bitstream includes Network Abstraction Layer (NAL) units for multiple video layers; wherein, the rules define that the sub-bitstream extraction process for generating an output bitstream including an Output Layer Set (OLS) includes one or more operations selectively performed in response to the following conditions: (1) the list of NAL unit header identifier values ​​in the OLS does not include all values ​​of NAL unit header identifiers in all Video Codec Layer (VCL) NAL units in the bitstream, and (2) the output bitstream contains Supplemental Enhancement Information (SEI) NAL units, which contain scalable nested SEI messages.

[0009] In yet another example, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the methods described above.

[0010] In yet another example, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement the methods described above.

[0011] In yet another example, a computer-readable medium on which code is stored is disclosed. This code embodies one of the methods described herein in the form of processor-executable code.

[0012] These and other features are described in this document. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating a video codec system according to some implementations of the disclosed technology.

[0014] Figure 2 This is a block diagram of an example hardware platform used for video processing.

[0015] Figure 3 This is a flowchart of an example method for video processing.

[0016] Figure 4 This is a block diagram illustrating an example video codec system.

[0017] Figure 5 This is a block diagram illustrating an encoder according to some implementations of the disclosed technology.

[0018] Figure 6 This is a block diagram illustrating a decoder according to some implementations of the disclosed technology.

[0019] Figure 7A and 7B This is a flowchart of an example method for video processing based on some implementations of the disclosed technology.

[0020] Figure 8 This is a flowchart of an example method for video processing based on some implementations of the disclosed technology. Detailed Implementation

[0021] The use of chapter headings in this document is for ease of understanding and does not limit the applicability of the techniques and embodiments disclosed in each chapter to that chapter only. Furthermore, the use of H.266 terminology in some descriptions is merely for ease of understanding and not to limit the scope of the disclosed techniques. Therefore, the techniques described herein are also applicable to other video codec protocols and designs.

[0022] 1. Introduction

[0023] This document relates to video codec technology. Specifically, it concerns improvements to the general sub-bitstream extraction process, signaling notification of picture-level HRD parameters, and the inclusion of SEI messages in the SEI NAL unit. These ideas can be applied individually or in various combinations to any video codec standard or non-standard video codec that supports multi-layer video codecs, such as the Multi-Functional Video Codec (VVC) under development.

[0024] 2. Abbreviation

[0025] APS Adaptive Parameter Set

[0026] AU Access Unit

[0027] AUD access unit separator

[0028] AVC Advanced Video Encoding and Decoding

[0029] CLVS codec layer video sequence

[0030] CPB image buffer

[0031] CRA Cleaning Random Access

[0032] CTU encoding / decoding tree unit

[0033] CVS encoded video sequence

[0034] DCI decoding capability information

[0035] DPB decode image buffer

[0036] EOB bitstream ends

[0037] EOS sequence ends

[0038] GDR is being gradually decoded and refreshed.

[0039] HEVC High-Efficiency Video Encoding and Decoding

[0040] HRD Hypothetical Reference Decoder

[0041] IDR Instant Decoding and Refresh

[0042] ILP Interlayer Prediction

[0043] ILRP interlayer reference image

[0044] JEM Joint Exploration Model

[0045] LTRP Long-Term Reference Image

[0046] MCTS Motion Restraint Piece Set

[0047] NAL Network Abstraction Layer

[0048] OLS Output Layer Set

[0049] PH image header

[0050] PPS Image Parameter Set

[0051] PTL grade, level, grade

[0052] PU image unit

[0053] RAP Random Access Point

[0054] RBSP raw byte sequence payload

[0055] SEI Supplemental Enhancement Information

[0056] SPS sequence parameter set

[0057] STRP Short-Term Reference Image

[0058] SVC Scalable Video Codec

[0059] VCL video codec layer

[0060] VPS Video Parameter Set

[0061] VTMVVC test model

[0062] VUI Video Availability Information

[0063] VVC Multifunctional Video Encoding and Decoding

[0064] 3. Preliminary Discussion

[0065] Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed the H.261 and H.263 standards, while ISO / IEC developed the MPEG-1 and MPEG-4 Visual standards. The two organizations jointly developed the H.262 / MPEG-2 video standard, the H.264 / MPEG-4 Advanced Video Codec (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). JVET meetings are held concurrently quarterly, and the goal of new codec standards is to reduce the bitrate by 50% compared to HEVC. The new video codec standard was officially named Multifunctional Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. With ongoing efforts to standardize VVC, new codec technologies have been adopted into the VVC standard at each JVET meeting. The VVC working draft and test model VTM are updated after each meeting. The current goal of the VVC project is to achieve Technical Completion (FDIS) at the meeting in July 2020.

[0066] 3.1. Image precision variations within a sequence

[0067] In AVC and HEVC, the spatial precision of an image cannot be changed unless a new sequence with a new SPS begins with an IRAP image. VVC allows changes to the precision of images within a sequence at locations where IRAP images are not encoded; IRAP images are always intra-frame encoded and decoded. This feature is sometimes called Reference Image Resampling (RPR) because it requires resampling the reference image used for inter-frame prediction when the reference image has a different precision than the current image being decoded.

[0068] The scaling ratio is limited to greater than or equal to 1 / 2 (2x downsampling from the reference image to the current image) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between the reference and current images. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, similar to the case of motion-compensated interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process where the scaling ratio ranges from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the image width and height, as well as the left, right, top, and bottom scaling offsets specified for the reference and current images.

[0069] Other aspects of VVC designs that support this feature, unlike HEVC, include: i) signaling the picture precision and corresponding consistency window in the PPS instead of the SPS, while in the SPS the signaling indicates the maximum picture precision; ii) for a single-layer bitstream, each picture storage (the slot in the DPB used to store one decoded picture) occupies the buffer size required to store the decoded picture with the maximum picture precision.

[0070] 3.2. Scalable Video Codec (SVC) in General and VVC

[0071] Scalable video codec (SVC, sometimes also called scalability in video codec) refers to video codec using a base layer (BL) (sometimes called a reference layer (RL)) and one or more scalable enhancement layers (EL). In SVC, the base layer can carry video data with a base quality level. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise ratio (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as a BL, while the top layer can be used as an EL. Intermediate layers can be used as ELs or RLs, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of a layer below the intermediate layer (e.g., a base layer or any intermediate enhancement layer) and simultaneously used as an RL of one or more enhancement layers above the intermediate layer. Similarly, in the Multiview or 3D extension of the HEVC standard, multiple views may exist, and information from one view can be used to codec (e.g., encode or decode) information from another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0072] In SVC, the parameters used by the encoder or decoder are grouped into parameter sets based on the codec level in which they can be used (e.g., video level, sequence level, picture level, stripe level, etc.). For example, parameters that can be used by one or more codec video sequences at different layers in a bitstream can be included in the Video Parameter Set (VPS), and parameters that can be used by one or more pictures in a codec video sequence can be included in the Sequence Parameter Set (SPS). Similarly, parameters used by one or more stripes in a picture can be included in the Picture Parameter Set (PPS), and additional parameters specific to a single strip can be included in the stripe header. Likewise, indications of which parameter set(s) a particular layer uses at a given time can be provided at various codec levels.

[0073] Because of VVC's support for Reference Picture Resampling (RPR), support for multi-layered bitstreams can be designed without requiring any additional signaling notification processing level codec tools. For example, two layers in VVC with SD and HD resolutions can be supported because the upsampling required for spatial scalability can be achieved using only RPR upsampling filters. However, supporting scalability requires a higher level of syntax changes (compared to not supporting scalability at all). Scalability support is specified in VVC version 1. Unlike scalability support in any earlier video codec standards, including extensions to AVC and HEVC, VVC scalability is designed to be as friendly as possible to single-layer decoder designs. The decoding capability of multi-layered bitstreams is specified as if there were only a single layer in the bitstream. For example, decoding capabilities such as DPB size are specified in a way that is independent of the number of layers in the bitstream to be decoded. Essentially, decoders designed for single-layered bitstreams do not require many changes to decode multi-layered bitstreams. Compared to the multi-layered extensions of AVC and HEVC, the HLS aspect is significantly simplified at the expense of some flexibility. For example, IRAP AU requires a picture of every layer present in CVS.

[0074] 3.3. Parameter Set

[0075] AVC, HEVC, and VVC specify parameter sets. Parameter set types include SPS, PPS, APS, and VPS. All AVC, HEVC, and VVC versions support SPS and PPS. VPS was introduced with HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0076] SPS is designed to carry sequence-level header information, and PPS is designed to carry infrequently changing image-level header information. Using SPS and PPS eliminates the need to repeat infrequently changing information for each sequence or image, thus avoiding redundant signaling notifications. Furthermore, using SPS and PPS enables out-of-band transmission of critical header information, thereby not only avoiding redundant transmission but also improving error recovery capabilities.

[0077] The VPS was introduced to carry sequence-level header information common to all layers in a multi-layer bitstream.

[0078] The purpose of APS is to carry such image-level or stripe-level information, which requires a considerable number of bits for encoding and decoding, can be shared by multiple images, and can have many different variations in the sequence.

[0079] 3.4. General Sub-Bitstream Extraction Process

[0080] The latest VVC text, clause C.6, specifies a general sub-bitstream extraction procedure, as follows:

[0081] C.6 Sub-bit Stream Extraction Process

[0082] The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the highest target TemporalId value tIdTarget.

[0083] The output of this process is the sub-bit stream outBitstream.

[0084] The requirement for bitstream consistency of the input bitstream is that any output sub-bitstream that satisfies all of the following conditions should be a compliant bitstream:

[0085] – The output sub-bitstream is the output of the process specified in this clause, where, as input, the bitstream targetOlsIdx is equal to the index of the list of OLS specified by the VPS, and tIdTarget is equal to any value in the range of 0 to 6 (inclusive).

[0086] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0087] – The output sub-bitstream contains at least one VCL NAL unit whose TemporalId is equal to tIdTarget.

[0088] Note—The conforming bitstream contains one or more encoded / decoded stripe NAL units with TemporalId equal to 0, but does not necessarily contain encoded / decoded stripe NAL units with nuh_layer_id equal to 0.

[0089] The derivation of the output sub-bitstream OutBitstream is as follows:

[0090] 1. Set the bitstream outBitstream to be the same as the bitstream inBitstream.

[0091] 2. Remove all NAL units from outBitstream whose TemporalId is greater than tIdTarget.

[0092] 3. Remove all NAL units from outBitstream whose nal_unit_type is not equal to any of VPS_NUT, DCI_NUT, and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx].

[0093] 4. Remove from outBitstream all VCL NAL units that are true under all of the following conditions, as well as their associated non-VCL NAL units whose nal_unit_type is equal to PH_NUT, FD_NUT, SUFFIX_SEI_NUT, and PREFIX_SEI_NUT, where PayloadType is not equal to 0, 1, or 130:

[0094] –nal_unit_type equals TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equals GDR_NUT and the associated ph_recovery_poc_cnt is not equal to 0.

[0095] – For a value of j in the range of 0 to NumLayersInOls[targetOlsIdx] – 1 (inclusive), nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j].

[0096] –TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0097] 5. Remove all SEI NAL cells from outBitstream that contain scalable nested SEI messages with sn_ols_flag equal to 1 and no i value in the range from 0 to sn_num_olss_minus1 (inclusive), such that NestingOlsIdx[i] equals targetOlsIdx.

[0098] 6. When LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units of the bitstream, the following applies:

[0099] a. Remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI).

[0100] b. When general_same_pic_timing_in_all_ols_flag equals 0, remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages (PT) with payloadType equal to 1.

[0101] c. When outBitstream contains SEI NAL units and is applied to outBitstream (NestingOlsIdx[i] equals targetOlsIdx), and these units contain scalable nested SEI messages with sn_ols_flag equal to 1, the following applies:

[0102] – If general_same_pic_timing_in_all_ols_flag equals 0, then extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (BP), 1 (PT), or 130 (DUI) from the scalable nested SEI messages and include these SEI messages in outBitstream.

[0103] - Otherwise (general_same_pic_timing_in_all_ols_flag equals 1), extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI) from the scalable nested SEI messages, and include these SEI messages in outBitstream.

[0104] 4. The technical problem solved by the disclosed technical solution

[0105] The existing design of the general sub-bitstream extraction process and other related parts in the latest VVC text (JVET-R2001-vA / v10) has the following problems:

[0106] 1) The value of tIdTarget is said to be in the range of 0 to 6 (inclusive) provided that the output sub-bitstream is required to be a compliant bitstream. However, in many bitstreams, the highest TemporalId value is less than 6, and this value is specified by the syntax element vps_max_sublayers_minus1.

[0107] 2) When present, the Access Unit Delimiter (AUD) NAL unit can have any nuh_layer_id value. However, step 3 of the sub-bitstream extraction process removes the AUD NAL unit, whose nuh_layer_id value is not included in the list LayerIdInOls[targetOlsIdx].

[0108] 3) Some SEI NAL units contain scalable nested SEI messages with sn_ols_flag equal to 0, and the applicable layer indicated in the scalable nested SEI message does not include any layer in the target OLS; that is, no applicable layer's nuh_layer_id value is not included in the list layeridols[targetOlsIdx]. These SEI NAL units should also be removed.

[0109] 4) The condition in step 6, namely "when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream", has the following two problems.

[0110] a. This condition does not apply to cases where a DCI, VPS, AUD, or EOB NAL cell exists and the nuh_layer_id is not equal to any nuh_layer_id value of the VCLNAL cell.

[0111] b. The phrase “bitstream” is unclear because it involves two bitstreams in the context: inBitstream and outBitstream.

[0112] 5) Step 6.c extracts the scalable nested SEI message from the scalable nested SEI message where sn_ols_flag equals 1 and sn_subpic_flag equals 1, to generate the non-scalable nested SEI message. This scalable nested SEI message only applies to a specific sub-image and therefore should not be extracted.

[0113] 6) In step 6.c, when multiple scalable nested SEI messages are extracted from a SEI NAL unit seiNalUnitA as non-scalable nested SEI messages, they should still be contained in a single SEI NAL unit seiNalUnitB, and SEI NAL unit seiNalUnitB should be contained in the same PU that contains SEI NAL unit seiNalUnitA. However, this is not specified.

[0114] 7) Step 6.c should remove all SEI NAL cells from outBitstream from which some SEI messages have been extracted and included as non-scalable nested SEI messages. However, this is not specified.

[0115] 8) The lack of constraints means that when an SEI NAL cell contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL cell should not contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting). This results in removing SEI messages in step 4 involving more than just removing the SEI NAL cell.

[0116] 9) The flag `general_same_pic_timing_in_all_ols_flag` only specifies whether non-scalable nested PTSEI messages apply to all OLS. However, the information carried in DUI SEI messages is similar in purpose to that in PT SEI messages.

[0117] 10) When a VCL NAL unit is removed from the output bitstream if both of the following conditions are true: a) nal_unit_type equals TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equals GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0, and b) TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]], this step also removes associated SEI NAL units containing SEI messages other than BP, PT, or DUI SEI messages. However, some of those removed SEI messages may be applicable to OLS or layers containing the remaining images in the output bitstream.

[0118] 11) Sub-picture level information SEI messages (when present) apply to OLS as other HRD-related SEI messages (i.e., BP, PT, DUI SEI messages). However, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, the SEI NAL unit containing a non-scalable nested SEI message (i.e., sub-picture level information SEI message) with payloadType equal to 203 is not addressed during extraction.

[0119] 12) The final step of making a scalable nested SEI message a non-scalable nested SEI message has the following problems:

[0120] a. Excluding SEI messages where sn_ols_flag equals 0 and sn_subpic_flag equals 0.

[0121] b. It does not specify where the obtained non-scalable nested SEI messages should be placed in the output bitstream (in which SEI NAL unit, and where the SEI NAL unit should be).

[0122] 5. List of technical solutions and embodiments

[0123] To address the aforementioned and other issues, a methodology summarized below is presented. These items should be considered as examples for explaining general concepts, rather than interpreted in a narrow sense. Furthermore, these items can be applied individually or in combination in any way.

[0124] 1) To solve problem 1, the condition that the output sub-bitstream must conform to the bitstream is specified, so that the value of tIdTarget is specified in the range of 0 to vps_max_sublayers_minus1 (inclusive).

[0125] a. Alternatively, specify the conditions that require the output sub-bitstream to be a compliant bitstream, such that when there are more than one layer in the input bitstream, the value of tIdTarget is specified in the range of 0 to vps_max_sublayers_minus1 (inclusive), and when there is only one layer in the input bitstream, the value of tIdTarget is specified in the range of 0 to sps_max_sublayers_minus1 (inclusive).

[0126] 2) To address problem 2, a generic sub-bitstream extraction procedure is defined to process AUD NAL units in the same way as NAL units with nal_unit_type equal to VPS_NUT, DCI_NUT, or EOB_NUT. In other words, based on the nuh_layer_id value, no AUD NAL units are removed from the output bitstream outBitstream.

[0127] 3) To address problem 3, a general sub-bitstream extraction procedure is specified, which removes the output bitstream outBitstream, containing SEI NAL cells of scalable nested SEI messages with sn_ols_flag equal to 0, where the applicable layer indicated in the scalable nested SEI message does not include any layer in the target OLS.

[0128] a. In one example, it is specified that all SEINAL cells containing scalable nested SEI messages with sn_ols_flag equal to 0 are removed from outBitstream, and for i in the range of nestingNumLayers – 1 (inclusive), there is no value in the list LayerIdInOls[targetOlsIdx] in the list nestingLayerId[i].

[0129] 4) To solve problem 4, the condition "when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream" is changed to "when the list LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all VCL NAL units in the bitstream inBitstream".

[0130] 5) To solve problem 5, a general sub-bit stream extraction process is defined, which extracts scalable nested SEI messages only from scalable nested SEI messages where sn_ols_flag is equal to 1 and sn_subpic_flag is equal to 0, in order to generate non-scalable nested SEI messages.

[0131] 6) To address problem 6, a general sub-bitstream extraction process is defined such that when multiple scalable nested SEI messages are extracted from a SEI NAL unit seiNalUnitA as non-scalable nested SEI messages, they are still included in a SEI NAL unit seiNalUnitB in the output bitstream outBitstream, and the SEI NAL unit seiNalUnitB is included in the PU containing the SEI NAL unit seiNalUnitA.

[0132] 7) To solve problem 7, a general sub-bitstream extraction process is specified, which removes all SEI messages extracted from the output bitstream outBitstream and includes them as SEI NAL units containing non-scalable nested SEI messages.

[0133] a. Alternatively, remove the SEI NAL unit from outBitstream when the scalable nested SEI messages in such a SEI NAL unit apply only to the target OLS (i.e., the targetOlsIdx OLS specified by the VPS).

[0134] b. Alternatively, remove the SEI NAL unit from outBitstream when there is no OLS containing all layers included in the list LayerIdInOls[targetOlsIdx] except for the target OLS in the OLS to which the scalable nested SEI message is applied in this SEI NAL unit.

[0135] 8) To address problem 8, add a constraint that when an SEI NAL cell contains an SEI message with payloadType equal to 0, 1, or 130, the SEI NAL cell should not contain an SEI message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nesting).

[0136] 9) To address issue 9, the flag general_same_pic_timing_in_all_ols_flag specifies whether non-scalable nested PT and DUI SEI messages apply to all OLS.

[0137] a. Alternatively, the flag general_same_pic_timing_in_all_ols_flag specifies whether non-scalable nested BP, PT, and DUI SEI messages apply to all OLS.

[0138] i. In one example, the flag general_same_pic_timing_in_all_ols_flag is renamed to the flag general_same_pic_level_hrd_info_in_all_ols_flag, which specifies whether non-scalable nested BP, PT, and DUI SEI messages are applied to all OLS.

[0139] b. Alternatively, add a new flag, for example, named general_same_dui_in_all_ols_flag, to specify whether non-scalable nested DUI SEI messages apply to all OLS.

[0140] c. Alternatively, add a new flag, for example, named general_same_bp_in_all_ols_flag, to specify whether non-scalable nested BP SEI messages apply to all OLS.

[0141] 10) To address issue 10, in one example, during the general sub-bitstream extraction process, when removing VCL NAL units from the output bitstream where both of the following conditions are true: a) nal_unit_type equals TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equals GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0, and b) TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]], instead of removing the associated SEI NAL units containing SEI messages other than BP, PT, or DUI SEI messages, all SEI NAL units containing SEI messages are removed, and these SEI messages apply only to one or more pictures or subpictures where all VCL NAL units are being removed.

[0142] a. In one example, or when removing such a VCL NAL cell, the associated SEI NAL cell is retained in the output bitstream.

[0143] 11) To address problem 11, in one example, during the general sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units of the bitstream, all SEI NAL units containing non-scalable nested SEI messages (i.e., sub-picture level information SEI messages) with payloadType equal to 203 are also removed.

[0144] 12) To address problem 12, in one example, during the general sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units of the bitstream, and outBitstream contains SEI NAL units seiNalUnitA containing scalable nested SEI messages with sn_subpic_flag equal to 0, which are applied to OLS (when sn_ols_flag equals 1) or layers with the same layer set as those in outBitstream (when sn_ols_flag equals 0), one or more of the following operations are performed:

[0145] a. Generate a new SEI NAL unit seiNalUnitB.

[0146] b. Include seiNalUnitB in the PU that contains seiNalUnitA.

[0147] c. Include seiNalUnitB in the PU that contains seiNalUnitA, immediately following seiNalUnitA.

[0148] d. Extract the scalable nested SEI messages from the scalable nested SEI messages and include them directly in seiNalUnitB (as non-scalable nested SEI messages).

[0149] e. Remove seiNalUnitA from outBitstream.

[0150] 13) In one example, during the general sub-bitstream extraction process, when LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all NAL units in the bitstream, and outBitstream contains SEINAL unit seiNalUnitA, which contains scalable nested SEI messages, seiNalUnitA is retained in the output bitstream.

[0151] 6. Example

[0152] The following are some example embodiments of some aspects of the invention outlined in Section 5 above, which can be applied to the VVC specification. The modified text is based on the latest VVC text in JVET-R2001-vA / v10. The most relevant parts that have been added or modified are highlighted in bold and italics, and some of the deleted parts are marked with double brackets (e.g., [[a]] indicates the deletion of the character "a"). There may be other changes that are essentially editorial and therefore are not highlighted.

[0153] 6.1. First Embodiment

[0154] This example applies to items 1, 2, 3, 3.a, 4, 5, 6, 7.b, and 8.

[0155] C.6 General Sub-Bitstream Extraction Process

[0156] The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the highest target TemporalId value tIdTarget.

[0157] The output of this process is the sub-bit stream outBitstream.

[0158] The requirement for bitstream consistency of the input bitstream is that any output sub-bitstream that satisfies all of the following conditions should be a compliant bitstream:

[0159] – The output sub-bitstream is the output of the process specified in this clause, where, as input, targetOlsIdx in the bitstream is equal to the index of the list of OLS specified by the VPS, and tIdTarget is equal to 0 to 1. Any value within the range (including endpoints).

[0160] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0161] – The output sub-bitstream contains at least one VCL NAL unit whose TemporalId is equal to tIdTarget.

[0162] Note – A valid bitstream contains one or more encoded / decoded stripe NAL units with TemporalId equal to 0, but does not necessarily contain encoded / decoded stripe NAL units with nuh_layer_id equal to 0.

[0163] OutBitstream is Derivation:

[0164] 1. Set the bitstream outBitstream to be the same as the bitstream inBitstream.

[0165] 2. Remove all NAL units from outBitstream whose TemporalId is greater than tIdTarget.

[0166] 3. Remove nal_unit_type values ​​that are not equal to DCI_NUT, VPS_NUT, from outBitstream. And any of the EOB_NUTs, and nuh_layer_id is not included in any of the NAL units in the list LayerIdInOls[targetOlsIdx].

[0167] 4. Remove all VCL NAL units from outBitstream that are true for all of the following conditions, and Associated non-VCL NAL units, nal_unit_type equals PH_NUT FD_NUT, SUFFIX_SEI_NUT PREFIX_SEI_NUT Not equal to 0 (BP), 1 (PT), or 130 (DUI) :

[0168] –nal_unit_type equals TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equals GDR_NUT and the associated ph_recovery_poc_cnt is not equal to 0.

[0169] –[[For values ​​of j in the range of 0 to NumLayersInOls[ targetOlsIdx ] – 1 (inclusive), nuh_layer_id equals LayerIdInOls[ targetOlsIdx ][ j ]]]

[0170] –TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0171] 5. Remove all SEI NAL cells from outBitstream that contain scalable nested SEI messages with sn_ols_flag equal to 1 and no i value in the range from 0 to sn_num_olss_minus1 (inclusive), such that NestingOlsIdx[i] equals targetOlsIdx.

[0172]

[0173] 7. When LayerIdInOls[targetOlsIdx] does not include the bitstream. All When all values ​​of nuh_layer_id are in the NAL unit, the following are... application:

[0174] a. Remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (BP) or 130 (DUI).

[0175] b. When general_same_pic_timing_in_all_ols_flag equals 0, remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages with payloadType equal to 1 (PT).

[0176] c. When outBitstream contains SEI NAL units, those SEI NAL units contain sn_ols_flag equal to 1. A scalable nested SEI message, which is applied to the targetOlsIdx-th OLS (i.e., there is at least one i value in the range from 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] equals targetOlsIdx), the following application:

[0177] i. For each scalable nested BP or DUI SEI message in such SEI NAL unit seiNalUnitA, In outBitstream .

[0178] ii. When general_same_pic_timing_in_all_ols_flag equals 0, for each scalable nested PT SEI message in such SEI NAL unit seiNalUnitA, In outBitstream .

[0179]

[0180] D.2.2 General SEI Load Semantics ...

[0182] The requirement for bitstream consistency applies to SEI messages contained in SEI NAL units as follows:

[0183] – When a SEI NAL cell contains a non-scalable nested BP SEI message, a non-scalable nested PT SEI message, or a non-scalable nested DUI SEI message, the SEI NAL cell must not contain any other SEI message whose payloadType is not equal to 0 (BP), 1 (PT), or 130 (DUI).

[0184] – When a SEI NAL cell contains a scalable nested BP SEI message, a scalable nested PT SEI message, or a scalable nested DUI SEI message, the SEI NAL cell must not contain any other SEI message whose payloadType is not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nested).

[0185] ...

[0187] 6.2. Second Embodiment

[0188] This embodiment focuses on items 10 to 12 (including end values), and the changes in text compared to the above embodiment 1 are highlighted.

[0189] C.6 General Sub-Bitstream Extraction Process

[0190] The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the highest target TemporalId value tIdTarget.

[0191] The output of this process is the sub-bit stream outBitstream.

[0192]

[0193] The requirement for bitstream consistency of the input bitstream is that any output sub-bitstream that satisfies all of the following conditions should be a compliant bitstream:

[0194] – The output sub-bitstream is the output of the process specified in this clause, wherein, as input, targetOlsIdx in the bitstream is equal to the index of the list of OLS specified by the VPS, and tIdTarget is equal to any value in the range of 0 to vps_max_sublayers_minus1 (inclusive).

[0195] – The output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id is equal to each nuh_layer_id value in LayerIdInOls[targetOlsIdx].

[0196] – The output sub-bitstream contains at least one VCL NAL unit whose TemporalId is equal to tIdTarget.

[0197] Note – A valid bitstream contains one or more encoded / decoded stripe NAL units with TemporalId equal to 0, but does not necessarily contain encoded / decoded stripe NAL units with nuh_layer_id equal to 0.

[0198] The output sub-bitstream, OutBitstream, is derived by applying the following ordered steps:

[0199] 1. Set the bitstream outBitstream to be the same as the bitstream inBitstream.

[0200] 2. Remove all NAL units from outBitstream whose TemporalId is greater than tIdTarget.

[0201]

[0202] 4. Remove all VCL NAL units from outBitstream that are true for all of the following conditions, [[and also remove these VCL NAL units from outBitstream]]. Associated non-VCL NAL units whose nal_unit_type is equal to PH_NUT or FD_NUT[[, or whose nal_unit_type is equal to SUFFIX_SEI_NUT or PREFIX_SEI_NUT and contains SEI messages whose payloadType is not equal to 0 (BP), 1 (PT), 130 (DUI), or 133 (scalable nested)]]:

[0203] –nal_unit_type equals TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or nal_unit_type equals GDR_NUT and is associated with ph_recovery_poc_cnt [[Not equal to]]0.

[0204] –TemporalId is greater than or equal to NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

[0205]

[0206] – Remove all SEI NAL cells from outBitstream that contain a scalable nested SEI message with sn_ols_flag equal to 1 and no i value in the range from 0 to sn_num_olss_minus1 (inclusive), such that NestingOlsIdx[i] equals targetOlsIdx.

[0207] – Remove all SEI NAL cells from outBitstream that contain scalable nested SEI messages with sn_ols_flag equal to 0, and for i in the range of nestingNumLayers – 1 (inclusive), there is no value in the list nestingLayerId[i] from the list LayerIdInOls[targetOlsIdx].

[0208]

[0209] 6. When LayerIdInOls[targetOlsIdx] does not include all values ​​of nuh_layer_id in all VCL NAL units of the bitstream inBitstream, the following apply in the order listed:

[0210] a. Remove all packets containing payloadType equal to 0 (BP) from outBitstream. [[or 130(DUI)]] is the SEI NAL unit of a non-scalable nested SEI message.

[0211] b. When general_same_pic_timing_in_all_ols_flag equals 0, remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages (PT) with payloadType equal to 1.

[0212]

[0213] When outBitstream contains an SEI NAL unit, the SEI NAL unit contains a scalable nested SEI message with sn_ols_flag equal to 1 and sn_subpic_flag equal to 0. This scalable nested SEI message is applied to the targetOlsIdx-th OLS (i.e., there is at least one i value in the range from 0 to sn_num_olss_minus1 (inclusive) such that NestingOlsIdx[i] equals targetOlsIdx), and is applied in the following order:

[0214] i. For each scalable nested BP or DUI SEI message in such SEI NAL unit seiNalUnitA, generate a non-scalable nested SEI message with the same payloadType and SEI payload, and include it in the SEI NAL unit in the PU containing seiNalUnitA in outBitstream.

[0215] ii. When general_same_pic_timing_in_all_ols_flag equals 0, for each scalable nested PT SEI message in such SEI NAL unit seiNalUnitA, generate a non-scalable nested SEI message with the same SEI payload and include it in the SEI NAL unit in the PU containing seiNalUnitA in outBitstream.

[0216] iii. When multiple SEI messages contained in a particular SEI NAL unit seiNalUnitA are made into non-scalable nested SEI messages, the non-scalable nested SEI messages are thus included in a single SEI NAL unit.

[0217] iv. When there is no OLS containing all layers included in the list LayerIdInOls[targetOlsIdx], except for the target OLS in the OLS to which the scalable nested SEI message applies in this SEI NAL unit seiNalUnitA, remove the SEI NAL unit seiNalUnitA from outBitstream.

[0218] Figure 1 This is a block diagram of an example video processing system 1900 that can implement the various techniques disclosed herein. Various implementations may include some or all of the components in system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0219] System 1900 may include a codec component 1904 capable of implementing the various codec or encoding methods described in this document. Codec component 1904 can reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1904 can be stored or transmitted via connected communication, as represented by component 1906. The stored or communicated bitstream (or codec) representation of the video received at input 1902 can be used by component 1908 to generate pixel values ​​or displayable video that are sent to display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation will be inverted by the decoder to retrieve the result of the codec.

[0220] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0221] Figure 2This is a block diagram of a video processing apparatus 3600. Apparatus 3600 can be used to implement one or more of the methods described herein. Apparatus 3600 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The processors 3602(s) can be configured to implement one or more methods described herein. The memories 3604(s) can be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 3606 can be used to implement some of the techniques described herein in hardware circuitry.

[0222] Figure 4 This is a block diagram illustrating an example video codec system 100 that can utilize the techniques disclosed herein.

[0223] like Figure 4 As shown, the video encoding / decoding system 100 may include a source device 110 and a target device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The target device 120 can decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device.

[0224] The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0225] Video source 112 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems that generate video data, or combinations of these sources. Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec images and associated data. A codec image is a codec representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax elements. I / O interface 116 includes a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to target device 120 via network 130a through I / O interface 116. Encoded video data may also be stored on storage medium / server 130b for access by target device 120.

[0226] The target device 120 may include an I / O interface 126, a video decoder 124, and a display device 122.

[0227] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may acquire encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with target device 120 or may be external to target device 120 configured to connect to an external display device.

[0228] The video encoder 114 and the video decoder 124 can operate according to video compression standards such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other current and / or other standards.

[0229] Figure 5 This is a block diagram illustrating an example of a video encoder 200, which may be... Figure 4 The video encoder 114 in the system 100 shown in the figure.

[0230] The video encoder 200 can be configured to perform any or all of the techniques disclosed herein. Figure 5 In the example, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0231] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 (which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra-frame prediction unit 206), a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0232] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0233] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, can be highly integrated, but for interpretive purposes... Figure 5 The examples are shown separately.

[0234] The segmentation unit 201 can segment an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0235] The mode selection unit 203 can, for example, select one of the intra-frame or inter-frame encoding / decoding modes based on the error result, and provide the obtained intra-frame or inter-frame encoded / decoded blocks to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the encoded / decoded blocks for use as reference images. In some examples, the mode selection unit 203 can select a combined intra-frame and inter-frame prediction (CIIP) mode, where the prediction is based on the inter-frame prediction signal and the intra-frame prediction signal. The mode selection unit 203 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the blocks in the inter-frame prediction case.

[0236] To perform inter-frame prediction for the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 can determine the predicted video block for the current video block based on the motion information of the images from buffer 213 (rather than the images associated with the current video block) and decoded samples.

[0237] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, for example, the different operations performed depend on whether the current video block is in an I-strip, a P-strip, or a B-strip.

[0238] In some examples, motion estimation unit 204 can perform unidirectional prediction of the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 204 can then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0239] In other examples, motion estimation unit 204 can perform bidirectional prediction of the current video block. Motion estimation unit 204 can search for a reference video block for the current video block in the reference images of list 0 and can also search for another reference video block for the current video block in the reference images of list 1. Motion estimation unit 204 can then generate a reference index indicating that the reference images in list 0 or list 1 contain the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 204 can output the reference index and the motion vector of the current video block as the motion information of the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0240] In some examples, the motion estimation unit 204 can output the complete set of motion information for the decoder's decoding process.

[0241] In some examples, motion estimation unit 204 may not output the complete set of motion information for the current video. Instead, motion estimation unit 204 may signal the motion information of the current video block by referencing the motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of adjacent video blocks.

[0242] In one example, the motion estimation unit 204 may indicate in the syntax structure associated with the current video block that the current video block has the same motion information value as another video block.

[0243] In another example, motion estimation unit 204 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicating video block. Video decoder 300 can use the motion vector of the indicating video block and the motion vector difference to determine the motion vector of the current video block.

[0244] As discussed above, the video encoder 200 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge pattern signaling notification.

[0245] Intra-prediction unit 206 can perform intra-prediction on the current video block. When intra-prediction unit 206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0246] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0247] In other examples, such as in skip mode, residual data for the current video block may not exist, and the residual generation unit 207 may not perform a subtraction operation.

[0248] The transform processing unit 208 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0249] After the transform processing unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0250] The inverse quantization unit 210 and the inverse transform unit 211 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to produce a reconstructed video block associated with the current block for storage in the buffer 213.

[0251] After the video block is reconstructed by the reconstruction unit 212, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0252] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, it can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream including the entropy encoded data.

[0253] Figure 6 This is a block diagram illustrating an example of a video decoder 300, which may be... Figure 4 The video decoder 114 in the system 100 shown in the figure.

[0254] The video decoder 300 can be configured to perform any or all of the techniques disclosed herein. Figure 6 In the example, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0255] exist Figure 6 In the example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform operations related to the video encoder 200 ( Figure 5 The decoding process is the overall inversion of the encoding process described.

[0256] Entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy-encoded video, and based on the entropy-encoded video data, motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. Motion compensation unit 302 can determine such information, for example, by performing AMVP and merge modes.

[0257] The motion compensation unit 302 can generate motion compensation blocks, possibly based on interpolation filters. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.

[0258] The motion compensation unit 302 can use the interpolation filter used by the video encoder 200 during the encoding of the video block to calculate the interpolation values ​​of a sub-integer number of pixels of the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0259] The motion compensation unit 302 can use some syntactic information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.

[0260] Intra-prediction unit 303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 303 applies an inverse transform.

[0261] The reconstruction unit 306 can sum the residual blocks using the corresponding prediction blocks generated by the motion compensation unit 202 or the intra-frame prediction unit 303 to form a decoded block. As desired, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation / intra-frame prediction and also produces the decoded video for presentation on the display device.

[0262] The following provides a list of preferred solutions for some embodiments.

[0263] The first set of solutions is presented below. The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Items 1-9).

[0264] 1. A video processing method (e.g., Figure 3 Method 600 in the middle includes: performing (602) a conversion between a video and a video codec representation comprising one or more video layers, the one or more video layers comprising one or more video pictures, wherein the codec representation conforms to format rules associated with the extraction of sub-bitstreams from the codec representation.

[0265] 2. The method according to Solution 1 further includes: extracting sub-bit streams from the codec representation according to format rules.

[0266] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 1).

[0267] 3. The method according to any one of solutions 1-2, wherein, during the extraction of sub-bit streams, the target id for extraction is allowed to be in the range of 0 to the value of the syntax field indicated in the set of video parameters for the codec representation.

[0268] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 2).

[0269] 4. The method according to any one of solutions 1-3, wherein the sub-bitstream is extracted without removing the Access Unit Delimiter Network Abstraction Layer (AUD NAL) from the output bitstream based on the layer ID.

[0270] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 3).

[0271] 5. The method according to any one of solutions 1-4, wherein sub-bit streams are extracted by selectively removing network abstraction layer units, which include scalable nested supplementary enhancement information messages that are not applicable to the output layer being extracted.

[0272] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 5).

[0273] 6. The method according to any one of solutions 1-5, wherein the sub-bitstream is extracted by constraining the extraction to generate a non-scalable nested SEI from the scalable nested supplementary enhancement information (SEI) message by setting a flag for the output layer set and a flag for disabling sub-images.

[0274] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 6).

[0275] 7. The method according to any one of solutions 1-6, wherein the sub-bit stream is extracted according to a rule that specifies the extraction of multiple scalable nested Supplemental Enhancement Information (SEI) messages from a single SEI network abstraction layer unit.

[0276] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 7).

[0277] 8. The method according to any one of solutions 1-7, wherein sub-bit streams are extracted according to the rule of removing all Supplemental Enhanced Information Network Abstraction Layer (SEI NAL) units from the codec representation, and some SEI messages have been extracted from these SEI NAL units and included as non-scalable nested SEI messages.

[0278] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Items 8-13).

[0279] 10. A video processing method, comprising: performing a conversion between a video and a codec representation of the video, wherein the codec representation includes one or more sub-bitstreams; wherein the conversion is performed according to rules defining the relationship between a sub-bitstream extraction process and one or more syntax elements of the codec representation.

[0280] 11. The method according to Solution 10, wherein the rule specifies that all Supplemental Enhancement Information (SEI) Network Abstraction Layer (NAL) units containing SEI messages applied to the removal of a picture or subpicture of a video codec layer shall be removed if one or more of the following two conditions are true: (1) the type of the NAL unit is a specific type, or (2) the temporal identifier of the video codec layer satisfies a specific condition.

[0281] 12. The method according to any one of solutions 10-11, wherein the rule specifies that if the layer identifier in the target output layer does not include all values ​​of the layer identifier in the codec representation, the process also removes all SEI NAL units containing non-scalable nested SEI messages with payloadType equal to 203.

[0282] 13. The method according to any one of solutions 1-12, wherein performing the conversion includes encoding the video into a codec representation.

[0283] 14. The method according to any one of solutions 1-12, wherein performing the conversion includes parsing and decoding the codec representation to generate a video.

[0284] 15. A video decoding apparatus, comprising a processor configured to implement one or more of the methods described in solutions 1 to 14.

[0285] 16. A video encoding apparatus, comprising a processor configured to implement one or more of the methods described in solutions 1 to 14.

[0286] 17. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method described in any one of solutions 1 to 14.

[0287] 18. The methods, apparatus or systems described in this document.

[0288] The second set of solutions shows example implementations of the techniques discussed in the previous chapter (e.g., items 10 and 11).

[0289] 1. A video data processing method (e.g., such as...) Figure 7A The method shown (700) includes: performing (702) a conversion between video and video bitstream, wherein the bitstream comprises Network Abstraction Layer (NAL) units for multiple video layers according to a rule; wherein the rule defines a sub-bitstream extraction process by which NAL units are removed from the bitstream to generate an output bitstream, and wherein the rule specifies, in response to a list of NAL unit header identifier values ​​in the output layer set (OLS) with a target OLS index, the removal of all Supplemental Enhancement Information (SEI) NAL units containing non-scalable nested SEI messages of a specific payload type, wherein the target OLS index does not include all values ​​of NAL unit header identifiers in all Video Codec Layer (VCL) NAL units input to the sub-bitstream extraction process.

[0290] 2. The method according to Solution 1, wherein the non-scalable nested SEI message with a specific payload type corresponds to a sub-image level information SEI message.

[0291] 3. The method described in Solution 1, wherein the specific load type is equal to 203.

[0292] 4. The method according to Solution 1, wherein the non-scalable nested SEI message with a specific payload type corresponds to a decoding unit information message.

[0293] 5. The method described in Solution 1, wherein the specific load type is equal to 130.

[0294] 6. A video data processing method (e.g., Figure 7B The method 710 shown includes: performing a conversion between a video and a video bitstream, wherein the bitstream can be divided into one or more sub-bitstreams according to rules governing a sub-bitstream extraction process to generate an output bitstream, and wherein the rules specify whether and how to remove all supplementary enhancement information SEI network abstraction layer NAL units containing images or sub-images of images or sub-images that are removed during the sub-bitstream extraction process, based on the type of the video codec layer (VCL) network abstraction layer (NAL) unit and the temporal identifier of the video codec layer associated with the VCL NAL unit.

[0295] 7. The method according to Solution 6, wherein the rule specifies the removal of all SEI NAL units in response to: (1) the type of VCL NAL unit equal to TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT or GDR_NUT, and (2) the temporal identifier of the video codec layer being greater than or equal to the number of sublayers in the video codec layer in the output layer set.

[0296] 8. The method according to Solution 7, wherein, for a VCL NAL unit of type equal to GDR_NUT, the syntax field indicating the recovery point of the decoded picture in the output order has a value greater than 0.

[0297] 9. The method according to Solution 6, wherein the rule specifies that SEI NAL cells are retained in the output bitstream.

[0298] 10. The method according to any one of solutions 1 to 9, wherein the conversion includes encoding the video into a bitstream.

[0299] 11. The method according to any one of solutions 1 to 9, wherein the conversion includes decoding video from a bitstream.

[0300] 12. The method according to any one of solutions 1 to 9, wherein the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0301] 13. A video processing apparatus comprising a processor configured to implement the method described in any one or more of solutions 1 to 12.

[0302] 14. A method for storing a bitstream of video, comprising the method according to any one of solutions 1 to 12, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0303] 15. A computer-readable medium storing program code that, when executed, causes a processor to implement the method according to any one or more of solutions 1 to 12.

[0304] 16. A computer-readable medium storing a encoded or decoded representation or bitstream generated according to any of the methods described above.

[0305] 17. A video processing apparatus for storing bit streams, wherein the video processing apparatus is configured to implement the method according to any one or more of solutions 1 to 12.

[0306] The third set of solutions shows example implementations of the techniques discussed in the previous chapter (e.g., items 12 and 13).

[0307] 1. A video data processing method (e.g., Figure 8 The method shown (800) includes: performing (802) a conversion between video and video bitstream according to a rule, wherein, according to the rule, the bitstream includes Network Abstraction Layer (NAL) units for multiple video layers; wherein, the rule defines the sub-bitstream extraction process for generating an output bitstream including an Output Layer Set (OLS) includes one or more operations selectively performed in response to the following conditions: (1) the list of NAL unit header identifier values ​​in the OLS does not include all values ​​of NAL unit header identifiers in all Video Codec Layer (VCL) NAL units in the bitstream, and (2) the output bitstream contains Supplemental Enhancement Information (SEI) NAL units, which contain scalable nested SEI messages.

[0308] 2. The method according to Solution 1, wherein the scalable nested SEI message is associated with a first flag equal to a specific value, the specific value being applied to a specific OLS or a specific layer having the same set of layers as in the output bitstream.

[0309] 3. According to the method of Solution 2, wherein a first flag equal to 0 indicates that a scalable nested SEI message applied to a specific OLS or a specific layer is applied to all sub-images of the specific OLS or a specific layer.

[0310] 4. The method according to Solution 1, wherein the scalable nested SEI message is associated with a second flag indicating whether the scalable nested SEI message is applied to a specific OLS or a specific layer.

[0311] 5. The method according to any one of solutions 1-4, wherein one or more operations include generating additional SEI NAL units.

[0312] 6. The method according to Solution 5, wherein one or more operations include including an additional SEI NAL unit in the image unit containing the SEI NAL unit.

[0313] 7. The method according to Solution 6, wherein the additional SEI NAL unit immediately follows the SEI NAL unit.

[0314] 8. The method according to any one of solutions 6, wherein one or more operations further include extracting the scalable nested SEI message from the scalable nested SEI message and including the scalable nested SEI message in an additional SEINAL unit.

[0315] 9. The method according to Solution 1, wherein one or more operations include removing SEI NAL units from the output bitstream.

[0316] 10. The method according to Solution 1, wherein one or more operations include holding the SEI NAL cell in the output bit stream.

[0317] 11. The method according to any one of solutions 1 to 10, wherein the conversion includes encoding the video into a bitstream.

[0318] 12. The method according to any one of solutions 1 to 10, wherein the conversion includes decoding video from a bitstream.

[0319] 13. The method according to any one of solutions 1 to 10, wherein the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0320] 14. A video processing apparatus comprising a processor configured to implement the method described in any one or more of solutions 1 to 13.

[0321] 15. A method for storing a bitstream of video, comprising the method according to any one of solutions 1 to 13, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0322] 16. A computer-readable medium storing program code that, when executed, causes a processor to implement the method described in any one or more of solutions 1 to 13.

[0323] 17. A computer-readable medium storing a bit stream generated according to any of the methods described above.

[0324] 18. A video processing apparatus for storing bit streams, wherein the video processing apparatus is configured to implement the method described in any one or more of solutions 1 to 13.

[0325] The disclosures and other schemes, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits or in computer software, firmware, or hardware, containing the structures disclosed in this document and their equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products encoded on a computer-readable medium, i.e., one or more computer program instruction modules for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a complex influencing machine-readable propagating signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagating signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.

[0326] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0327] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be performed by special-purpose logic circuitry (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), and the apparatus can be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs).

[0328] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical disc) for storing data, or operatively coupled to receive data from or transfer data to a mass storage device (e.g., magneto-optical, magneto-optical, or optical disc), or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0329] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular art. In this patent document, certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in various suitable sub-combinations. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in the same manner, in certain circumstances one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0330] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all the operations shown, in order to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0331] Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and shown in this patent document.

Claims

1. A video data processing method, comprising: According to the rules, a conversion between the video (including one or more layers) and the bitstream of the video is performed. The rules specify a sub-bitstream extraction process, in which network abstraction layer (NAL) units are removed from the input bitstream to generate an output sub-bitstream. The rule further stipulates that when the list of NAL unit header identifier values ​​in the OLS with the target output layer set OLS index does not include all values ​​of NAL unit header identifiers in all video codec layer VCL NAL units in the input bitstream, all supplementary enhancement information SEI NAL units containing non-scalable nested SEI messages with a specific payload type are removed. The non-scalable nested SEI message having the specific payload type includes a non-scalable nested SEI message having a first specific payload type corresponding to a sub-image level information SEI message.

2. The method according to claim 1, wherein, The first specific load type is equal to 203.

3. The method according to claim 1, wherein, The non-scalable nested SEI message having the specific payload type also includes a non-scalable nested SEI message having a second specific payload type corresponding to the decoding unit information message.

4. The method according to claim 3, wherein, The second specific load type is equal to 130.

5. The method according to claim 1, wherein, The rule also specifies the removal of NAL units that meet the first set of conditions, wherein the first set of conditions includes: 1) the NAL unit has a nuh_layer_id that is not included in the list of NAL unit header identifier values ​​in the OLS that has the target output layer set OLS index; 2) the NAL unit is not a DCINAL unit, a VPS NAL unit, an AUD NAL unit, or an EOB NAL unit; and 3) the NAL unit is not a NAL unit that contains a specific non-scalable nested SEI message.

6. The method according to claim 1, wherein, The rule also specifies the removal of all VCLNAL units that satisfy the second set of conditions, wherein the second set of conditions includes: nal_unit_type equal to TRAIL_NUT, STSA_NUT, RADL_NUT or RASL_NUT, or nal_unit_type equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0.

7. The method according to any one of claims 1-6, wherein, The conversion includes encoding the video into the bitstream.

8. The method according to any one of claims 1-6, wherein, The conversion includes decoding the video from the bitstream.

9. A video data processing apparatus, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, it causes the processor to: According to the rules, a conversion between the video (including one or more layers) and the bitstream of the video is performed. The rules specify a sub-bitstream extraction process, in which network abstraction layer (NAL) units are removed from the input bitstream to generate an output sub-bitstream. The rule further stipulates that when the list of NAL unit header identifier values ​​in the OLS with the target output layer set OLS index does not include all values ​​of NAL unit header identifiers in all video codec layer VCL NAL units in the input bitstream, all supplementary enhancement information SEI NAL units containing non-scalable nested SEI messages with a specific payload type are removed. The non-scalable nested SEI message having the specific payload type includes a non-scalable nested SEI message having a first specific payload type corresponding to a sub-image level information SEI message.

10. The apparatus according to claim 9, wherein, The first specific payload type is equal to 203, wherein the non-scalable nested SEI message having the specific payload type further includes a non-scalable nested SEI message having a second specific payload type corresponding to the decoding unit information message, and wherein the second specific payload type is equal to 130.

11. The apparatus according to claim 9, wherein, The rule also specifies the removal of NAL units that meet the first set of conditions, wherein the first set of conditions includes: 1) the NAL unit has a nuh_layer_id that is not included in the list of NAL unit header identifier values ​​in the OLS that has the target output layer set OLS index; 2) the NAL unit is not a DCINAL unit, a VPS NAL unit, an AUD NAL unit, or an EOB NAL unit; and 3) the NAL unit is not a NAL unit that contains a specific non-scalable nested SEI message.

12. The apparatus according to claim 9, wherein, The rule also specifies the removal of all VCLNAL units that satisfy the second set of conditions, wherein the second set of conditions includes: nal_unit_type equal to TRAIL_NUT, STSA_NUT, RADL_NUT or RASL_NUT, or nal_unit_type equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0.

13. A non-transitory computer-readable storage medium storing instructions that cause a processor to: According to the rules, a conversion between the video (including one or more layers) and the bitstream of the video is performed. in, The rules define the sub-bitstream extraction process, which involves removing Network Abstraction Layer (NAL) units from the input bitstream to generate the output sub-bitstream. The rule further stipulates that when the list of NAL unit header identifier values ​​in the OLS with the target output layer set OLS index does not include all values ​​of NAL unit header identifiers in all video codec layer VCL NAL units in the input bitstream, all supplementary enhancement information SEI NAL units containing non-scalable nested SEI messages with a specific payload type are removed. The non-scalable nested SEI message having the specific payload type includes a non-scalable nested SEI message having a first specific payload type corresponding to a sub-image level information SEI message.

14. The non-transitory computer-readable storage medium according to claim 13, wherein, The first specific payload type is equal to 203, wherein the non-scalable nested SEI message having the specific payload type further includes a non-scalable nested SEI message having a second specific payload type corresponding to the decoding unit information message, and wherein the second specific payload type is equal to 130.

15. The non-transitory computer-readable storage medium according to claim 13, wherein, The rule also specifies the removal of NAL units that meet the first set of conditions, wherein the first set of conditions includes: 1) the NAL unit has a nuh_layer_id that is not included in the list of NAL unit header identifier values ​​in the OLS that has the target output layer set OLS index; 2) the NAL unit is not a decoding capability information DCI NAL unit, a video parameter set VPS NAL unit, an access unit separator AUD NAL unit, or a bitstream end EOB NAL unit; and 3) the NAL unit is not a NAL unit that contains a specific non-scalable nested SEI message.

16. The non-transitory computer-readable storage medium according to claim 13, wherein, The rule also specifies the removal of all VCL NAL units that satisfy the second set of conditions, wherein the second set of conditions includes: nal_unit_type equal to TRAIL_NUT, STSA_NUT, RADL_NUT or RASL_NUT, or nal_unit_type equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0.

17. A non-transitory computer-readable recording medium storing a computer program and a bit stream, wherein, When the computer program is executed by the video processing device, it implements a video encoding method to generate the bitstream, the video encoding method comprising: Generate a bitstream of the video, comprising one or more layers, according to the rules. The rules specify a sub-bitstream extraction process, in which network abstraction layer (NAL) units are removed from the input bitstream to generate an output sub-bitstream. The rule further stipulates that when the list of NAL unit header identifier values ​​in the OLS with the target output layer set OLS index does not include all values ​​of NAL unit header identifiers in all video codec layer VCL NAL units in the input bitstream, all supplementary enhancement information SEI NAL units containing non-scalable nested SEI messages with a specific payload type are removed. The non-scalable nested SEI message having the specific payload type includes a non-scalable nested SEI message having a first specific payload type corresponding to a sub-image level information SEI message.

18. The non-transitory computer-readable recording medium according to claim 17, wherein, The first specific payload type is equal to 203, wherein the non-scalable nested SEI message having the specific payload type further includes a non-scalable nested SEI message having a second specific payload type corresponding to the decoding unit information message, and wherein the second specific payload type is equal to 130.

19. The non-transitory computer-readable recording medium according to claim 17, wherein, The rule also specifies the removal of NAL units that meet the first set of conditions, wherein the first set of conditions includes: 1) the NAL unit has a nuh_layer_id that is not included in the list of NAL unit header identifier values ​​in the OLS that has the target output layer set OLS index; 2) the NAL unit is not a decoding capability information DCI NAL unit, a video parameter set VPS NAL unit, an access unit separator AUD NAL unit, or a bitstream end EOB NAL unit; and 3) the NAL unit is not a NAL unit that contains a specific non-scalable nested SEI message.

20. The non-transitory computer-readable recording medium according to claim 17, wherein, The rule also specifies the removal of all VCL NAL units that satisfy the second set of conditions, wherein the second set of conditions includes: nal_unit_type equal to TRAIL_NUT, STSA_NUT, RADL_NUT or RASL_NUT, or nal_unit_type equal to GDR_NUT and the associated ph_recovery_poc_cnt is greater than 0.

21. A method for storing a video bitstream, comprising: The bitstream is generated by performing a video encoding method; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The video encoding method includes: Generate a bitstream of the video, comprising one or more layers, according to the rules. The rules specify a sub-bitstream extraction process, in which network abstraction layer (NAL) units are removed from the input bitstream to generate an output sub-bitstream. The rule further stipulates that when the list of NAL unit header identifier values ​​in the OLS with the target output layer set OLS index does not include all values ​​of NAL unit header identifiers in all video codec layer VCL NAL units in the input bitstream, all supplementary enhancement information SEI NAL units containing non-scalable nested SEI messages with a specific payload type are removed. The non-scalable nested SEI message having the specific payload type includes a non-scalable nested SEI message having a first specific payload type corresponding to a sub-image level information SEI message.

22. A video data processing method, comprising: Perform a conversion between the video and its bitstream, wherein the bitstream can be divided into one or more sub-bitstreams according to rules governing a sub-bitstream extraction process to generate an output bitstream, and The rules specify whether and how to remove all supplementary enhancement information (SEI) SEI units containing images or sub-images that are used to remove the VCL NAL unit during the sub-bitstream extraction process, based on the type of the VCL NAL unit and the temporal identifier of the VCL NAL unit. The rule specifies the removal of all SEI NAL units in response to: 1) the type of the VCL NAL unit being equal to TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, or GDR_NUT, and 2) the temporal identifier of the video codec layer being greater than or equal to the number of sublayers in the video codec layer in the output layer set.

23. The method according to claim 22, wherein, For the type of the VCL NAL unit equal to GDR_NUT, the syntax field indicating the recovery point of the decoded picture in the output order has a value greater than 0.

24. The method according to claim 22, wherein, The rule stipulates that SEINAL units are retained in the output bitstream.

25. The method according to any one of claims 22 to 24, wherein, The conversion includes encoding the video into the bitstream.

26. The method according to any one of claims 22 to 24, wherein, The conversion includes decoding the video from the bitstream.

27. The method according to any one of claims 22 to 24, wherein, The conversion includes generating the bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

28. A video processing apparatus comprising a processor configured to implement the method of any one of claims 22 to 27.

29. A method for storing a video bitstream, comprising: The bitstream is generated by performing the method according to any one of claims 22 to 25; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

30. A computer-readable medium storing program code, which, when executed, causes a processor to perform the method according to any one of claims 22 to 27.

31. A computer-readable medium storing a computer program and a bit stream, wherein, When the computer program is executed by the video processing device, the method of any one of claims 22 to 25 generates the bitstream.

32. A video processing apparatus for storing bitstreams, wherein, The video processing apparatus includes a processor and a computer-readable storage medium, the processor being configured to generate the bitstream by performing the method according to any one of claims 22 to 25, and the computer-readable storage medium being configured to store the bitstream.

Images