Constraints on the number of sub-pictures in a video picture

Abstract

Several techniques for video encoding and decoding are described. One exemplary method includes converting between video and its bitstream according to a rule that specifies that a subpicture sequence includes (1) all subpictures in a target CVS that have the same subpicture index and belong to a layer in a multi-subpicture layer, and (2) all subpictures in a target CVS that have a subpicture index of 0 and belong to a layer in the OLS but do not belong to a multi-subpicture layer.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS Books This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 036,365, filed June 8, 2020. Mainly Stretch Based on International Patent Application No. PCT / US2021 / 036257, filed June 7, 2021 For all legal purposes, the entire disclosure of the above application is incorporated by reference as part of the disclosure of this specification.

[0002] This patent specification relates to image and video coding and decoding. [Background technology]

[0003] Digital video accounts for the largest bandwidth usage on the Internet and other digital communications networks, and the bandwidth demands for digital video use are expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases. Summary of the Invention

[0004] This specification discloses techniques that video encoders and decoders can use to process coded representations of videos or images.

[0005] In one exemplary aspect, a method for processing video data is disclosed. The method includes converting between a video including one or more output layer sets (OLS) and a bitstream of the video according to a rule. The rule specifies that a sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information regarding the level of a sub-picture sequence in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies. The syntax structure of the SLI SEI message includes: (1) a first syntax element that specifies a maximum number of sub-layers for the sub-picture sequence; (2) a second syntax element that specifies whether level information for the sub-picture sequence exists for one or more sub-layer representations; and (3) a loop for a number of sub-layers each associated with a portion of the level restriction of the bitstream, and a level indicator that indicates the level to which each sub-picture sequence conforms.

[0006] In another exemplary aspect, a method for processing video data is disclosed. The method includes converting between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule. The rule specifies that a sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information regarding a level of a sub-picture sequence in a set of coded video sequences of one or more OLS to which the SLI SEI message applies. The SLI SEI messages continue in decoding order from the current access unit to the end of the bitstream or to a next access unit that includes a subsequent SLI SEI message that includes different content than the SLI SEI message.

[0007] In another exemplary aspect, a method for processing video data is disclosed. The method includes converting between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule. A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information about a level of a sub-picture sequence in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies. A variable in a reference sequence parameter set indicating a number of sub-pictures in one or more OLSs that is greater than one is called a multi-sub-picture layer. A coded video sequence in a set of OLSs is called a target coded video sequence (CVS). The rule specifies that one sub-picture sequence includes (1) all sub-pictures in a target CVS that have the same sub-picture index and belong to a layer in the multi-sub-picture layer, and (2) all sub-pictures in a target CVS that have a sub-picture index of 0 and belong to a layer of the OLS but do not belong to the multi-sub-picture layer.

[0008] In another exemplary aspect, a video processing method is disclosed that includes converting between video including one or more video sub-layers and a coded representation of the video, the coded representation conforming to a format rule that specifies the coded representation to include a syntax structure spanning multiple sub-layers in the coded representation and one or more syntax fields indicating each sub-layer included in the syntax structure, the syntax structure including information regarding a signaled portion and a reference level indicator.

[0009] In another exemplary aspect, another video processing method is disclosed that includes converting between a video including one or more sub-pictures and a coded representation of the video, the conversion using or generating supplemental enhancement information for one or more sub-picture level information.

[0010] In yet another exemplary aspect, a video encoder apparatus is disclosed, the video encoder comprising a processor configured to implement the above-described method.

[0011] In yet another exemplary aspect, a video decoder apparatus is disclosed, the video decoder comprising a processor configured to implement the above-described method.

[0012] In yet another exemplary aspect, a computer readable medium is disclosed having code stored thereon, the code performing one of the methods described herein in the form of processor executable code.

[0013] These and other features are described throughout the specification. [Brief description of the drawings]

[0014] [Figure 1] An example of raster scan slice division of a picture is shown, where the picture is divided into 12 tiles and 3 raster scan slices. [Diagram 2] An example of rectangular slice division of a picture is shown, where the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices. [Diagram 3] An example of a picture divided into tiles and rectangular slices is shown, where the picture is divided into four tiles (two columns of tiles and two rows of tiles) and four rectangular slices. [Figure 4] A picture is shown divided into 15 tiles, 24 slices, and 24 subpictures. [Diagram 5] FIG. 1 is a block diagram illustrating an example of a video processing system. [Figure 6] FIG. 1 is a block diagram of a video processing device. [Figure 7] 1 is a flowchart illustrating an example of a video processing method. [Figure 8] 1 is a block diagram illustrating a video coding system according to some embodiments of the present disclosure. [Figure 9] 1 is a block diagram illustrating an encoder according to some embodiments of the present disclosure. [Figure 10] 2 is a block diagram illustrating a decoder according to some embodiments of the present disclosure. [Figure 11] 1 shows an example of a typical sub-picture based viewport-dependent 360° video coding scheme. [Figure 12] We present a viewport-dependent 360° video coding scheme based on subpicture and spatial scalability. [Figure 13] 1 is a flowchart illustrating a method of video data processing in accordance with one or more embodiments of the present technology. [Figure 14] 11 is a flowchart illustrating another method of processing video data in accordance with one or more embodiments of the present technology. [Figure 15] 11 is a flowchart illustrating yet another method of processing video data in accordance with one or more embodiments of the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Section headings are used herein for ease of understanding and are not intended to limit the applicability of the technology and embodiments described in each section to that section alone. Additionally, the term H.266 is used in certain descriptions for ease of understanding only and is not intended to limit the scope of the disclosed technology. As such, the technology described herein is applicable to other video codec protocols and designs. In this specification, editorial changes are indicated in the text by strikethrough to indicate text that has been struck out and highlighting (including bold italics) to indicate text that has been added relative to the current draft of the VVC standard.

[0016] 1. Overview This specification relates to video coding technology, and in particular to defining and signaling level information for sub-picture sequences. The present invention may be applied to any video coding standard or non-standard video codec that supports single-layer and multi-layer video coding, such as the currently under development Generic Video Coding (VVC).

[0017] 2. Abbreviations APS Adaptation Parameter Set AU Access Unit AUD Access Unit Delimiter AVC Advanced Video Coding BP Buffering Period CLVS Coded Layer Video Sequence CPB Coded Picture Buffer CRA Clean Random Access CTU Coding Tree Unit CVS Coded Video Sequence DPB Decoded Picture Buffer DPS Decoding Parameter Set DUI Decoding Unit Information EOB End Of Bitstream EOS End Of Sequence GCI General Constraints Information GDR Gradual Decoding Refresh HEVC High Efficiency Video Coding HRD Hypothetical Reference Decoder IDR Instantaneous Decoding Refresh JEM Joint Exploration Model MCTS Motion-Constrained Tile Sets NAL Network Abstraction Layer OLS Output Layer Set PH Picture Header PPS Picture Parameter Set PT Picture Timing PTL Profile, Tier and Level PU Picture Unit RRP Reference Picture Resampling RBSP Raw Byte Sequence Payload SEI Supplemental Enhancement Information SH Slice Header SLI Subpicture Level Information SPS Sequence Parameter Set SVC Scalable Video Coding VCL Video Coding Layer VPS Video Parameter Set VTM VVC Test Model VUI Video Usability Information VVC Versatile Video Coding

[0018] 3. Initial discussions Video coding standards have evolved primarily through the development of well-known ITU-T and ISO / IEC standards. ITU-T created H.261 and H.263, ISO / IEC created MPEG-1 and MPEG-4 Visual, and the two organizations jointly created H.262 / MPEG-2Video, H.264 / MPEG-4AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on hybrid video coding structures where temporal prediction and transform coding are utilized. In 2015, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) to explore future video coding technologies beyond HEVC. Since then, many new methods have been adopted by JVET and incorporated into reference software called the Joint Exploration Model (JEM). JVET meets quarterly, and the new coding standard aims for a 50% bitrate reduction compared to HEVC. At the April 2018 JVET meeting, the new video coding standard was officially named "VVC (Versatile Video Coding)" and the first version of the VVC Test Model (VTM) was released at that time. As efforts to contribute to the standardization of VVC continue, new coding techniques are adopted for the VVC standard at every JVET meeting. After every meeting, the VVC working draft and test model VTM are updated. The VVC project is currently aiming for technical finalization (FDIS) at the July 2020 meeting.

[0019] 3.1 Picture Partitioning Scheme in HEVC HEVC has four different image partitioning schemes - regular slices, dependent slices, tiles and Wavefront Parallel Processing (WPP) - that can be applied to achieve maximum transmission unit (MTU) size matching, parallel processing and reduced end-to-end latency.

[0020] Regular slices are similar to H.264 / AVC: each regular slice is encapsulated in its own NAL unit, and in-picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are disabled. In this way, one regular slice can be reconstructed independently of other regular slices in the same picture (but there may still be interdependencies due to loop filtering operations).

[0021] Regular slices are the only tool available for parallelization, and can be used in almost the same form in H.264 / AVC. Parallelization based on regular slices does not require much inter-processor or inter-core communication (except for inter-processor or inter-core data sharing for motion compensation when decoding predictively coded pictures, which is usually much heavier than inter-processor or inter-core data sharing for in-picture prediction). However, for the same reason, using regular slices can result in a larger coding overhead due to the bit cost of slice headers and the lack of prediction across slice boundaries. Furthermore, regular slices (in contrast to other tools described below) also serve as a key mechanism for partitioning the bitstream to accommodate MTU size requirements, due to the in-picture independence of regular slices and the encapsulation of each regular slice in its own NAL unit. In many cases, the parallelization goal and the MTU size matching goal impose conflicting demands on the slice layout in the picture. Realizing this situation has led to the development of parallelization tools such as:

[0022] Dependent slices have short slice headers and allow for partitioning the bitstream at treeblock boundaries without interrupting any intra-picture prediction. Essentially, dependent slices reduce the end-to-end delay by fragmenting regular slices into multiple NAL units and allowing parts of a regular slice to be sent before the coding of the entire regular slice is completed.

[0023] In WPP, a picture is split into single rows of coding tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other splits. Parallel processing is possible by parallel decoding of CTB rows, where the start of decoding of one CTB row is delayed by two CTBs, thereby ensuring that data for the CTBs above and to the right of the target CTB is available before the target CTB is decoded. By using this staggered start (which looks like a wavefront when represented graphically), it is possible to parallelize with up to the number of processors / cores a picture contains CTB rows. Since in-picture prediction between one in-picture neighboring tree block rows is allowed, the inter-processor / inter-core communication required to enable in-picture prediction may be sufficient. WPP splitting does not result in the generation of additional NAL units compared to the case where it is not applied, and therefore WPP is not a tool for MTU size matching. However, regular slices can be used in WPP if MTU size matching is required, with certain coding overhead.

[0024] A tile defines horizontal and vertical boundaries that divide a picture into tile columns and rows. Tile columns run from the top to the bottom of a picture. Similarly, tile rows run from the left to the right of a picture. The number of tiles in a picture can be obtained by simply multiplying the number of tile columns by the number of tile rows.

[0025] The scan order of the CTBs is changed to be local within one tile (in the order of the CTB raster scan of one tile), and then the top-left CTB of the next tile is decoded according to the order of the tile raster scan of one picture. As with regular slices, tiles lose in-picture prediction dependencies and entropy decoding dependencies. However, these do not need to be included in individual NAL units (same as WPP in this respect), and therefore tiles cannot be used for MTU size matching. Each tile may be processed by one processor / core, and with the inter-processor / inter-core communication required for in-picture prediction between processing units, the decoding of neighboring tiles is limited to the communication of shared slice headers and sharing related to loop filtering of reconstructed samples and metadata if a slice spans two or more tiles. If a slice contains two or more tiles or WPP segments, the entry point byte offset of each tile or WPP segment other than the first one in the slice is signaled in the slice header.

[0026] For simplicity, HEVC specifies restrictions on the application of four different picture partitioning schemes. A given coded video sequence cannot contain both tiles and wavefronts for most of the profiles specified for HEVC. For each slice and tile, one or both of the following conditions must be satisfied: 1) all coded treeblocks in a slice belong to the same tile, and 2) all coded treeblocks in a tile belong to the same slice. Finally, a wavefront segment contains exactly one CTB row, and if a slice starts within a CTB row when WPP is used, it must end in the same CTB row.

[0027] A recent HEVC amendment is the JCT-VC output document JCTVC-AC1005, J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sulivan, A. Tourapis, and Y.-K. Wang (editors), "HEVC Additional Acquisition Enhancement Information (Draft 4)," 24 October 2017, available at: http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. With this amendment, HEVC specifies three MCTS-related SEI messages: the Temporal MCTS SEI message, the MCTS Extraction Information Set SEI message, and the MCTS Extraction Information Nesting SEI message.

[0028] The temporal MCTS SEI message indicates the presence of an MCTS in the bitstream and signals the MCTS. In each MCTS, motion vectors are restricted to point to full sample positions within the MCTS and fractional sample positions that require only full sample positions within the MCTS for interpolation, and the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is not allowed. In this way, each MCTS may be decoded independently, with no tiles not included in the MCTS.

[0029] The MCTS extraction information set SEI message provides supplemental information that can be used in MCTS sub-bitstream extraction (defined as part of the semantics of the SEI message) to generate a conforming bitstream for an MCTS set. This information includes multiple extraction information sets, each of which defines multiple MCTS sets and includes the RBSP bytes of alternative VPS, SPS, and PPS used in the MCTS sub-bitstream extraction process. When extracting a sub-bitstream through the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, PPS) need to be rewritten or replaced because one or all of the syntax elements related to slice addresses (including first_slice_segment_in_pic_flag and slice_segment_address) need to be different values.

[0030] 3.2. Picture Segmentation in VVC In VVC, a picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of CTUs that covers a rectangular area of ​​a picture. The CTUs in a tile are scanned in raster scan order within the tile.

[0031] A slice consists of an integer number of complete tiles or an integer number of complete contiguous CTU rows within a picture tile.

[0032] Two modes of slicing are supported: raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains one complete sequence of tiles in the tile raster scan of a picture. In rectangular slice mode, a slice contains either multiple complete tiles that collectively form a rectangular area of ​​the picture, or multiple consecutive complete CTU rows of a tile that collectively form a rectangular area of ​​the picture. The tiles in a rectangular slice are scanned in tile raster scan order within the rectangular area corresponding to the slice.

[0033] A subpicture contains one or more slices that collectively span a rectangular area of ​​a picture.

[0034] FIG. 1 shows an example of raster scan slice division of a picture, where the picture is divided into 12 tiles and 3 raster scan slices.

[0035] FIG. 2 shows an example of rectangular slice division of a picture, where the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices.

[0036] FIG. 3 shows an example of a picture divided into tiles and rectangular slices, where the picture is divided into four tiles (two tile columns and two tile rows) and four rectangular slices.

[0037] FIG. 4 shows an example of dividing a picture into subpictures, where a picture is divided into 18 tiles, the 12 on the left containing one slice of a 4×4 CTU each, and the 6 tiles on the right containing vertically stacked slices of a 2×2 CTU each, resulting in a total of 24 slices and 24 subpictures of different dimensions (each slice is one subpicture).

[0038] 3.3 Changing picture resolution in a sequence In AVC and HEVC, the spatial resolution of a picture cannot be changed unless a new sequence using the new SPS starts with an IRAP picture. VVC allows changing the resolution of a picture within a sequence at a position without always encoding an intra-coded IRAP picture. This feature is called reference picture resampling (RPR) because it is necessary to resample the reference picture used for inter prediction when the reference picture has a different resolution than the current picture being decoded.

[0039] The scaling ratio is limited to 1 / 2 (2x downsampling from reference picture to current picture) and 8 (8x upsampling). To address various scaling ratios between the reference picture and the current picture, we define three sets of resampling filters with different frequency cutoffs. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, 1 / 1.75 to 1 / 1.25, and 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luma and 32 phases for chroma, as in the case of motion compensation interpolation filters. In practice, the normal MC interpolation process is a special case of the resampling process with scaling ratios ranging from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height, as well as the left, right, top and bottom scaling offsets defined for the reference picture and the current picture.

[0040] Other aspects of the VVC design to support this feature that differ from HEVC include: i) signaling image resolution and corresponding compatibility window in PPS instead of SPS, and signaling maximum picture resolution in SPS; ii) In case of single layer bitstream, each picture storage area (one slot in DPB for storing one decoded picture) occupies the buffer size required to store a decoded picture with maximum picture resolution.

[0041] 3.4. Scalable Video Coding (SVC) in General and in VVC Scalable Video Coding (SVC, sometimes referred to as scalability in video coding) refers to video coding in which a Base Layer (BL) (sometimes referred to as a Reference Layer (RL)) and one or more Enhancement Layers (EL) are used. In SVC, a base layer may carry a base quality level of video data. One or more enhancement layers may carry additional video data, e.g., to support higher spatial, temporal, and / or signal-to-noise (SNR) levels. An enhancement layer may be defined relative to a previous, coded layer. For example, a lower layer may function as a BL and an upper layer may function as an EL. An intermediate layer may function as either an EL or an RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the top layer) may be an EL for a layer below the intermediate layer, e.g., a base layer or any intervening enhancement layer, and at the same time serve as an RL for one or more enhancement layers above the intermediate layer. Similarly, in multiview or 3D extensions of the HEVC standard, there may be multiple views and information of one view may be utilized to code (e.g., encode or decode) information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0042] In SVC, parameters used by an encoder or decoder are grouped into parameter sets based on the coding level at which they can be utilized (e.g., video level, sequence level, picture level, slice level, etc.). For example, parameters that can be utilized by a coded video sequence at different layers in a bitstream may be included in a video parameter set (VPS), and parameters utilized by one or more pictures in a coded video sequence may be included in a sequence parameter set (SPS). Similarly, parameters utilized by one or more slices of a picture may be included in a picture parameter set (PPS), and other parameters specific to a slice may be included in a slice header. Similarly, an indication of which parameter set a particular layer is using at a given time may be provided at various coding levels.

[0043] Thanks to the support of Reference Picture Resampling (RPR) in VVC, the upsampling required for spatial scalability support only requires the use of an RPR upsampling filter, so that a bitstream containing multiple layers, e.g., two layers of SD and HD resolution in VVC, can be designed to support without requiring additional signal processing level coding tools. Nevertheless, a high level of syntax change (compared to not supporting scalability) is required for scalability support. Scalability support is specified in VVC version 1. Unlike scalability support in any previous video coding standard, including the extensions of AVC and HEVC, the design of scalability in VVC has been made as suitable as possible for the design of a single-layer decoder. The decoding capabilities of a multi-layer bitstream are specified as if there was only one layer in the bitstream. For example, the decoding capabilities, such as DPB size, are specified to be independent of the number of layers of the bitstream being decoded. Essentially, a decoder designed for a single-layer bitstream does not require many modifications to be able to decode a multi-layer bitstream. Compared to the multi-layer extension designs of AVC and HEVC, the HLS aspects have been significantly simplified at the expense of some flexibility: for example, an IRAP AU is required to contain an image for each layer present in the CVS.

[0044] 3.5 Subpicture-Based Viewport-Dependent 360° Video Streaming In streaming 360° video, i.e., streaming omnidirectional video, at any particular moment, only a subset of the entire omnidirectional video sphere (i.e., the current viewport) is rendered to the user, while the user can turn his head at any time to change his gaze direction and, therefore, the current viewport. It is desirable for the client side to have at least some lower quality representation of the areas not covered by the current viewport ready to be rendered to the user, but only for the current viewport that is immediately being rendered to the user, in case the user suddenly changes his gaze direction to anywhere on the sphere. Such optimization is enabled by splitting the high-quality representation of the entire omnidirectional video into sub-pictures with appropriate granularity. Using VVC, the two representations can be coded as two layers independent of each other.

[0045] A typical sub-picture based viewport dependent 360° video delivery scheme is shown in Figure 11, where a higher resolution representation of the full video consists of sub-pictures, while a lower resolution representation of the full video does not use sub-pictures and can be coded with less frequent random access points than the higher resolution representation. The client receives the full low resolution video, and for the higher resolution video it receives and decodes only the sub-pictures that cover the current viewport.

[0046] Also, the recent VVC draft specification supports an improved 360° video coding scheme, as shown in Figure 12. The only difference compared to the approach shown in Figure 11 is that inter-layer prediction (ILP) is applied for the approach shown in Figure 12.

[0047] 3.6. Parameter Set AVC, HEVC, and VVC specify parameter sets. Parameter set types are SPS, PPS, APS, VPS, etc. SPS and PPS are supported by all of AVC, HEVC, and VVC. VPS was introduced by HEVC and is included in both HEVC and VVC. APS was not included in AVC or HEVC, but is included in the recent VVC draft text.

[0048] The SPS was designed to carry sequence-level header information, and the PPS was designed to carry picture-level header information that does not change frequently. With the SPS and PPS, redundant signaling of frequently changing information can be avoided because there is no need to repeat this information for every sequence or picture. Furthermore, using the SPS and PPS enables out-of-band transmission of important header information, thereby improving error resilience as well as avoiding the need for redundant transmission.

[0049] The VPS was introduced to carry sequence-level header information that is common to all layers of a multi-layer bitstream.

[0050] APS was introduced to carry such picture-level or slice-level information that requires significant bits to code, is shared by multiple pictures, and can exist in many different variations in a sequence.

[0051] 3.7. Profiles, Tiers and Levels Video coding standards typically specify profiles and levels, and some also specify tiers, such as HEVC and the currently developing VVC.

[0052] Profiles, tiers, and levels specify restrictions on a bitstream and therefore limit the capabilities required to decode the bitstream, and may be used to indicate interoperability between individual decoder implementations.

[0053] Each profile specifies a subset of algorithmic features and limitations that should be supported by all decoders conforming to that profile. Note that an encoder is not required to use all features or characteristics supported by a profile, and a decoder conforming to a profile is required to support all coding tools or characteristics.

[0054] Each level of a tier specifies a set of restrictions on the values ​​that bitstream syntax elements can take. The same set of tier and level definitions is typically used for all profiles, but individual implementations may support different tiers, and within a tier, different levels for each supported profile. For any given profile, the level of a tier generally corresponds to a particular decoder processing load and memory capabilities.

[0055] The capabilities of a video decoder compliant with a video codec specification are specified by its ability to decode video streams that conform to the profile, tier, and level constraints specified in the video codec specification. When expressing the capabilities of a decoder for a specified profile, the tiers and levels supported for that profile should also be expressed.

[0056] 3.8. Definition and Signaling of Level Information for Subpicture Sequences in VVC In recent VVC draft texts, the definition and signaling of sub-picture sequence level information in VVC is via the Sub-picture Level Information (SLI) SEI message, which allows the extraction of sub-picture sequences from the bitstream by applying the Sub-picture Sub-bitstream Extraction Procedure specified in VVC clause C.7.

[0057] The syntax and semantics of the sub-picture level information SEI message in the recent VVC draft text are as follows:

[0058] D.7.1 Sub-picture level information SEI message syntax [Table 1]

[0059] D.7.2 Sub-picture level information SEI message semantics The Sub-picture Level Information SEI message contains information about the level of conformance of the sub-picture sequences in a bitstream when testing the conformance of an extracted bitstream containing sub-picture sequences according to Appendix A. If a sub-picture level information SEI message exists for any picture in a CLVS, then a sub-picture level information SEI message shall exist for the first picture in a CLVS. Sub-picture level information SEI messages persist in decoding order from the current picture to the end of the CLVS for the current layer. All sub-picture level information SEI messages that apply to the same CLVS shall have the same content. One sub-picture sequence consists of all sub-pictures in one CLVS, and these sub-pictures have the same sub-picture index value. If there is one sub-picture level information SEI message for one CLVS, it is a bitstream conformance requirement that the value of sps_subpic_treated_as_pic_flag[i] be equal to 1 for each value of i in the range from 0 to sps_num_subpics_minus1. [ka]

[0060] [ka]

[0061] [ka]

[0062] The variable SubpicSizeY[j] is set equal to (sps_subpic_width_minus1[j]+1)*CtbSizeY*(sps_subpic_height_minus1[j]+1)*CtbSizeY. If sli_ref_level_fraction_minus1[i][j] is not present, its value is inferred to be equal to Ceil(256*SubpicSizeY[j]÷PicSizeInSamplesY*MaxLumaPs(general_level_idc)÷MaxLumaPs(sli_ref_level_idc[i])-1. The variable LayerRefLevelFraction[i][j] is set equal to sli_ref_level_fraction_minus1[i][j]+1. The variable OlsRefLevelFraction[i][j] is set equal to sli_non_subpic_layers_fraction[i]+(256-sli_non_subpic_layers_fraction[i])÷256*(sli_ref_level_fraction_minus1[i][j]+1). The variables SubpicCpbSizeVcl[i][j] and SubpicCpbSizeNal[i][j] are derived as follows. SubpicCpbSizeVcl[i][j]=Floor(CpbVclFactor*MaxCPB*OlsRefLevelFraction[i][j]÷256) (D.6) SubpicCpbSizeNal[i][j]=Floor(CpbNalFactor*MaxCPB*OlsRefLevelFraction[i][j]÷256) (D.7) Use the MaxCPB derived from sli_ref_level_idc[i] as specified in Section A.4.2. The variables SubpicBitRateVcl[i][j] and SubpicBitRateNal[i][j] are derived as follows. SubpicBitRateVcl[i][j]=Floor(CpbVclFactor*ValBR*OlsRefLevelFraction[0][j]÷256) (D.8) SubpicBitRateNal[i][j]=Floor(CpbNalFactor*ValBR*OlsRefLevelFraction[0][j]÷256) (D.9) Here, the value of ValBR is derived as follows. - If bit_rate_value_minus1[Htid][ScIdx] is available in the respective HRD parameters in the VPS or SPS, ValBR is (bit_rate_value_minus1[Htid][ScIdx]+1)*2 (6+bit_rate_scale) where Htid is the considered sublayer index and ScIdx is the considered schedule index. - Otherwise, ValBR is set equal to MaxBR derived from sli_ref_level_idc[0] as specified in Section A.4.2. NOTE 1 - When extracting subpictures, the resulting bitstream has a CpbSize (as indicated or inferred in either the VPS or SPS) greater than or equal to SubpicCpbSizeVcl[i][j] and SubpicCpbSizeNal[i][j], and a BitRate (as indicated or inferred in either the VPS or SPS) greater than or equal to SubpicBitRateVcl[i][j] and SubpicBitRateNal[i][j]. In the input bitstream to the extraction process, the jth subpicture, j in the range from 0 to sps_num_subpics_minus1, is extracted from layers with sps_num_subpics_minus1 greater than 0, and each layer in the bitstream obtained by conforming to a profile with general_tier_flag equal to 0 and level equal to sli_ref_level_idc[i] (i in the range from 0 to num_ref_level_minus1) shall comply with the following constraints in each bitstream conformance test specified in Appendix C. - Ceil(256*SubpicSizeY[j]÷LayerRefLevelFraction[i][j]) shall be less than or equal to MaxLumaPs, where MaxLumaPs is specified in Table A.1 for level sli_ref_level_idc[i]. The value of -Ceil(256*(sps_subpic_width_minus1[j]+1)*CtbSizeY÷LayerRefLevelFraction[i][j]) shall be less than or equal to Sqrt(MaxLumaPs*8). The value of -Ceil(256*(sps_subpic_height_minus1[j]+1)*CtbSizeY÷LayerRefLevelFraction[i][j]) shall be less than or equal to Sqrt(MaxLumaPs*8). - The value of SubpicWidthInTiles[j] shall be less than or equal to MaxTileCols and the value of SubpicHeightInTiles[j] shall be less than or equal to MaxTileRows, where MaxTileCols and MaxTileRows are specified in Table A.1 for level sli_ref_level_idc[i]. - The value of SubpicWidthInTiles[j]*SubpicHeightInTiles[j] shall be less than or equal to MaxTileCols*MaxTileRows*LayerRefLevelFraction[i][j], where MaxTileCols and MaxTileRows are specified in Table A.1 for level sli_ref_level_idc[i]. The bitstream conformance requirement is that each layer obtained by extracting the jth subpicture, where j is in the range from 0 to sps_num_subpics_minus1, and conforming to a profile with general_tier_flag equal to 0 and level equal to ref_level_idc[i] (i in the range from 0 to sli_num_ref_level_minus1), shall comply with the following constraints for each bitstream conformance test specified in Appendix C. - for the value of SubpicSizeInSamplesY of AU0, the sum of the NumBytesInNalUnit variables of AU0 corresponding to the jth subpicture shall be less than or equal to FormatCapabilityFactor*(Max(SubpicSizeY[j],fR*MaxLumaSr*OlsRefLevelFraction[i][j]÷256)+MaxLumaSr*(AuCpbRemovalTime[0]-AuNominalRemovalTime[0])*OlsRefLevelFraction[i][j])÷(256*MinCr) for the value of SubpicSizeInSamplesY of AU0, where MaxLumaSr and FormatCapabilityFactor are the values ​​specified in Tables A.2 and A.3, respectively, that apply to AU0 at level sli_ref_level_idc[i], and MinCr is derived as indicated in A.4.2. - For the value of SubpicSizeInSamplesY of AU0, the sum of the NumBytesInNalUnit variables of AUn corresponding to the jth subpicture, for n greater than 0, shall be less than or equal to FormatCapabilityFactor*MaxLumaSr*(AuCpbRemovalTime[n]-AuCpbRemovalTime[n-1])*OlsRefLevelFraction[i][j]÷(256*MinCr), where MaxLumaSr and FormatCapabilityFactor are the values ​​specified in Tables A.2 and A.3, respectively, that apply to AUn at level sli_ref_level_idc[i], and MinCr is derived as indicated in A.4.2. The value of the sub-picture sequence level indicator SubpicLevelIdc is derived as follows. SubpicLevelIdc=general_level_idc SubpicLevelIdx=0 for(i=num_ref_level_minus1;i>=1;i--) if(OlsLevelFraction[i]<=256) SubpicLevelIdc=sli_ref_level_idc[i] SubpicLevelIdx=i } (D.10) A subpicture sequence bitstream conforming to a profile with general_tier_flag equal to 0 and level equal to SubpicLevelIdc shall obey the following constraints for each bitstream conformance test, as specified in Appendix C. - For VCL HRD parameters, SubpicCpbSizeVcl[i] shall be less than or equal to CpbVclFactor*MaxCPB, where CpbVclFactor is specified in Table A.3 and MaxCPB is specified in Table A.1 in units of CpbVclFactor bits. For NAL HRD parameters, SubpicCpbSizeNal[i] shall be less than or equal to CpbNalFactor*MaxCPB, where CpbNalFactor is specified in Table A.3 and MaxCPB is specified in Table A.1 in units of CpbNalFactor bits. - For VCL HRD parameters, SubpicBitRateVcl[i] shall be less than or equal to CpbVclFactor*MaxBR, where CpbVclFactor is specified in Table A.3 and MaxBR is specified in Table A.1 in units of CpbVclFactor bits. - For NAL HRD parameters, SubpicBitRateNal[i] shall be less than or equal to CpbNalFactor*MaxBR, where CpbNalFactor is specified in Table A.3 and MaxBR is specified in Table A.1 in units of CpbNalFactor bits. NOTE 2 - When a subpicture sequence is extracted, the resulting bitstream has a CpbSize (as indicated or inferred in either the VPS or SPS) that is greater than or equal to SubpicCpbSizeVcl[i][j] and SubpicCpbSizeNal[i][j], and a BitRate (as indicated or inferred in either the VPS or SPS) that is greater than or equal to SubpicBitRateVcl[i][j] and SubpicBitRateNal[i][j].

[0063] 4. Technical problem solved by the disclosed technical solution The existing VVC design for defining and signaling level information of sub-picture sequences has the following problems.

[0064] (1) The SLI SEI message signals only a single set of level information for a sub-picture sequence, regardless of the highest TemporalId value. However, similar to a bitstream with one sub-picture per picture, different sub-layer representations may conform to different levels.

[0065] (2) The SLI SEI message is specified to be available only if it is included in the bitstream. However, like parameter sets and other HRD-related SEI messages, the SLI SEI message should also be made available by external means.

[0066] (3) The persistence scope of an SLI SEI message is specified to be within one CVS, but in most cases an SLI SEI message applies to many consecutive CVSs, often across an entire bitstream.

[0067] (4) The definition of a sub-picture sequence does not include the case where there is more than one layer with one sub-picture per picture.

[0068] (5) If there is one SLI SEI message for one CVS, the constraint requiring that the value of sps_num_subpics_minus1 be the same for all SPSs referenced by pictures in a layer with multiple subpictures per picture is missing. Otherwise, it does not make sense to require that the value of sli_num_subpics_minus1 be equal to the value of sps_num_subpics_minus1.

[0069] (6) The semantics of sli_num_subpics_minus1 do not work when there is one or more layers with multiple subpictures per picture.

[0070] (7) Because different subpicture sequences extracted from the same original bitstream can conform to different levels, the variables SubpicLevelIdc and SubpicLevelIdx need to be defined to be subpicture sequence specific.

[0071] 5. List of Solutions and Implementations In order to solve the above problems, the following methods are disclosed. The present invention should be regarded as an example for explaining a general concept, and should not be interpreted in a narrow sense. Furthermore, the present invention may be applied individually or in any combination. 1) To solve the first problem, add sli_max_sublayers_minus1, sli_sublayer_info_present_flag, and loop of sublayers of signaled parts and reference level indicators to be consistent with signaling of level information in the PTL syntax structure. In one example, furthermore, sli_cbr_constraint_flag is also made sub-layer specific, i.e., changed to sli_cbr_constraint_flag[k], and moved into the loop for the sub-layer. i. In one example, furthermore, if sli_cbr_constraint_flag[k] of the lower sublayer is not present, it is inferred to be equal to sli_cbr_constraint_flag[k+1]. b. In one example, furthermore, if a fraction or reference level indicator for a lower sublayer is not present, it is inferred to be the same as that for the next higher sublayer. 2) To solve the second problem, allow the SLI SEI message to be either included in the bitstream or provided by external means, and to be consistent with the parameter set and the other three conformance / HRD related SEI messages, namely PT, BP, and DUI SEI messages. 3) To solve the third problem, change the persistence scope from one CVS to one or more CVSs, making the level information consistent with the VPS and SPS that are or may be signaled. 4) To solve the fourth problem, the definition of a sub-picture sequence is modified to include the case where there is more than one layer with one sub-picture per picture. 5) To solve the fifth problem, if there is one SLI SEI message for one CVS, the value of sps_num_subpics_minus1 is required to be the same for all SPSs referenced by pictures in a layer that has multiple subpictures per picture. 6) To solve the sixth problem, the semantics of sli_num_subpics_minus1 are defined such that the syntax element is for subpictures of layers that have multiple subpictures per picture. 7) To solve the seventh problem, in the last set of constraints in the semantics of the SLI SEI message, add an array index, the index of the subpicture sequence, to both variables SubpicLevelIdc and SubpicLevelIdx and to arrays SubpicCpbSizeVcl, SubpicCpbSizeNal, SubpicBitRateVcl, and SubpicBitRateNal.

[0072] 6. Embodiments Below are some example embodiments for some of the inventive aspects summarized above in this section, as applicable to the VVC specification. The most relevant parts that have already been added or modified are underlined in bold italics, and some of the parts that have been removed are indicated using [[]].

[0073] 6.1. First embodiment This embodiment relates to items 1 to 7 and some of the subitems.

[0074] D.7.1 Sub-picture level information SEI message syntax [Table 2]

[0075] D.7.2 Sub-picture level information SEI message semantics [ka]

[0076] [ka]

[0077] [ka]

[0078] [ka]

[0079]

change

[0080]

change

[0081]

change

[0082]

change

[0083]

change

[0084]

change

[0085]

change

[0086]

change

[0087]

Chemistry

[0088]

change

[0089] [ka]

[0090] [ka]

[0091] [ka]

[0092] [ka]

[0093] [ka]

[0094] FIG. 5 is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of the system 1900. The system 1900 may include an input unit 1902 for receiving video content. The video content may be received in a raw or uncompressed format, e.g., 8 or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. The input unit 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0095] The system 1900 may include a coding component 1904 that may implement various coding or encoding methods described herein. The coding component 1904 may reduce the average bit rate of the video from the input 1902 to the output of the coding component 1904, generating a coded representation of the video. Thus, this coding technique may be referred to as a video compression or video transcoding technique. The output of the coding component 1904 may be stored or transmitted via a connected communication, as represented by component 1906. The bitstream (or coded) representation of the video received at the input 1902, stored or communicated, may be used by component 1908 to generate pixel values ​​or displayable video that are transmitted to a display interface 1910. The process of generating video viewable by a user from the bitstream representation may be referred to as video unfolding. Additionally, although certain video processing operations are referred to as "coding" operations or tools, it will be understood that the coding tools or operations are performed by an encoder and corresponding decoding tools or operations that reverse the results of the coding, performed by a decoder.

[0096] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High-Definition Multimedia Interface (HDMI), or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described herein may be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0097] FIG. 6 is a block diagram of a video processing device 3600. The device 3600 may be used to implement one or more of the methods described herein. The device 3600 may be implemented in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. The device 3600 may include one or more processing devices 3602, one or more memories 3604, and video processing hardware 3606. The one or more processing devices 3602 may be configured to implement one or more methods described herein. The memory(s) 3604 may be used to store data and codes used to implement the methods and techniques described herein. The video processing hardware 3606 may be used to implement the techniques described herein in a hardware circuit.

[0098] FIG. 8 is a block diagram illustrating an example video encoding system 100 that can utilize techniques of this disclosure.

[0099] 8, video coding system 100 may include source device 110 and destination device 120. Source device 110 generates encoded video data and may also be referred to as a video encoding device. Destination device 120 may decode the encoded video data generated by source device 110 and may also be referred to as a video decoding device.

[0100] Source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0101] The video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 and generates a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include a coded picture and associated data. A coded picture is a coded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be transmitted directly to the destination device 120 via the I / O interface 116 over the network 130a. The encoded video data may be stored on a recording medium / server 130b for access by the destination device 120.

[0102] Destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0103] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 configured to interface with an external display device.

[0104] Video encoder 114 and video decoder 124 may operate in accordance with a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVVM) standard, and other current and / or future standards.

[0105] FIG. 9 is a block diagram illustrating an example of a video encoder 200, which may be the video encoder 114 in the system 100 shown in FIG.

[0106] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 9, video encoder 200 includes multiple functional modules. Techniques described in this disclosure may be shared among various components of video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0107] The functional components of the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214, and the prediction unit 202 includes a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206.

[0108] In other examples, video encoder 200 may include more, fewer, or different functional components. In one example, prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0109] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, may be highly integrated, but are depicted separately in the example of FIG. 9 for purposes of illustration.

[0110] Division unit 201 may divide a picture into one or more video blocks. Video encoder 200 and video decoder 300 may support a variety of video block sizes.

[0111] The mode selection unit 203 may, for example, select one of intra or inter coding modes based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a Combination of Intra and Inter Prediction (CIIP) mode that performs prediction based on an inter prediction signal and an intra prediction signal. The mode selection unit 203 may also select a resolution (e.g., sub-pixel or integer pixel accuracy) of the motion vector of the block in the case of inter prediction.

[0112] To perform inter prediction on a current video block, motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from buffer 213. Motion compensation unit 205 may determine a prediction video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0113] Motion estimation unit 204 and motion compensation unit 205 may, for example, perform different operations on the current video block based on whether the current video block is an I slice, a P slice, or a B slice.

[0114] In some examples, motion estimation unit 204 may perform unidirectional prediction on the current video block, and motion estimation unit 204 may search reference pictures in list 0 or list 1 for a reference video block for the current video block. Motion estimation unit 204 may generate a reference index indicating a reference picture in list 0 or list 1 including the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, prediction direction indicator, and motion vector as motion information of the current video block. Motion compensation unit 205 may generate a prediction video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0115] In another example, motion estimation unit 204 may bidirectionally predict the current video block, and motion estimation unit 204 may search for a reference video block among the reference pictures in list 0 for the current video block, and may search for another reference video block among the reference pictures in list 1 for the current video block. Motion estimation unit 204 may generate a reference index indicating the reference pictures in list 0 and list 1 that contain the reference video block, and a motion vector indicating a spatial displacement between the reference video block and the current video block. Motion estimation unit 204 may output the reference index and the motion vector of the current video block as motion information of the current video block. Motion compensation unit 205 may generate a prediction video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0116] In some examples, the motion estimation unit 204 may output a full set of motion information for the decoding process of the decoder.

[0117] In some examples, motion estimation unit 204 may not output a full set of motion information for the current picture. Rather, motion estimation unit 204 may signal motion information of the current video block by reference to motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0118] In one example, motion estimation unit 204 may indicate, in a syntax structure associated with the current video block, a value that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0119] In another example, motion estimation unit 204 may identify another video block and a Motion Vector Difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0120] As mentioned above, video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0121] Intra prediction unit 206 may perform intra prediction on the current video block. If intra prediction unit 206 intra predicts the current video block, intra prediction unit 206 may generate predictive data for the current video block based on decoded samples of other video blocks in the same picture. The predictive data for the current video block may include a predicted video block and various syntax elements.

[0122] Residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0123] In other examples, such as in skip mode, there may be no residual data for the current video block, and residual generation unit 207 may not perform the subtraction operation.

[0124] Transform processing unit 208 may generate one or more transform coefficient image blocks for the current video block by applying one or more transforms to a residual video block associated with the current video block.

[0125] After transform processing unit 208 generates a transform coefficient image block associated with the current video block, quantization unit 209 may quantize the transform coefficient image block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0126] Inverse quantization unit 210 and inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient image block to reconstruct a residual video block from the transform coefficient image block. Reconstruction unit 212 may add the reconstructed residual video block to samples corresponding to one or more prediction video blocks generated by prediction unit 202 to generate a reconstructed video block associated with the current block for storage in buffer 213.

[0127] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0128] Entropy encoding unit 214 may receive data from other functional components of video encoder 200. If entropy encoding unit 214 receives the data, it may perform one or more entropy encoding operations to generate entropy encoded data, and output a bitstream that includes the entropy encoded data.

[0129] FIG. 10 is a block diagram of an example video decoder 300, which may be the video decoder 114 in the system 100 shown in FIG.

[0130] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 10, video decoder 300 comprises multiple functional modules. The techniques described in this disclosure may be shared among various components of video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0131] 10 example, video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306, as well as a buffer 307. Video decoder 300 may, in some examples, perform a decoding path that is generally the reverse of the encoding path described with respect to video encoder 200 (FIG. 9).

[0132] The entropy decoding unit 301 retrieves an encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 301 decodes the entropy encoded video data, and from the entropy decoded video data, the motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 may determine such information by performing, for example, AMVP and merge mode.

[0133] The motion compensation unit 302 may generate motion compensated blocks, possibly performing the interpolation based on an interpolation filter. The syntax element may include an identifier for the interpolation filter to be used with sub-pixel precision.

[0134] Motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of the reference block using an interpolation filter as used by video encoder 200 during encoding of the video block. Motion compensation unit 302 may determine the interpolation filter used by video encoder 200 based on the received syntax information and use the interpolation filter to generate the predictive block.

[0135] The motion compensation unit 302 may use some of the following information to decode the encoded video sequence: syntax information to determine the size of blocks used to encode frames and / or slices of the encoded video sequence, partitioning information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-encoded block, and other information.

[0136] The intra prediction unit 303 may form a prediction block from spatially neighboring blocks, for example, using an intra prediction mode received in the bitstream. The inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0137] Reconstruction unit 306 may sum the residual blocks with corresponding prediction blocks generated by motion compensation unit 202 or intra prediction unit 303 to form decoded blocks. If desired, a deblocking filter may be applied to filter the decoded blocks to remove block artifacts. The decoded video blocks are stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and generates decoded video for display on a display device.

[0138] Next, preferred solutions are listed in some embodiments.

[0139] The following solutions represent exemplary implementations of the techniques discussed in the previous section (eg, item 1).

[0140] 1. A video processing method (e.g., method 700 shown in FIG. 7 ), comprising converting (702) between video including one or more video sub-layers and a coded representation of the video, the coded representation conforming to a format rule that specifies the coded representation to include a syntax structure spanning multiple sub-layers in the coded representation and one or more syntax fields indicating each sub-layer included in the syntax structure, the syntax structure including information regarding signaled portions and reference level indicators.

[0141] 2. The method of solution 1, wherein the formatting rules specify that certain parts not explicitly included in the syntax structure are interpreted as having the same value as the next higher sublayer.

[0142] The following solutions represent exemplary implementations of the techniques discussed in the previous sections (eg, items 2, 5, 6).

[0143] 3. A method of video processing comprising converting between video including one or more sub-pictures and a coded representation of the video, the conversion using or generating supplemental enhancement information for the one or more sub-picture level information.

[0144] 4. The method of solution 3, wherein the supplemental enhancement information is included in the coded representation.

[0145] 5. The method according to solution 3, wherein the supplemental enhancement information is excluded from the coded representation and communicated between the encoding end and the decoding end using a different mechanism than the coded representation.

[0146] 6. The method according to solution 4, wherein the coded representation complies with a format rule that specifies that in each sequence parameter set, in a layer having multiple sub-pictures per picture, the same value for each sequence parameter set indicating the number of sub-pictures is signaled.

[0147] 7. A method according to any one of solutions 1 to 6, wherein the conversion comprises encoding the video into the coded representation.

[0148] 8. A method according to any of solutions 1 to 6, wherein the transforming comprises decoding the coded representation to generate pixel values ​​of the image.

[0149] 9. A video decoding device comprising a processing device configured to perform the methods described in one or more of solutions 1 to 8.

[0150] 10. A video encoding device comprising a processing device configured to perform the methods according to one or more of solutions 1 to 8.

[0151] 11. A computer program product having computer code stored thereon, the code causing the processor to perform a method according to any one of solutions 1 to 8 when executed by the processor.

[0152] 12. Any method, apparatus or system described herein.

[0153] 13 is a flow chart illustrating a method 1300 of video data processing according to one or more embodiments of the present technology. The method 1300 includes, at operation 1310, converting between a video including one or more output layer sets (OLS) and a bitstream of the video according to a rule. The rule specifies that a sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information regarding the level of a sub-picture sequence in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies. The syntax structure of the SLI SEI message includes (1) a first syntax element that specifies the maximum number of sub-layers for the sub-picture sequence, (2) a second syntax element that specifies whether level information for the sub-picture sequence exists for one or more sub-layer representations, and (3) a loop for a number of sub-layers each associated with a portion of the level restriction of the bitstream and a level indicator indicating the level to which each sub-picture sequence conforms.

[0154] In some embodiments, a value of the first syntax element ranges from 0 to the maximum number of sublayers indicated in the video parameter set minus 1. In some embodiments, in response to the second syntax element not being included in the bitstream, the second syntax element is inferred to be 0. In some embodiments, in response to the absence of a portion of the level restriction of the bitstream associated with one sublayer k, the portion is inferred to be equal to the portion associated with one sublayer k+1. In some embodiments, in response to the absence of a level indicator associated with one sublayer k, the level indicator is inferred to be equal to the level indicator associated with one sublayer k+1.

[0155] In some embodiments, the syntax structure further includes a third syntax element that specifies, for each sub-layer, a portion of the level restriction of the bitstream associated with the level indicator. In response to the absence of a third syntax element associated with sub-layer k, the third syntax element is inferred to be equal to the syntax element associated with sub-layer k+1. In some embodiments, the syntax structure further includes a fourth syntax element that specifies a number of reference levels signaled for each sub-picture sequence. In some embodiments, the syntax structure further includes a fifth syntax element that specifies whether a virtual stream scheduler (HSS) operates in an intermittent bit rate mode or a constant bit rate (CBR) mode for a sub-picture sequence.

[0156] 14 is a flow chart illustrating a method 1400 of video data processing according to one or more embodiments of the present technology. The method 1400 includes, at operation 1410, converting between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule. The rule specifies that a sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information regarding a level of a sub-picture sequence in a set of coded video sequences of one or more OLS to which the SLI SEI message applies. The SLI SEI messages continue in decoding order from the current access unit to the end of the bitstream or to the next access unit that includes a subsequent SLI SEI message that includes different content than the SLI SEI message.

[0157] In some embodiments, this rule specifies that all SLI SEI messages applicable to the same CVS have the same content. In some embodiments, the SLI SEI messages are present in the current access unit by being included in the bitstream or provided by external means. In some embodiments, a first variable indicating a sub-picture level index is defined to include a value for each sub-picture sequence. In some embodiments, a second variable indicating a sub-picture level index is defined to include a value for each sub-picture sequence.

[0158] FIG. 15 is a flow chart illustrating a method 1500 of video data processing according to one or more embodiments of the present technology. The method 1500 includes, at operation 1510, converting between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule. A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information about the level of a sub-picture sequence in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies. A variable in a reference sequence parameter set indicating the number of sub-pictures in one or more OLSs is greater than one is called a multi-sub-picture layer. A coded video sequence in a set of OLSs is called a target coded video sequence (CVS). The rule specifies that one sub-picture sequence includes (1) all sub-pictures in a target CVS that have the same sub-picture index and belong to a layer in the multi-sub-picture layer, and (2) all sub-pictures in a target CVS that have a sub-picture index of 0 and belong to a layer of the OLS but do not belong to the multi-sub-picture layer.

[0159] In some embodiments, the bitstream specifies based on a format rule that specifies that, in response to an SLI SEI message being present for a coded video sequence, all sequence parameter sets referenced by pictures in multiple sub-picture layers have the same number of sub-pictures. In some embodiments, in response to an SLI SEI message being present for any access unit of a coded video sequence (CVS) in one or more OLSs, the rule specifies that the SLI SEI message is present for the first access unit of the CVS. In some embodiments, a syntax element in a syntax structure of an SLI SEI message specifies the number of sub-pictures in pictures in multiple sub-picture layers in a target CVS.

[0160] In some embodiments, the conversion includes encoding the video into a bitstream. In some embodiments, the conversion includes decoding the video from the bitstream.

[0161] In the solutions described herein, an encoder may comply with the format rules by generating a coded representation according to the format rules, which a decoder may use to generate decoded video by parsing syntax elements in the coded representation, knowing their presence or absence, according to the format rules.

[0162] In this specification, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of a current video block may correspond to bits spread to the same or different locations in the bitstream, for example, as specified by a syntax. For example, one macroblock may be coded in terms of transformed and coded error residual values ​​and using bits in a header and other fields in the bitstream. Furthermore, during conversion, the decoder may parse the bitstream with the knowledge that some fields may or may not be present based on the determination, as described in the above solution. Similarly, the encoder may generate a coded representation by determining whether a particular syntax field should or should not be included and accordingly including or excluding that syntax field in the coded representation.

[0163] Implementations of the disclosed and other solutions, examples, embodiments, modules, and functional operations described herein, including the structures disclosed herein and their structural equivalents, may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or in one or more combinations thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for implementation by or for controlling the operation of a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors, or computers. In addition to hardware, the apparatus may include code that creates an environment for the execution of the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, for example a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiving device.

[0164] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be recorded as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), may be stored in a single file dedicated to the program, or may be stored in multiple coordinating files (e.g., files that store one or more modules, subprograms, or code portions). A computer program can be deployed to run on one computer, or it can be deployed to run on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0165] The processing and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processing and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).

[0166] Processors suitable for executing computer programs include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer may include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or may be operatively coupled to receive data from or transfer data to these mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, EPROMs, EEPROMs, flash storage devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and semiconductor storage devices such as CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0167] Although this patent specification includes many features, these should not be construed as limiting the scope of any subject matter or the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described in this patent document in the context of separate embodiments may be implemented in combination in one example. Conversely, various features described in the context of one example may be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be extracted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0168] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desired results, nor should the separation of various system components in the embodiments described in this patent specification be understood as requiring such separation in all embodiments.

[0169] Only certain implementations and examples have been described; other embodiments, extensions and variations are possible based on the content described and illustrated in this patent document.

Claims

1. 1. A method for processing video data, comprising the steps of: performing a conversion between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule; A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information describing a level of sub-picture sequences in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies, and a layer in which a reference sequence parameter set in one or more OLSs has a first syntax element with a specific value indicating that the number of sub-pictures is greater than one is called a multi-sub-picture layer; A coded video sequence (CVS) in the set of one or more OLSs is called a target CVS; A method for processing video data, wherein the rule specifies that a first subpicture sequence of the subpicture sequences includes (1) all subpictures in a target CVS that have the same subpicture index and are included in a layer in the multi-subpicture layer, and (2) all subpictures in a target CVS that have a subpicture index of 0 and are included in a layer of an OLS but not in the multi-subpicture layer.

2. 2. The method of claim 1 , wherein the rules specify that bitstream conformance requires that a value of the first syntax element be the same for all sequence parameter sets referenced by pictures of the multiple sub-picture layer in response to the presence of the SLI SEI message for a coded video sequence.

3. 3. The method of claim 1 or 2, wherein in response to the presence of an SLI SEI message for any access unit of a coded video sequence (CVS) of one or more OLSs, the rule specifies that the SLI SEI message is present for a first access unit of the CVS.

4. The method of claim 1 , wherein the SLI SEI message is allowed to be present for the current access unit by either being included in the bitstream or being provided externally.

5. The method of claim 1 , wherein a second syntax element in the syntax structure of the SLI SEI message specifies a number of sub-pictures in a picture in a multi-sub-picture layer in a target CVS.

6. 6. The method of claim 1, wherein the SLI SEI messages continue in decoding order from a current access unit to the end of the bitstream or to a next access unit that includes a subsequent SLI SEI message that includes different content than the SLI SEI message.

7. The method of claim 1 , wherein the rules stipulate that all SLI SEI messages applicable to the same CVS have the same content.

8. A method according to any one of claims 1 to 7, wherein a first variable indicating an index of a sub-picture level is defined to contain a value for each sub-picture sequence.

9. A method according to any one of claims 1 to 8, wherein a second variable indicating a sub-picture level index is defined to contain a value for each sub-picture sequence.

10. The method of claim 1 , wherein the converting comprises encoding the video into the bitstream.

11. The method of claim 1 , wherein the converting comprises decoding the video from the bitstream.

12. 1. An apparatus for processing video data, comprising: a processor; and a non-transitory memory comprising instructions that, when executed by the processor, cause the processor to: performing a conversion between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule; A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information describing a level of sub-picture sequences in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies, and a layer in which a reference sequence parameter set in one or more OLSs has a first syntax element with a specific value indicating that the number of sub-pictures is greater than one is called a multi-sub-picture layer; A coded video sequence (CVS) in the set of one or more OLSs is called a target CVS; An apparatus for processing video data, wherein the rule specifies that a first subpicture sequence of the subpicture sequences includes (1) all subpictures in a target CVS that have the same subpicture index and are included in a layer in the multi-subpicture layer, and (2) all subpictures in a target CVS that have a subpicture index of 0 and are included in a layer of an OLS but not in the multi-subpicture layer.

13. A non-transitory computer-readable storage medium storing instructions, the instructions configuring a processor to: performing a conversion between a current access unit of a video having one or more output layer sets (OLS) and a bitstream of the video according to a rule; A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information describing a level of sub-picture sequences in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies, and a layer in which a reference sequence parameter set in one or more OLSs has a first syntax element with a specific value indicating that the number of sub-pictures is greater than one is called a multi-sub-picture layer; A coded video sequence (CVS) in the set of one or more OLSs is called a target CVS; A non-transitory computer-readable storage medium, wherein the rule specifies that a first sub-picture sequence of the sub-picture sequences includes (1) all sub-pictures in a target CVS that have the same sub-picture index and are included in a layer in the multi-sub-picture layer, and (2) all sub-pictures in a target CVS that have a sub-picture index of 0 and are included in a layer of an OLS but not in the multi-sub-picture layer.

14. 1. A method for storing a video bitstream, comprising the steps of: generating a bitstream of a video from the video having one or more output layer sets (OLS) according to a rule; storing the bitstream on a non-transitory computer-readable recording medium; A sub-picture level information (SLI) supplemental enhancement information (SEI) message includes information describing a level of sub-picture sequences in a set of coded video sequences of one or more OLSs to which the SLI SEI message applies, and a layer in which a reference sequence parameter set in one or more OLSs has a first syntax element with a specific value indicating that the number of sub-pictures is greater than one is called a multi-sub-picture layer; A coded video sequence (CVS) in the set of one or more OLSs is called a target CVS; The method, wherein the rule specifies that a first subpicture sequence of the subpicture sequences includes (1) all subpictures in the target CVS that have the same subpicture index and are included in a layer in the multi-subpicture layer, and (2) all subpictures in the target CVS that have a subpicture index of 0 and are included in a layer of an OLS but not in the multi-subpicture layer.

Images