Video processing method, video data processing device, and medium

Abstract

A method for video processing is described. The method includes: performing a conversion between a video including one or more pictures and a bitstream representation of the video according to a rule, the one or more pictures including one or more sub-pictures, the one or more sub-pictures including one or more stripes, and wherein the bitstream representation includes a plurality of coding / decoding units, and wherein the rule specifies that the decoding order of the coding / decoding units within a sub-picture is in ascending order of the sub-picture level stripe index value of the coding / decoding units.

Description

[0001] Cross - Reference to Related Applications

[0002] Under the provisions of the applicable patent laws and / or the Paris Convention, this application timely claims the priority and benefits of U.S. Provisional Application No. 62 / 953,812, filed on December 26, 2019, and U.S. Provisional Application No. 62 / 954,375, filed on December 27, 2019. For all legal purposes, the entire disclosure of the above applications is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video coding and decoding technologies, systems, and devices. Background Art

[0004] In the Internet and other digital communication networks, digital video occupies the largest bandwidth. With the increase in the number of connected user devices capable of receiving and displaying video, the bandwidth demand for digital video use is expected to continue to grow. Summary of the Invention

[0005] This disclosure describes devices, systems, and methods related to digital video coding and decoding, including specifying the decoding order of video coding layer (VCL) network abstraction layer (NAL) units within coded pictures. The described methods can be applied to existing video coding and decoding standards (e.g., High Efficiency Video Coding (HEVC) and / or Versatile Video Coding (VVC)) and future video coding and decoding standards or video codecs.

[0006] In a representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: performing a conversion between a video including one or more pictures and a bit - stream representation of the video according to a rule, the one or more pictures including one or more sub - pictures, the one or more sub - pictures including one or more strips, and wherein the bit - stream representation includes a number of coding and decoding units, and wherein the rule specifies that the decoding order of the coding and decoding units within a sub - picture is in ascending order of the sub - picture - level strip index value of the coding and decoding units.

[0007] In a representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: performing a conversion between a video including one or more pictures and a bit - stream representation of the video according to a rule, the one or more pictures including one or more sub - pictures, the one or more sub - pictures including one or more strips, and wherein the bit - stream representation includes a number of coding and decoding units, and wherein the rule specifies that the decoding order of the coding and decoding units is in ascending order of the sub - picture - related value of the sub - pictures from one or more sub - pictures including the coding and decoding units.

[0008] In another representative aspect, the above method is implemented in the form of processor-executable code and stored in a computer-readable program medium.

[0009] In yet another representative aspect, a device configured or operable to execute the above method is disclosed. The device may include a processor programmed to implement this method.

[0010] In yet another representative aspect, a video decoder device may implement the method as described herein.

[0011] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the description, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of a picture with 18×12 luminance coding tree units (CTUs) is shown, which is segmented into 12 slices and 3 raster scan stripes.

[0013] Figure 2 An example of a picture with 18×12 luminance CTUs is shown, which is segmented into 24 slices and 9 rectangular stripes.

[0014] Figure 3 An example of a picture segmented into 4 slices and 4 rectangular stripes is shown.

[0015] Figure 4 An example of a picture segmented into 15 slices, 24 stripes, and 24 sub-pictures is shown.

[0016] Figure 5 A flowchart of an example method of video processing is shown.

[0017] Figure 6 A block diagram of an example of a video processing device is shown.

[0018] Figure 7 A block diagram of an example video coding / decoding system is shown.

[0019] Figure 8 A block diagram of an example encoder is shown.

[0020] Figure 9 A block diagram of an example decoder is shown.

[0021] Figure 10 A block diagram of an example video processing system that can implement the disclosed technology is shown.

[0022] Figure 11 A flowchart of an example method of video processing based on some implementations of the disclosed technology is shown. DETAILED DESCRIPTION

[0023] Due to the increasing demand for higher-resolution videos, video encoding and decoding methods and technologies are prevalent in modern technologies. A video codec typically includes electronic circuits or software for compressing or decompressing digital videos and is continuously improved to provide higher encoding and decoding efficiency. A video codec converts an uncompressed video into a compressed format and vice versa. There is a complex relationship among video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, and end-to-end latency (delay). Compressed formats generally conform to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Generic Video Coding standard to be completed, or other current and / or future video coding standards.

[0024] Embodiments of the disclosed technology can be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve compression performance. It specifically relates to the merge mode in video encoding and decoding. Section headings are used in this document to enhance the readability of the description and do not limit the discussion or embodiments (and / or implementations) to only the corresponding sections in any way.

[0025] 1. Overview of Example Embodiments

[0026] Embodiments of the disclosed technology are directed to the decoding order of video coding layer (VCL) network abstraction layer (NAL) units within a coded picture specified in a coded video bitstream. It can be applied to any video coding standard that supports splitting a picture into slices and sub-pictures, such as the Generic Video Coding (VVC) being developed, or any other video coding standard or video codec.

[0027] 2. List of Abbreviations Used in This Document

[0028] APS Adaptive Parameter Set

[0029] AU Access Unit

[0030] AUD Access Unit Delimiter

[0031] AVC Advanced Video Coding

[0032] CRA Clean Random Access

[0033] CTU Coding Tree Unit

[0034] CVS Coded Video Sequence

[0035] DPS Decoding Parameter Set

[0036] EOB End Of Bitstream

[0037] EOS End Of Sequence

[0038] GDR Gradual Decoding Refresh

[0039] HEVC High Efficiency Video Coding

[0040] IDR Instantaneous Decoding Refresh

[0041] JEM Joint Exploration Model

[0042] MCTS Motion-Constrained Tile Sets

[0043] NAL Network Abstraction Layer

[0044] PH Picture Header

[0045] PPS Picture Parameter Set

[0046] PU Picture Unit

[0047] RBSP Raw Byte Sequence Payload

[0048] SEI Supplemental Enhancement Information

[0049] SPS Sequence Parameter Set

[0050] VCL (Video Coding Layer)

[0051] VPS (Video Parameter Set)

[0052] VTM (VVC Test Model)

[0053] VUI (Video Usability Information)

[0054] VVC (Versatile Video Coding)

[0055] 3. Background

[0056] Video coding standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, and ISO / IEC developed MPEG-1 and MPEG-4 Visual. The two organizations jointly developed the H.262 / MPEG-2 video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure that utilizes temporal prediction plus transform coding. To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and incorporated them into a reference software called the Joint Exploration Model (JEM). JVET meetings are held quarterly. Compared with HEVC, the goal of the new coding standard is to reduce the bit rate by 50%. The new video coding standard was officially named Versatile Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time.

[0057] 3.1 Picture segmentation techniques in HEVC

[0058] HEVC includes four different image segmentation schemes, namely regular slice, dependent slice, tile, and Wavefront Parallel Processing (WPP), which can be used for maximum transmission unit (MTU) size matching, parallel processing, and reducing end-to-end latency.

[0059] Regular slices are similar to those in H.264 / AVC. Each regular slice is encapsulated in its own NAL unit, and in-picture prediction (intra-sample prediction, motion information prediction, codec mode prediction) and entropy coding / decoding dependencies across slice boundaries are disabled. Thus, a regular slice can be reconstructed independently of other regular slices within the same picture (although there may still be dependencies due to loop filter operations).

[0060] Regular slices are the only tool available for parallelization and are also available in H.264 / AVC in almost the same form. Parallelization based on regular slices requires little inter-processor or inter-core communication (except for the inter-processor or inter-core data sharing for motion compensation when decoding predicted codec pictures, which is typically much more severe due to in-picture prediction). However, for the same reason, using regular slices results in a large amount of codec overhead due to the bit cost of slice headers and the lack of prediction across slice boundaries. In addition, due to the in-picture independence of regular slices and each regular slice being encapsulated in its own NAL unit, regular slices (compared to other tools mentioned below) also serve as a key mechanism for bitstream segmentation to match MTU size requirements. In many cases, the goals of parallelization and MTU size matching pose conflicting requirements for the slice layout in a picture. Awareness of this situation led to the development of the parallelization tools mentioned below.

[0061] Dependency slices have short slice headers and allow the bitstream to be segmented at tree block boundaries without breaking any in-picture prediction. Basically, dependency slices provide a way to split a regular slice into multiple NAL units to provide reduced end-to-end latency by allowing a portion of a regular slice to be sent out before the encoding of the entire regular slice is complete.

[0062] In WPP, a picture is segmented into single-row codec tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other segments. Through parallel decoding of CTB rows, parallel processing is possible, where the start of decoding of a CTB row is delayed by two CTBs to ensure that data related to CTBs above and to the right of the target CTB is available before the target CTB is decoded. Using this staggered start (which looks like a wavefront when represented graphically), parallelization can use as many processors / cores as there are CTB rows in the picture. Since in-picture prediction between adjacent tree block rows within a picture is allowed, the inter-processor / inter-core communication required to implement in-picture prediction can be significant. Compared to when it is not applied, WPP segmentation does not produce additional NAL units, so WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular slices can be used with WPP, but there will be some codec overhead.

[0063] Tiles define the horizontal and vertical boundaries that divide the picture into tile columns and tile rows. Tile columns extend from the top to the bottom of the picture. Similarly, tile rows extend from the left to the right of the picture. The number of tiles in a picture can be simply obtained by multiplying the number of tile columns by the number of tile rows.

[0064] Before decoding the top-left CTB of the next tile in the raster scan order of the picture tiles, the scan order of CTBs is changed to be local within the tile (in the raster scan order of the tile's CTBs). Similar to regular slices, tiles break the prediction dependency and entropy decoding dependency within the picture. However, they do not need to be included in a single NAL unit (the same as WPP in this regard); thus, tiles cannot be used for MTU size matching. Each tile can be processed by one processor / core, and the inter-processor / inter-core communication required for intra-picture prediction between processing units decoding adjacent tiles is limited to transmitting the shared slice header in cases where a slice spans more than one tile, and loop filtering related to the sharing of reconstructed samples and metadata. When a slice or WPP segment contains more than one tile, the byte offset of the entry point of each slice or WPP segment except the first one in the slice is signaled in the slice header.

[0065] For simplicity, HEVC specifies restrictions applied to four different picture partitioning schemes. For most profiles specified in HEVC, a given coded video sequence cannot contain both tiles and wavefronts simultaneously. For each slice and tile, one or both of the following conditions must be satisfied: 1) all coded tree blocks in the slice belong to the same tile; 2) all coded tree blocks in the tile belong to the same slice. Finally, a wavefront segment exactly contains one CTB row, and when using WPP, if a slice starts at a CTB row, it must end at the same CTB row.

[0066] The latest modifications to HEVC are specified in the JCT-VC output file JCTVC-AC1005, publicly released on October 24, 2017, J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, Y.-K. Wang (editors), “HEVC Additional Supplemental Enhancement Information (Draft 4)”. The public information can be referenced at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. With this revision, HEVC specifies three SEI messages related to MCTS, namely the Temporal MCTS SEI message, the MCTS Extraction Information Set SEI message, and the MCTS Extraction Information Nested SEI message.

[0067] The Temporal MCTS SEI message indicates the presence of MCTS in the bitstream and signals the MCTS. For each MCTS, the motion vectors are restricted to point to full-sample positions within the MCTS and fractional-sample positions that only require full-sample positions within the MCTS for interpolation, and motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS are not allowed. In this way, each MCTS can be decoded independently without slices that are not included in the MCTS.

[0068] The MCTS Extraction Information Set SEI message provides supplementary information (specified as part of the SEI message semantics) that can be used in MCTS sub-bitstream extraction to generate a bitstream conforming to the MCTS set. This information consists of multiple extraction information sets, each of which defines multiple MCTS sets and contains the RBSP bytes of the replacement VPS, SPS, and PPS to be used during the MCTS sub-bitstream extraction process. When extracting the sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice headers need to be slightly updated because one or all of the syntax elements related to the slice address (including first_slice_segment_in_pic_flag and slice_segment_address) usually need to have different values.

[0069] 3.2 Picture Segmentation in VVC

[0070] In VVC, a picture is partitioned into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs that cover a rectangular region of the picture. The multiple CTUs in a slice are scanned in raster scan order within that slice.

[0071] A slice consists of an integer number of consecutive complete CTU rows within an integer number of complete tiles or tiles of a picture.

[0072] Two slice modes are supported, namely the raster scan slice mode and the rectangular slice mode. In the raster scan slice mode, a slice contains a series of complete tiles in the tile raster scan of a picture. In the rectangular slice mode, a slice contains multiple complete tiles that together form a rectangular region of a picture, or multiple consecutive complete CTU rows of a single tile that together form a rectangular region of a picture. The tiles within a rectangular slice are scanned in tile raster scan order within the rectangular region corresponding to that slice.

[0073] A sub - picture contains one or more slices that together cover a rectangular region of a picture.

[0074] Figure 1 An example of the raster scan slice partitioning of a picture is shown, where the picture is divided into 12 tiles and 3 raster scan slices.

[0075] Figure 2 An example of the rectangular slice partitioning of a picture is shown, where the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices.

[0076] Figure 3 An example of a picture partitioned into tiles and rectangular slices is shown, where the picture is divided into 4 tiles (2 tile columns and 2 tile rows) and 4 rectangular slices.

[0077] Figure 4 An example of the sub - picture partitioning of a picture is shown, where the picture is divided into 18 tiles. Each of the 12 left - hand - side tiles covers a strip of 4×4 CTUs, and each of the 6 right - hand - side tiles covers 2 vertically - stacked strips of 2×2 CTUs, resulting in a total of 24 strips and 24 sub - pictures of different dimensions (each strip is a sub - picture).

[0078] 3.3 Signaling of Sub - pictures, Slices, and Tiles in VVC

[0079] In the latest VVC draft text, the information of sub - pictures is signaled in the SPS, which includes the sub - picture layout (i.e., the number of sub - pictures per picture and the position and size of each sub - picture) and other sequence - level sub - picture information. The order of the sub - pictures signaled in the SPS defines the sub - picture index. For example, a list of sub - picture IDs can be explicitly signaled in the SPS or PPS, and each sub - picture has an ID.

[0080] Slices in VVC are conceptually the same as those in HEVC, i.e., each picture is divided into slice columns and slice rows, but different syntax is used for slice signaling in the PPS.

[0081] In VVC, the slice mode is signaled in the PPS. When the slice mode is the rectangular slice mode, the slice layout of each picture (i.e., the number of slices per picture and the position and size of each slice) is signaled in the PPS. The order of the rectangular slices within the picture signaled in the PPS defines the picture-level slice index. The sub-picture-level slice index is defined as the order of the slices within the sub-picture in ascending order of the picture-level slice index. The position and size of the rectangular slices are signaled / derived based on the sub-picture position and size signaled in the SPS (when each sub-picture contains only one slice), or based on the tile position and size signaled in the PPS (when a sub-picture may contain more than one slice). When the slice mode is the raster scan slice mode, similar to HEVC, the slice layout within pictures with different details is signaled within the slices themselves.

[0082] 3.4 Sub-picture ID and slice address for coded slices in VVC

[0083] In VVC, a VCL NAL unit corresponds to a coded slice NAL unit. Each coded slice includes a slice header, which includes a sub-picture ID (slice_subpic_id) and a slice address (slice_address). This pair of parameters indicates the position of the samples coded in the slice within the picture.

[0084] If the slice mode is the rectangular slice mode (i.e., rect_slice_flag equals 1), the slice address specifies the (sub-picture-level) slice index of the slice within the sub-picture's slices.

[0085] Otherwise (rect_slice_flag equals 0, the slice mode is the raster scan slice mode, and in this case, the whole picture is a sub-picture), the slice address specifies the tile index within the picture.

[0086] For convenience, the semantics of slice_subpic_id and slice address, which are part of the general slice header semantics, are listed below.

[0087] 7.4.8.1 General slice header semantics

[0088] When present, the value of the slice header syntax element slice_pic_order_cnt_lsb should be the same in all slice headers of the coded picture. ...

[0089] The slice_subpic_id specifies the sub-picture identifier of the sub-picture containing the slice. If slice_subpic_id exists, the value of the derived variable SubPicIdx is such that SubpicIdList[SubPicIdx] is equal to slice_subpic_id. Otherwise (slice_subpic_id does not exist), the variable SubPicIdx is derived to be equal to 0. The length of slice_subpic_id (in bits) is derived as follows:

[0090] - If sps_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to

[0091] sps_subpic_id_len_minus1 + 1.

[0092] - Otherwise, if ph_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to

[0093] ph_subpic_id_len_minus1 + 1.

[0094] - Otherwise, if pps_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to pps_subpic_id_len_minus1 + 1.

[0095] - Otherwise, the length of slice_subpic_id is equal to Ceil(Log2(sps_num_subpics_minus1 + 1)).

[0096] The slice_address specifies the slice address of the slice. When it does not exist, the value of slice_address is inferred to be equal to 0.

[0097] If rect_slice_flag is equal to 0, the following conditions apply:

[0098] - The slice address is the raster scan slice index.

[0099] - The length of slice_address is Ceil(Log2(NumTilesInPic)) bits.

[0100] The value of slice_address shall be in the range of 0 to NumTilesInPic-1, inclusive of 0 and NumTilesInPic-1.

[0101] Otherwise (when rect_slice_flag equals 1), the following applies:

[0102] - The slice address is the slice index of the slice within the SubPicIdx-th sub-picture.

[0103] - The length of slice_address is Ceil(Log2(NumSlicesInSubpic[SubPicIdx])) bits.

[0104] - The value of slice_address shall be in the range of 0 to NumSlicesInSubpic[SubPicIdx]-1, inclusive of 0 and

[0105] NumSlicesInSubpic[SubPicIdx]-1.

[0106] The following constraints are applied for bitstream consistency:

[0107] - If rect_slice_flag equals 0 or subpics_present_flag equals 0, the value of slice_address shall not be equal to the value of slice_address of any other decoded slice NAL unit of the same decoded picture.

[0108] - Otherwise, the pair of slice_subpic_id and slice_address values shall not be equal to the pair of slice_subpic_id and slice_address values of any other decoded slice NAL unit of the same decoded picture.

[0109] - When rect_slice_flag equals 0, the slices of the picture shall be arranged in ascending order of their slice_address values.

[0110] - The shape of the picture slices should be such that when decoded, the entire left boundary and the entire upper boundary of each CTU should be composed of the picture boundary or the boundary of previously decoded CTUs. ...

[0111] 3.5 Decoding Order of NAL Units

[0112] In VVC, in Clause 7.4.2.4 and its sub-clauses of the latest VVC draft text, as well as in the SPS semantics (Clause 7.4.3.3) and the general strip header semantics (Clause 7.4.8.1), the decoding order of NAL units (i.e., the order of NAL units in the VVC bitstream) is specified. For convenience, the relevant texts are copied and pasted below.

[0113] 7.4.2.4 Order of NAL Units, and Its Relationship with Encoded / Decoded Pictures, PUs, AUs, and Encoded Video Sequence 7.4.2.4.1 Overview

[0114] This clause specifies the constraints on the order of NAL units in the bitstream.

[0115] Any order of NAL units in the bitstream that complies with these constraints is referred to as the decoding order of NAL units in the text. In NAL units, the syntax in Clauses 7.3 and D.2 specifies the decoding order of syntax elements. When any SEI message or VUI parameter specified in ITU-T H.SEI|ISO / IEC 23002-7 is included in the NAL units specified in this specification, the syntax of the SEI message or VUI parameter specified in ITU-T H.SEI|ISO / IEC 23002-7 specifies the decoding order of those syntax elements. The decoder shall be able to receive NAL units and their syntax elements in the decoding order.

[0116] 7.4.2.4.2 Order of AUs and Its Relationship with CVS

[0117] A bitstream compliant with this specification consists of one or more CVSs.

[0118] A CVS consists of one or more AUs. The order of NAL units and encoded / decoded pictures and their association with AUs are described in Section 7.4.2.4.3.

[0119] The first AU of a CVS is the CVSS AU, where each existing PU is a CLVSS PU, which is an IRAP PU with NoIncorrectPicOutputFlag equal to 1 or a GDRPU with NoIncorrectPicOutputFlag equal to 1.

[0120] Each CVSS AU shall have a picture at each layer in the CVS.

[0121] The requirement for bitstream conformance is that when present, each PU in the next AU after an AU containing an EOS NAL unit or an EOB NAL unit should be an IRAP PU, which can be an IDR PU or a CRA PU, or a GDRPU.

[0122] 7.4.2.4.3 NAL Unit and Coding Picture Order and Their Relationship with PUs and AUs

[0123] This clause specifies the order of NAL units and coded pictures and their association with PUs and AUs of a CVS that conforms to one or more profiles specified in Annex A and is decoded using the decoding processes specified in Clauses 2 to 9.

[0124] A PU consists of a PH NAL unit and a coded picture that includes one or more VCL NAL units and zero or more non-VCL NAL units. The association of VCL NAL units with the coded picture is described in Clause 7.4.2.4.4.

[0125] An AU consists of zero or one AU delimiter NAL unit and one or more PUs, which are arranged in ascending order of nuh_layer_id.

[0126] The first AU in the bitstream starts from the first NAL unit in the bitstream. There can be at most one AU delimiter NAL unit in an AU.

[0127] The first VCL NAL unit of a picture is the first VCL NAL unit that follows the PH NAL unit in the decoding order of the picture.

[0128] When a VCL NAL unit is the first VCL NAL unit of a picture and one or more of the following conditions hold, the VCL NAL unit is the first VCL NAL unit of an AU (and thus the picture containing the first VCL NAL unit is the first picture of the AU):

[0129] – The value of nuh_layer_id of the VCL NAL unit is less than that of the previous picture in decoding order.

[0130] – The value of slice_pic_order_cnt_lsb of the VCL NAL unit is different from the PicOrderCntVal of the previous picture in decoding order.

[0131] – The PicOrderCntVal derived for the VCL NAL unit is different from the PicOrderCntVal of the previous picture in decoding order.

[0132] Let firstVclNalUnitInAu be the first VCL NAL unit of an AU. The first of any of the following NAL units, if any, that comes before firstVclNalUnitInAu and after the last VCL NAL unit before firstVclNalUnitInAu specifies the start of a new access unit:

[0133] – An AUD NAL unit, if any,

[0134] – A DPS NAL unit, if any,

[0135] – A VPS NAL unit, if any,

[0136] – An SPS NAL unit, if any,

[0137] – A PPS NAL unit, if any,

[0138] – A pre-APS NAL unit, if any,

[0139] – A PH NAL unit,

[0140] – A pre-SEI NAL unit, if any,

[0141] – A NAL unit with nal_unit_type equal to RSV_NVCL_26, when present,

[0142] – A NAL unit with nal_unit_type in the range UNSPEC28 to UNSPEC29, when present.

[0143] Note - If any, the first NAL unit that comes before firstVclNalUnitInAu and after the last VCL NAL unit before firstVclNalUnitInAu can only be one of the NAL units listed above.

[0144] The order of coded pictures and non-VCL NAL units within a PU or AU shall comply with the following constraints:

[0145] – When an AU delimiter NAL unit is present in an AU, it shall be the first NAL unit of the AU.

[0146] – The PH NAL unit in a PU shall come before the first VCL NAL of the PU.

[0147] – When any DPS NAL unit, VPS NAL unit, SPS NAL unit, PPS NAL unit, prefix APS NAL unit, prefix SEI NAL unit, NAL unit with nal_unit_type equal to RSV_NVCL_26 or NAL unit with nal_unit_type in the range of UNSPEC_28 to UNSPEC_29 exists in a PU, they shall not follow after the last VCL NAL unit of the PU.

[0148] – When any DPS NAL unit, VPS NAL unit, SPS NAL unit or PPS NAL unit exists in a PU, they shall be before the PH NAL unit of the PU.

[0149] – NAL units in a PU with nal_unit_type equal to SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27, or in the range of UNSPEC_30 to UNSPEC_31 shall not be before the first VCL NAL unit.

[0150] – When an EOS NAL unit exists in a PU, it shall be the last NAL unit among all NAL units in the PU except the EOB NAL unit (if it exists).

[0151] – When an EOB NAL unit exists in an AU, it shall be the last NAL unit in the AU.

[0152] 7.4.3.3 Sequence parameter set RBSP semantics

[0153] The SPS RBSP shall be available for the decoding process before being referenced, including in at least one AU with TemporalId equal to 0 or provided externally. ...

[0154] The following constraint requirements are applied for bitstream consistency:

[0155] – For any two sub-pictures subpicA and subpicB, when the sub-picture index of subpicA is less than the sub-picture index of subpicB, any coded or decoded slice NAL unit of subPicA shall be before any coded or decoded slice NAL unit of subPicB in the decoding order.

[0156] – ... ...

[0157] 7.4.8.2 General slice header semantics

[0158] When present, the value of the slice_pic_order_cnt_lsb syntax element in the slice header shall be the same for all slice headers of the coded picture. ...

[0159] The following constraints are applied for bitstream compliance:

[0160] – ...

[0161] – When rect_slice_flag is equal to 0, the slices of the picture shall be ordered in ascending order of their slice_address values.

[0162] – ... ...

[0163] 4. Deficiencies of Existing Implementations

[0164] The existing VVC designs have the following problems:

[0165] (1) When the slice mode is the rectangular slice mode, the decoding order of the VCL NAL units within the sub - picture (i.e., the coded slice NAL units) is not specified. Thus, when the slice mode is the rectangular slice mode, compliant decoder implementations need to be tested and ensured that they can correctly decode bitstreams with VCL NAL units in any order within the sub - picture. This will impose a heavy burden on decoder implementations, such as during the implementation for compliance testing.

[0166] (2) When slice_subpic_id does not exist in the slice header, the value needs to be inferred, for example, the value is needed to specify the NAL unit decoding order of the VCL NAL units within the picture.

[0167] (3) The sub - picture ID values signaled explicitly in the SPS or PPS need to be restricted to avoid any order of sub - picture ID values relative to the increasing order of the sub - picture index values, which will also increase the decoder implementation burden in compliance testing, etc.

[0168] 5. Example Embodiments of the Present Disclosure

[0169] The following detailed embodiments should be considered as examples for explaining general concepts. These embodiments should not be interpreted narrowly. Additionally, these embodiments can be combined in any way.

[0170] 1) When the slice mode is the rectangular slice mode, the decoding order of the VCL NAL units within the sub - picture (i.e., the coded slice NAL units) is specified as the ascending order of the sub - picture ID values of the sub - picture containing the VCL NAL units.

[0171] Alternatively, when the stripe pattern is a rectangular stripe pattern, the decoding order of VCL NAL units (i.e., coded / decoded stripe NAL units) within a sub-picture is specified as the ascending order of the sub-picture index values of the sub-picture containing the VCL NAL units.

[0172] 2) When the stripe pattern is a rectangular stripe pattern, the decoding order of VCL NAL units (i.e., coded stripe NAL units) within a sub-picture is specified as the ascending order of the sub-picture level stripe index values of the VCL NAL units. Note that the sub-picture level stripe index value of the coded / decoded stripe NAL unit is actually the value of the slice_address syntax element in the stripe header.

[0173] 3) When slice_subpic_id does not exist, the value of slice_subpic_id is inferred, for example, the value of slice_subpic_id is inferred to be equal to 0.

[0174] 4) It is required that the values of the sub-picture IDs increase in ascending order of the sub-picture indices.

[0175] Note that in the latest VVC draft text, it has been stipulated that the decoding order of any two VCL NAL units (i.e., coded / decoded stripe NAL units) within a picture but belonging to different sub-pictures is in ascending order of the sub-picture index values of the sub-picture containing the VCL NAL units.

[0176] After adding this constraint, the decoding order of any two VCL NAL units within a picture but belonging to different sub-pictures can be specified as the ascending order of the sub-picture ID values of the sub-picture containing the VCL NAL units.

[0177] The examples described above can be incorporated into the context of the method (e.g., method 500) described below, which can be implemented at a video decoder or a video encoder.

[0178] Figure 5 A flowchart of an example method 500 for video processing is shown. The method includes, at operation 510, determining that the stripe pattern of the stripe including the current video segment is a rectangular stripe pattern for the conversion between the current video segment of a video and the bitstream representation of the video including a plurality of video coded layer (VCL) network abstraction layer (NAL) units.

[0179] The method includes, at operation 520, performing the conversion based on the determination, wherein the bitstream representation further includes one or more syntax elements that signal the decoding order of the plurality of VCL NAL units.

[0180] 6 Additional Example Embodiments

[0181] The following are some example embodiments that can be applied to the VVC standard. The changed text is based on the latest VVC text in JVET-P2001-v14. The added, modified, and most relevant parts are marked with marked, and the deleted parts are enclosed in [[double bold brackets]]. There are also some other changes that are editorial and thus not emphasized or marked differently.

[0182] 6.1 First Embodiment

[0183] 6.1.1 Definitions (VVC Clause 3) ...

[0184] ...

[0185] ...

[0186] 6.1.2 Definitions of CTB raster scan, slice scan, and sub-picture scan processes (VVC Clause 6.5.1) ...

[0187] Specify the number of rectangular stripes in the i-th sub-picture and the lists NumSlicesInSubpic[i] and SliceSubpicToPicIdx[i][k] of the k-th stripe in the i-th sub-picture are derived as follows:

[0188]

[0189] 6.1.3 Order of NAL units in the bitstream (VVC Clause 7.4.2.4)

[0190] 7.4.2.4 Order of NAL units in the bitstream

[0191] 7.4.2.4.1 Overview

[0192] The sub-clauses of Clause 7.4.2.4 specify the constraints on the order of NAL units in the bitstream. Any order of NAL units in the bitstream that complies with these constraints is referred to in the text as the decoding order of NAL units.

[0193] In the NAL unit, the syntax in clauses 7.3 and D.2 specifies the decoding order of syntax elements. When any SEI message or VUI parameter specified in ITU-T H.SEI|ISO / IEC 23002-7 is included in the NAL unit specified in this specification, the syntax that specifies the SEI message or VUI parameter in ITU-T H.SEI|ISO / IEC 23002-7 specifies the decoding order of those syntax elements. The decoder shall be able to receive the NAL unit and its syntax elements in the decoding order.

[0194] 7.4.2.4.2 Order of AUs and its relationship with CVS

[0195] The bitstream consists of one or more CVSs. A CVS consists of one or more AUs. The order of PUs and their relationship with AUs are described in clause 0.

[0196] The first AU of a CVS is the CVSS AU, where each existing PU is a CLVSS PU, which is an IRAP PU with NoIncorrectPicOutputFlag equal to 1 or a GDRPU with NoIncorrectPicOutputFlag equal to 1.

[0197] Each CVSS AU shall have PUs for each layer present in the CVS.

[0198] The requirement for bitstream conformance is that when present, the next AU after the AU containing the EOB NAL unit shall be the CVSS AU.

[0199] 7.4.2.4.3 Order of PUs and its relationship with AU

[0200] An AU consists of one or more PUs in ascending order of nuh_layer_id. The order of NAL units and their relationship with coded and decoded pictures and their relationship with PUs are described in clause 7.4.2.4.4.

[0201] There can be at most one AUD NAL unit in an AU. When an AUD NAL unit is present in an AU, it shall be the first NAL unit of the AU, and thus, it is the first NAL unit of the first PU of the AU.

[0202] There can be at most one EOB NAL unit in an AU. When an EOB NAL unit is present in an AU, it shall be the last NAL unit of the AU, and thus, it is the last NAL unit of the last PU of the AU.

[0203] When the VCL NAL unit is the first VCL NAL unit after a PH NAL unit and one or more of the following conditions are true, the VCL NAL unit is the first VCL NAL unit of an AU (and thus the PU containing the VCL NAL unit is the first PU of an AU):

[0204] – The value of nuh_layer_id of the VCL NAL unit is less than the nuh_layer_id of the previous picture in decoding order.

[0205] – The value of slice_pic_order_cnt_lsb of the VCL NAL unit is different from the PicOrderCntVal of the previous picture in decoding order.

[0206] – The PicOrderCntVal derived for the VCL NAL unit is different from the PicOrderCntVal of the previous picture in decoding order.

[0207] Let firstVclNalUnitInAu be the first VCL NAL unit of an AU. Any NAL unit that comes before firstVclNalUnitInAu and is the first, if any, after the last VCL NAL unit before firstVclNalUnitInAu specifies the start of a new AU:

[0208] – AUD NAL unit (if any),

[0209] – DPS NAL unit (if any),

[0210] – VPS NAL unit (if any),

[0211] – SPS NAL unit (if any),

[0212] – PPS NAL unit (if any),

[0213] – Preceding APS NAL unit (if any),

[0214] – PH NAL unit,

[0215] – Preceding SEI NAL unit (if any),

[0216] – NAL unit with nal_unit_type equal to RSV_NVCL_26 (when present),

[0217] – NAL unit with nal_unit_type in the range of UNSPEC28 to UNSPEC29 (when present).

[0218] Note - If any, the first NAL unit before firstVclNalUnitInAu and the first NAL unit after the last VCL NAL unit before firstVclNalUnitInAu can only be one of the NAL units listed above.

[0219] The requirement for bitstream conformance is that when present, the next PU in a particular layer after a PU belonging to the same layer and containing an EOS NAL unit or an EOB NAL unit shall be a CLVSS PU, which can be an IRAP PU with NoIncorrectPicOutputFlag equal to 1 or a GDR PU with NoIncorrectPicOutputFlag equal to 1.

[0220] 7.4.2.4.4 NAL unit and coded picture order and their relationship to PUs

[0221] A PU includes a PH NAL unit, a coded picture (which includes one or more VCL NAL units), and zero or more other non-VCL NAL units. The relationship between VCL NAL units and coded pictures is described in Clause 7.4.2.4.4.

[0222] The first VCL NAL unit of a picture is the first VCL NAL unit after the PH NAL unit in the picture.

[0223] The order of non-VCL NAL units (except AUD and EOB NAL units) within a PU shall comply with the following constraints:

[0224] – The PH NAL unit in a PU shall be before the first VCL NAL in the PU.

[0225] – When any DPS NAL unit, VPS NAL unit, SPS NAL unit, PPS NAL unit, pre-APSNAL unit, pre-SEI NAL unit, NAL unit with nal_unit_type equal to RSV_NVCL_26, or NAL unit with nal_unit_type in the range UNSPEC_28 to UNSPEC_29 is present in a PU, they shall not follow the last VCLNAL unit in the PU.

[0226] – When any DPS NAL unit, VPS NAL unit, SPS NAL unit, or PPS NAL unit is present in a PU, they shall be before the PH NAL unit in the PU.

[0227] – NAL unit types equal to SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or RSV_NVCL_27, or NAL units in the range of UNSPEC_30 to UNSPEC_31 shall not be before the first VCL NAL unit of a PU.

[0228] – When an EOS NAL unit exists in a PU, it shall be the last NAL unit among all NAL units in the PU except for the EOB NAL unit (if it exists).

[0229] 7.4.2.4.5 Order of VCL NAL units and its relationship with coded pictures

[0230]

[0231] –

[0232] –

[0233]

[0234] 6.1.4 Sequence parameter set RBSP semantics (VVC clause 7.4.3.3)

[0235] The SPS RBSP shall be available for the decoding process before being referenced, including in at least one AU with TemporalId equal to 0 or provided externally. ...

[0236] The following constraints are applied to require bitstream consistency:

[0237] – For any two sub-pictures subpicA and subpicB, when the sub-picture index of subpicA is less than the sub-picture index of subpicB, any coded slice NAL unit of subPicA shall be before any coded slice NAL unit of subPicB in decoding order.

[0238] – The shape of a sub-picture shall be such that when decoded, its entire left boundary and entire upper boundary are composed of the picture boundary or the boundary of a previously decoded sub-picture. ...

[0239] 6.1.5 General slice header semantics (VVC clause 7.4.8.2)

[0240] When present, the value of the slice_pic_order_cnt_lsb syntax element in the slice header shall be the same for all slice headers of a coded picture. ...

[0241] slice_subpic_id specifies the sub-picture identifier of the sub-picture containing the slice.

[0242] [[If slice_subpic_id is present,]] the value of the derived variable SubPicIdx is such that SubpicIdList[SubPicIdx] is equal to slice_subpic_id. [[Otherwise (slice_subpic_id is not present), the variable SubPicIdx is derived to be equal to 0.]]

[0243]

[0244]

[0245] The length of slice_subpic_id (in bits) is derived as follows:

[0246] – If sps_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to

[0247] sps_subpic_id_len_minus1 + 1.

[0248] – Otherwise, if ph_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to ph_subpic_id_len_minus1 + 1.

[0249] – Otherwise, if pps_subpic_id_signalling_present_flag is equal to 1, the length of slice_subpic_id is equal to pps_subpic_id_len_minus1 + 1.

[0250] – Otherwise, the length of slice_subpic_id is equal to Ceil(Log2(sps_num_subpics_minus1 + 1)).

[0251] slice_address specifies the slice address of the slice. When not present, the value of slice_address is inferred to be equal to 0.

[0252] If rect_slice_flag is equal to 0, the following conditions apply:

[0253] – The strip address is the raster scan slice index.

[0254] – The length of slice_address is Ceil(Log2(NumTilesInPic)) bits.

[0255] – The value of slice_address shall be in the range of 0 to NumTilesInPic - 1, inclusive of 0 and NumTilesInPic - 1. Otherwise (rect_slice_flag equal to 1), the following applies:

[0256] – The strip address is the [[strip index within the]] [[SubPicIdx-th sub-picture]] of the strip.

[0257] – The length of slice_address is Ceil(Log2(NumSlicesInSubpic[SubPicIdx])) bits.

[0258] – The value of slice_address shall be in the range of 0 to NumSlicesInSubpic[SubPicIdx] - 1, inclusive of 0 and

[0259] NumSlicesInSubpic[SubPicIdx] - 1.

[0260] The following constraints are applied for bitstream consistency:

[0261] - If rect_slice_flag is equal to 0 or subpics_present_flag is equal to 0, the value of slice_address shall not be equal to the value of slice_address of any other coded strip NAL unit of the same coded picture.

[0262] - Otherwise, the pair of values of slice_subpic_id and slice_address shall not be equal to the pair of values of slice_subpic_id and slice_address of any other coded strip NAL unit of the same coded picture.

[0263] - When rect_slice_flag is equal to 0, the slices of the picture shall be sorted in ascending order of their slice_address values. - The shape of the picture slices shall be such that when decoded, the entire left boundary and the entire upper boundary of each CTU shall be composed of the picture boundary or the boundary of the previously decoded CTU. ...

[0264] 6.2 Second Embodiment

[0265] In this embodiment, the following changes are made with respect to the first embodiment:

[0266] 1) The following constraints are removed as in the general slice header semantics.

[0267] The requirement for bitstream consistency is that for any i and j in the range from 0 to sps_num_subpics_minus1 (including 0 and sps_num_subpics_minus1), when i is less than j, SubpicIdList[i] shall be less than SubpicIdList[j].

[0268] Optionally, this constraint remains unchanged.

[0269] 2) The text used to specify the order of VCL NAL units and their relationship to the coded / decoded pictures is changed to the following. The added, modified, and most relevant parts are in [[double bold brackets]]. The removed parts are enclosed in [[double bold brackets]].

[0270] - The order of VCL NAL units within a coded / decoded picture is constrained as follows:

[0271] - For any two coded / decoded slice NAL units A and B of a coded / decoded picture, let and

[0272] be their [[slice_subpic_id]] values, and sliceAddrA and sliceAddrB be their slice_address values.

[0273] - The coded / decoded slice NAL unit A shall be before the coded / decoded slice NAL unit B when any of the following conditions is true:

[0274] - is less than

[0275] - is equal to and sliceAddrA is less than sliceAddrB.

[0276] 7. Example implementations of the disclosed technology

[0277] Figure 6 is a block diagram of a video processing apparatus 600. The apparatus 600 can be used to implement one or more methods described herein. The apparatus 600 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 600 can include one or more processors 602, one or more memories 604, and video processing circuitry 606. The processor 602 can be configured to implement one or more methods described in this document. The memory 604 can be used to store data and code for implementing the methods and techniques described herein. The video processing circuitry 606 can be used to implement some of the techniques described in this document in hardware circuitry.

[0278] Figure 7 shows a block diagram of an example video coding and decoding system 700 that can utilize the techniques of the present invention.

[0279] As Figure 7 shown, the video coding and decoding system 700 can include a source device 710 and a destination device 720. The source device 710 generates encoded video data, which can be referred to as a video encoding device. The destination device 720 can decode the encoded video data generated by the source device 710, which can be referred to as a video decoding device.

[0280] The source device 710 can include a video source 712, a video encoder 714, and an input / output (I / O) interface 716.

[0281] The video source 712 can include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data can include one or more pictures. The video encoder 714 encodes the video data from the video source 712 to generate a bitstream. The bitstream can include a sequence of bits forming an encoded representation of the video data. The bitstream can include encoded pictures and associated data. An encoded picture is an encoded representation of a picture. The associated data can include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 716 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be directly transmitted to the destination device 720 via the I / O interface 716 over a network 730a. The encoded video data can also be stored on a storage medium / server 730b for access by the destination device 720.

[0282] The destination device 720 can include an I / O interface 726, a video decoder 724, and a display device 722.

[0283] The I / O interface 726 may include a receiver and / or a modem. The I / O interface 726 may obtain encoded video data from a source device 710 or a storage medium / server 730b. The video decoder 724 may decode the encoded video data. The display device 722 may display the decoded video data to a user. The display device 722 may be integrated with the destination device 720 or may be external to the destination device 720, and the destination device 720 is configured to interface with an external display device.

[0284] The video encoder 714 and the video decoder 724 may operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVM) standard, and other current and / or future standards.

[0285] Figure 8 is a block diagram showing an example of a video encoder 800, and the video encoder 800 may be Figure 7 the video encoder 714 in the system 700 shown.

[0286] The video encoder 800 may be configured to perform any or all of the techniques of the present disclosure. In Figure 8 an example, the video encoder 800 includes a plurality of functional components. The techniques described in the present invention may be shared among various components of the video encoder 800. In some examples, a processor may be configured to perform any or all of the techniques described in the present disclosure.

[0287] The functional components of the video encoder 800 may include a segmentation unit 801, a prediction unit 802 including a mode selection unit 803, a motion estimation unit 804, a motion compensation unit 805, and an intra prediction unit 806, a residual generation unit 807, a transform unit 808, a quantization unit 809, an inverse quantization unit 810, an inverse transform unit 811, a reconstruction unit 812, a buffer 813, and an entropy coding unit 814.

[0288] In other examples, the video encoder 800 may include more, fewer, or different functional components. In one example, the prediction unit 802 may include an Intra Block Copy (IBC) unit. The IBC unit may perform prediction in the IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0289] In addition, some components (e.g., the motion estimation unit 804 and the motion compensation unit 805) may be highly integrated but are shown separately in Figure 8 an example for explanatory purposes.

[0290] The segmentation unit 801 may divide a picture into one or more video blocks. The video encoder 800 and the video decoder 900 may support various video block sizes.

[0291] The mode selection unit 803 may select, for example, one of a plurality of coding / decoding modes (intra or inter) based on an error result, and provide the resulting intra or inter coded block to the residual generation unit 807 to generate residual block data, and to the reconstruction unit 812 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 803 may select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on both inter prediction signaling and intra prediction signaling. In the case of inter prediction, the mode selection unit 803 may also select the resolution of the motion vector for the block (e.g., sub-pixel or integer-pixel precision).

[0292] To perform inter prediction on a current video block, the motion estimation unit 804 may generate motion information for the current video block by comparing one or more reference frames from the buffer 813 with the current video block. The motion compensation unit 805 may determine a predicted video block for the current video block based on motion information from pictures other than the picture associated with the current video block from the buffer 813 and decoded samples.

[0293] The motion estimation unit 804 and the motion compensation unit 805 may perform different operations on the current video block, e.g., depending on whether the current video block is in an I-slice, a P-slice, or a B-slice.

[0294] In some examples, the motion estimation unit 804 may perform uni-directional prediction on the current video block, and the motion estimation unit 804 may search for a reference video block for the current video block in the reference pictures of list 0 or list 7. The motion estimation unit 804 may then generate a reference index indicating the reference picture in list 0 or list 7, which includes the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 804 may output the reference index, a prediction direction indicator, and the motion vector as the motion information for the current video block. The motion compensation unit 805 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0295] In other examples, the motion estimation unit 804 may perform bidirectional prediction on a current video block. The motion estimation unit 804 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 7. The motion estimation unit 804 may then generate a reference index and a motion vector. The reference index indicates the reference pictures in list 0 and list 7 that contain the reference video block, and the motion vector indicates the spatial displacement between the reference video block and the current video block. The motion estimation unit 804 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 805 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0296] In some examples, the motion estimation unit 804 may output complete motion information for the decoding process of the decoder.

[0297] In some examples, the motion estimation unit 804 may not output the complete set of motion information of the current video. Instead, the motion estimation unit 804 may signal the motion information of the current video block by referring to the motion information of another video block. For example, the motion estimation unit 804 may determine that the motion information of the current video block is similar enough to the motion information of an adjacent video block.

[0298] In one example, the motion estimation unit 804 may indicate a value in the syntax structure associated with the current video block, and this value indicates to the video decoder 900 that the current video block has the same motion information as another video block.

[0299] In another example, the motion estimation unit 804 may identify another video block and a motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 900 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0300] As described above, the video encoder 800 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 800 include advanced motion vector predication (AMVP) and merge mode signaling.

[0301] The intra prediction unit 806 can perform intra prediction on the current video block. When the intra prediction unit 806 performs intra prediction on the current video block, the intra prediction unit 806 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0302] The residual generation unit 807 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0303] In other instances, the current video block may not have residual data for the current video block, such as in a skip mode, and the residual generation unit 807 may not perform the subtraction operation.

[0304] The transform processing unit 808 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0305] After the transform processing unit 808 generates the transform coefficient video block associated with the current video block, the quantization unit 809 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0306] The inverse quantization unit 810 and the inverse transform unit 811 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 812 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 802 to generate a reconstructed video block associated with the current block for storage in the buffer 813.

[0307] After the reconstruction unit 812 reconstructs the video block, a loop filtering operation can be performed to reduce the block effect in the video block.

[0308] The entropy coding unit 814 can receive data from other functional components of the video encoder 800. When the entropy coding unit 814 receives data, the entropy coding unit 814 can perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0309] Figure 9 is a block diagram illustrating an example of a video decoder 900, and the video decoder 900 can be Figure 7 the video decoder 714 in the system 700 shown.

[0310] Video decoder 900 may be configured to perform any or all of the techniques of this disclosure. In Figure 9 an example, video decoder 900 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video decoder 900. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0311] In Figure 9 an example, video decoder 900 includes an entropy decoding unit 901, a motion compensation unit 902, an intra prediction unit 903, an inverse quantization unit 904, an inverse transform unit 905, a reconstruction unit 906, and a buffer 907. In some examples, video decoder 900 may perform a decoding process that is generally inverse to the encoding process described for video encoder 800 ( Figure 8 ).

[0312] Entropy decoding unit 901 may retrieve an encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded blocks of video data). Entropy decoding unit 901 may decode the entropy-coded video data, and from the entropy-decoded video data, motion compensation unit 902 may determine motion information, which includes motion vectors, motion vector precision, reference picture list indices, and other motion information. Motion compensation unit 902 may determine this information (e.g.) by performing AMVP and merge mode.

[0313] Motion compensation unit 902 may generate a motion-compensated block and may perform interpolation based on an interpolation filter. An identifier for the interpolation filter to be used with sub-pixel precision may be included in the syntax elements.

[0314] Motion compensation unit 902 may use the interpolation filter used by video encoder 800 during the encoding of a video block to calculate the interpolation values of sub-integer pixels of a reference block. Motion compensation unit 902 may determine the interpolation filter used by video encoder 800 according to the received syntax information and use the interpolation filter to generate a prediction block.

[0315] Motion compensation unit 902 may use some syntax information to determine the size of the blocks of a frame and / or slice of an encoded video sequence, the partitioning information that describes how each macroblock of a picture of the encoded video sequence is partitioned, the mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the encoded video sequence.

[0316] The intra prediction unit 903 can form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 903 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 901. The inverse transform unit 903 applies an inverse transform.

[0317] The reconstruction unit 906 can add the residual block to the corresponding prediction block generated by the motion compensation unit 802 or the intra prediction unit 903 to form a decoded block. If needed, a deblocking filter can also be applied to filter the decoded block to remove blocking artifact. The decoded video block is then stored in the buffer 907, which provides reference blocks for subsequent motion compensation / intra prediction and also generates the decoded video for presentation on a display device.

[0318] Figure 10 FIG. 7 is another example of a block diagram of a video processing system 1000 that can implement the disclosed techniques. Various implementations can include some or all of the components of system 1000. System 1000 can include an input 1002 to receive video content. The video content can be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values) or in a compressed or encoded format. The input 1002 can represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0319] System 1000 can include a codec component 1004 that can implement various codec or encoding methods described in this document. The codec component 1004 can reduce the average bitrate of the video from the input 1002 to the output of the codec component 1004 to produce a coded representation of the video. Thus, codec techniques are sometimes referred to as video compression or video transcoding techniques. As represented by component 1006, the output of the codec component 1004 can be stored or transmitted via the connected communication. The stored or transmitted bitstream representation (or coded representation) of the video received at the input 1002 can be used by component 1008 to generate pixel values or viewable video, which are sent to the display interface 1010. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Additionally, although certain video processing operations are referred to as "codec" operations or tools, it should be understood that encoding tools or encoding operations are used at the encoder, and the corresponding decoding tools or decoding operations that reverse the encoding result will be performed by the decoder.

[0320] Examples of peripheral bus interfaces or display interfaces can include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Displayport, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, and so on. The techniques described in this document can be implemented in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of performing digital data processing and / or video display.

[0321] From the foregoing, it will be understood that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not limited except as by the appended claims.

[0322] The implementation of the subject matter and the functional operations described in this patent document can be implemented as various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their equivalent structures, or as a combination of one or more of them. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a substance composition implementing a machine-readable propagated signal, or a combination of one or more of them. The term "data processing unit" or "data processing apparatus" encompasses all devices, apparatus, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer programs being discussed, e.g., code constituting processor firmware, protocol stack, database management system, operating system, or a combination of one or more of them.

[0323] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a stand-alone program or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0324] The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processing and logical flows can also be performed by, or implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0325] Processors suitable for the execution of a computer program include both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as, magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, such as, EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0326] Some embodiments may be described using the following clause-based format. The first set of clauses illustrates example embodiments of the techniques discussed in the previous sections.

[0327] A1. A method for video processing, comprising: determining that a slice mode of a slice including a current video segment is a rectangular slice mode for a conversion between the current video segment of a video and a bitstream representation of the video including a plurality of video coding layer (VCL) network abstraction layer (NAL) units; and performing the conversion based on the determination, wherein the bitstream representation further includes one or more syntax elements that signal a decoding order of the plurality of VCL NAL units.

[0328] A2-1. The method according to clause A1, wherein the decoding order includes sub-picture ID values of sub-pictures including the plurality of VCL NAL units arranged in ascending order.

[0329] A2-2. The method according to clause A1, wherein the decoding order includes sub-picture level slice index values of the plurality of VCL NAL units arranged in ascending order.

[0330] A3-1. The method according to clause A1, wherein the decoding order includes sub-picture index values of sub-pictures including a plurality of VCL NAL units arranged in ascending order.

[0331] A3-2. The method according to clause A1, wherein one of the sub-picture level strip index values is the value of the slice_address syntax element in the strip header.

[0332] A4. The method according to any one of clauses A1 to A3, wherein when it is determined that the bitstream representation does not include the slice_subpic_id parameter, the value of the slice_subpic_id parameter is inferred to be zero.

[0333] A5. The method according to clauses A2 and A3, wherein the sub-picture ID value increases monotonically with the sub-picture index value.

[0334] A6. The method according to any one of clauses A1 to A5, wherein the conversion generates a current video segment from the bitstream representation.

[0335] A7. The method according to any one of clauses A1 to A5, wherein the conversion generates a bitstream representation from the current video segment.

[0336] A8. The method according to any one of clauses A1 to A7, wherein the current video segment is a current strip slice, current block, current slice or current sub-picture.

[0337] A9. A video processing apparatus including a processor configured to execute the method according to any one or more of clauses A1 to A8.

[0338] A10. A computer-readable recording medium having recorded thereon a program including code for a processor to execute the method according to any one of clauses A1 to A8.

[0339] A11. A computer program product stored on a non-transitory computer-readable medium, the computer program product including program code for executing the method according to any one of clauses A1 to A8.

[0340] The second set of clauses describes certain features and aspects of the technology disclosed in the previous section.

[0341] 1. A method for video processing (e.g., Figure 11The method described in (1100) includes: According to the rule, performing (1110) the conversion between a video including one or more pictures and a bitstream representation of the video, where the one or more pictures include one or more sub-pictures, the one or more sub-pictures include one or more stripes, and where the bitstream representation includes a number of codec units, and where the rule specifies that the decoding order of the codec units within a sub-picture is in ascending order of the sub-picture level stripe index value of the codec units.

[0342] 2. The method according to claim 1, wherein the codec unit corresponds to a video coding layer (VCL) network abstraction layer (NAL) unit.

[0343] 3. The method according to claim 1, wherein the rule is applied in the case where the stripe mode is a rectangular stripe mode.

[0344] 4. The method according to claim 1, wherein the sub-picture level stripe index value is the value of the syntax element slice_address in the stripe header that specifies the stripe address of the stripe.

[0345] 5. The method according to any one of claims 1 to 4, wherein when it is determined that the bitstream representation does not include the parameter slice_subpic_id, the value of the parameter slice_subpic_id is inferred.

[0346] 6. The method according to claim 5, wherein the parameter slice_subpic_id specifies the sub-picture identification (ID) of the sub-picture containing the stripe.

[0347] 7. The method according to claim 5, wherein based on the determination, the value of the parameter slice_subpic_id is inferred to be equal to 0.

[0348] 8. The method according to any one of claims 1 - 7, wherein the sub-picture ID value of a sub-picture including a number of codec units increases monotonically with the sub-picture index value of the sub-picture including a number of codec units.

[0349] 9. The method according to any one of claims 5 to 7, wherein the decoding order of the decoding units in a picture including a first strip of network abstraction layer (NAL) units and a second strip of NAL units is specified such that, in the following cases, the first strip of NAL units is before the second strip of NAL units: i) the sub-picture level strip index value of the first strip of NAL units is less than the sub-picture level strip index value of the second strip of NAL units, or ii) the sub-picture level strip index value of the first strip of NAL units is equal to the sub-picture level strip index value of the second strip of NAL units, and the value of the syntax element slice_address of the first strip of NAL units is less than the value of the syntax element slice_address of the second strip of NAL units.

[0350] 10. A video processing method, comprising:

[0351] performing a conversion between a video including one or more pictures and a bitstream representation of the video according to rules, the one or more pictures including one or more sub-pictures, the one or more sub-pictures including one or more strips, and

[0352] wherein the bitstream representation includes a number of decoding units,

[0353] wherein the rules specify that the decoding order of the decoding units is in ascending order of sub-picture related values of sub-pictures from one or more sub-pictures including the decoding units.

[0354] 11. The method according to claim 10, wherein the decoding units correspond to video coding layer (VCL) network abstraction layer (NAL) units.

[0355] 12. The method according to claim 10, wherein the rules are applied in the case where the strip mode is a rectangular strip mode.

[0356] 13. The method according to claim 10, wherein the sub-picture related values correspond to the identification (ID) values of the sub-pictures including the decoding units.

[0357] 14. The method according to claim 10, wherein the sub-picture related values correspond to the sub-picture index values of the sub-pictures including the decoding units.

[0358] 15. The method according to any one of claims 10 to 14, wherein, when it is determined that the bitstream representation does not include the parameter slice_subpic_id, the value of the parameter slice_subpic_id is inferred.

[0359] 16. The method according to claim 15, wherein based on the determination, the value of the parameter slice_subpic_id is inferred to be equal to 0.

[0360] 17. The method according to any one of claims 10 to 16, wherein the sub-picture ID value of a sub-picture including a plurality of codec units monotonically increases with the sub-picture index value of the sub-picture including a plurality of codec units.

[0361] 18. The method according to any one of claims 10 to 16, wherein the conversion includes encoding the video into a bitstream representation.

[0362] 19. The method according to any one of claims 10 to 16, wherein the conversion includes decoding the video from the bitstream representation.

[0363] 20. A video processing apparatus including a processor, the processor being configured to execute the method according to any one or more of claims 1-19.

[0364] 21. A computer-readable medium storing program code, which when executed, causes a processor to implement the method according to any one or more of claims 1 to 19.

[0365] 22. A computer-readable medium storing a codec representation or a bitstream representation generated according to any of the above methods.

[0366] 23. A video processing apparatus for storing a bitstream representation, wherein the video processing apparatus is configured to implement the method according to any one or more of claims 1 to 19.

[0367] Although this patent document contains many details, these details should not be construed as limitations on any invention or the scope of what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although the features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may relate to a sub-combination or a variation of a sub-combination.

[0368] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired effect. Additionally, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0369] Only some embodiments and examples are described, and other embodiments, enhancements, and variations can be made based on what is described and shown in this patent document.

Claims

1. A video processing method, comprising: performing a conversion between a video including one or more pictures and a bitstream of the video according to a rule, wherein the one or more pictures include one or more sub-pictures, and the one or more sub-pictures include one or more strips, wherein the bitstream includes a plurality of codec units, and wherein the rule specifies that the decoding order of the codec units within a sub-picture is in ascending order of the value of the syntax element slice_address in the strip header, and the syntax element slice_address specifies the strip address of the strip corresponding to the codec unit, wherein when the strip mode is a rectangular strip mode, the value of the syntax element slice_address in the strip header is equal to the sub-picture level strip index, wherein the sub-picture level strip index specifies the strip index of the strip list in the sub-picture, and the order is the same as the order signaled in the picture parameter set.

2. The method according to claim 1, wherein, the codec unit corresponds to a video coding layer (VCL) network abstraction layer (NAL) unit.

3. The method according to claim 1, wherein, when it is determined that the bitstream does not include the slice_subpic_id parameter, inferring the value of the slice_subpic_id parameter, and wherein the slice_subpic_id parameter specifies the sub-picture identifier (ID) of the sub-picture containing the strip.

4. The method according to claim 3, wherein, in the determined case, the value of the slice_subpic_id parameter is inferred to be equal to 0.

5. The method according to claim 1, wherein, the sub-picture ID value of the sub-picture including the plurality of codec units increases monotonically with the sub-picture index value of the sub-picture including the plurality of codec units.

6. The method according to claim 1, wherein, specifying the decoding order of the codec units within a picture including a first strip network abstraction layer (NAL) unit and a second strip NAL unit, such that the first strip NAL unit is before the second strip NAL unit in the following cases: i) the sub-picture index value of the first strip NAL unit is less than the sub-picture index value of the second strip NAL unit, or ii) the sub-picture index value of the first strip NAL unit is equal to the sub-picture index value of the second strip NAL unit and the value of the syntax element slice_address of the first strip NAL unit is less than the value of the syntax element slice_address of the second strip NAL unit.

7. The method according to claim 1, wherein, the conversion includes encoding the video into the bitstream.

8. The method according to claim 1, wherein, the conversion includes decoding the video from the bitstream.

9. A video data processing device, comprising a processor and a non-transitory memory having instructions thereon, wherein, the instructions, when executed by the processor, cause the processor: Perform a conversion between a video including one or more pictures and a bitstream of the video according to a rule, where the one or more pictures include one or more sub-pictures, and the one or more sub-pictures include one or more stripes. where the bitstream includes a plurality of codec units, and where the rule specifies that the decoding order of the codec units within a sub-picture is in ascending order of the value of a syntax element slice_address in a stripe header, and the syntax element slice_address specifies the stripe address of the stripe corresponding to the codec unit. where when the stripe mode is a rectangular stripe mode, the value of the syntax element slice_address in the stripe header is equal to the sub-picture level stripe index. where the sub-picture level stripe index specifies the stripe index of a stripe list in the sub-picture, in the same order as signaled in the picture parameter set.

10. The apparatus according to claim 9, where the codec unit corresponds to a video coding layer (VCL) network abstraction layer (NAL) unit.

11. The apparatus according to claim 9, where specify the decoding order of the codec units within a picture including a first stripe network abstraction layer (NAL) unit and a second stripe NAL unit such that the first stripe NAL unit is before the second stripe NAL unit in the following cases: i) the sub-picture index value of the first stripe NAL unit is less than the sub-picture index value of the second stripe NAL unit, or ii) the sub-picture index value of the first stripe NAL unit is equal to the sub-picture index value of the second stripe NAL unit and the value of the syntax element slice_address of the first stripe NAL unit is less than the value of the syntax element slice_address of the second stripe NAL unit.

12. A non-transitory computer-readable storage medium having instructions stored thereon that cause a processor to: Perform a conversion between a video including one or more pictures and a bitstream of the video according to a rule. where the one or more pictures include one or more sub-pictures, and the one or more sub-pictures include one or more stripes. where the bitstream includes a plurality of codec units, and where the rule specifies that the decoding order of the codec units within a sub-picture is in ascending order of the value of a syntax element slice_address in a stripe header, and the syntax element slice_address specifies the stripe address of the stripe corresponding to the codec unit. where when the stripe mode is a rectangular stripe mode, the value of the syntax element slice_address in the stripe header is equal to the sub-picture level stripe index. where the sub-picture level stripe index specifies the stripe index of a stripe list in the sub-picture, in the same order as signaled in the picture parameter set.

13. The non-transitory computer-readable storage medium according to claim 12, where Specify the decoding order of coding units within a picture that includes a first strip network abstraction layer (NAL) unit and a second strip NAL unit such that the first strip NAL unit is before the second strip NAL unit in the following cases: i) the sub-picture index value of the first strip NAL unit is less than the sub-picture index value of the second strip NAL unit, or ii) the sub-picture index value of the first strip NAL unit is equal to the sub-picture index value of the second strip NAL unit and the value of the syntax element slice_address of the first strip NAL unit is less than the value of the syntax element slice_address of the second strip NAL unit.

14. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method executed by a video processing device, the method comprising: generating, according to rules, a bitstream of a video including one or more pictures, wherein the one or more pictures include one or more sub-pictures, and the one or more sub-pictures include one or more strips, wherein the bitstream includes a plurality of coding units, and wherein the rules specify that the decoding order of coding units within a sub-picture is in ascending order of the value of the syntax element slice_address in the strip header, and the syntax element slice_address specifies the strip address of the strip corresponding to the coding unit, wherein when the strip mode is the rectangular strip mode, the value of the syntax element slice_address in the strip header is equal to the sub-picture level strip index, wherein the sub-picture level strip index specifies the strip index of the strip list in the sub-picture in the same order as signaled in the picture parameter set.

15. The non-transitory computer-readable recording medium according to claim 14, wherein specify the decoding order of coding units within a picture that includes a first strip network abstraction layer (NAL) unit and a second strip NAL unit such that the first strip NAL unit is before the second strip NAL unit in the following cases: i) the sub-picture index value of the first strip NAL unit is less than the sub-picture index value of the second strip NAL unit, or ii) the sub-picture index value of the first strip NAL unit is equal to the sub-picture index value of the second strip NAL unit and the value of the syntax element slice_address of the first strip NAL unit is less than the value of the syntax element slice_address of the second strip NAL unit.

16. A method for storing a bitstream of a video, comprising: generating, according to rules, a bitstream of a video including one or more pictures, wherein the one or more pictures include one or more sub-pictures, and the one or more sub-pictures include one or more strips; storing the bitstream in a non-transitory computer-readable recording medium, wherein the bitstream includes a plurality of coding units, and Among them, the rule specifies that the decoding order of the codec units within the sub-picture is in ascending order of the value of the syntax element slice_address in the slice header, and the syntax element slice_address specifies the slice address of the slice corresponding to the codec unit. Among them, when the slice mode is the rectangular slice mode, the value of the syntax element slice_address in the slice header is equal to the sub-picture level slice index. Among them, the sub-picture level slice index specifies the slice index of the slice list in the sub-picture, and the order is the same as the order signaled in the picture parameter set.

17. A video processing apparatus including a processor, the processor being configured to execute the method according to any one of claims 3 to 5.

18. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method according to any one of claims 2 to 5.

19. A computer-readable medium storing a bitstream generated by a method executed by a video processing apparatus. Among them, the video processing apparatus is configured to implement the method according to any one of claims 2 to 5.

20. A video processing apparatus for storing a bitstream. Among them, the bitstream is generated by the method executed by the video processing apparatus, and the video processing apparatus is configured to implement the method according to any one of claims 1 to 6.

Images