Use of sub-picture information in video coding and decoding

Abstract

Methods and apparatuses for processing video are described. The processing may include video encoding, decoding, or transcoding. An example video processing method includes performing a conversion between a video including one or more pictures including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule of a syntax of network abstraction layer (NAL) units in a specified bitstream, and wherein the format rule specifies that NAL units of a video coding layer (VCL) NAL unit type include content associated with a specific type of picture or a specific type of sub-picture.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority and benefit of U.S. Provisional Patent Application No. 62 / 992,724, filed on Mar. 20, 2020, based on International Patent Application No. PCT / US2021 / 022970, filed on Mar. 18, 2021. All of the above - mentioned patent applications are hereby incorporated by reference in their entirety. Technical Field

[0003] This patent document relates to image and video encoding, decoding, and transcoding. Background Art

[0004] In the Internet and other digital communication networks, digital video occupies the largest bandwidth. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video use is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process the encoded and decoded representations of video using various syntax rules.

[0006] In one example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub - pictures and a bitstream of the video, wherein the bitstream conforms to format rules that define the syntax of network abstraction layer (NAL) units in the specified bitstream, and wherein the format rules specify that NAL units of video coding layer (VCL) NAL unit types include content associated with a particular type of picture or a particular type of sub - picture.

[0007] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture including sub - pictures and a bitstream of the video, wherein the bitstream conforms to format rules, and wherein the format rules specify that a sub - picture is a random - access - type sub - picture in response to the sub - picture being a pre - sub - picture of an intra - random - access - point sub - picture.

[0008] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture including sub - pictures and a bitstream of the video, wherein the bitstream conforms to format rules, and wherein the format rules specify that one or more random - access - skip pre - sub - pictures are not present in the bitstream in response to the one or more random - access - skip pre - sub - pictures being associated with an instantly decoded refresh sub - picture.

[0009] In another exemplary aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to one or more randomly accessible decodable pre-sub-pictures being associated with an instant decoding refresh sub-picture having a type of network abstraction layer (NAL) unit indicating that the instant decoding refresh sub-picture is not associated with a pre-picture, one or more randomly accessible decodable pre-sub-pictures do not exist in the bitstream.

[0010] In another exemplary aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value that indicates whether each of the two adjacent sub-pictures in a coded layer video sequence is considered a picture.

[0011] In another exemplary aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the format rule specifies that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and where the format rule specifies that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0012] In another exemplary aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to a syntax element indicating that each picture of a reference picture parameter set (PPS) of the video has multiple video coding layer (VCL) network abstraction layer (NAL) units that do not have the same type of VCL NAL unit, a picture is allowed to include more than two different types of VCL NAL units.

[0013] In another exemplary aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures that include one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that a post-sub-picture sequentially associated with an intra-random access point sub-picture or a progressive decoding refresh sub-picture is after the intra-random access point sub-picture or the progressive decoding refresh sub-picture.

[0014] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a second-order sub-picture being before an intra-random access point sub-picture, (2) the sub-picture and the intra-random access point sub-picture having the same first value for a layer to which a network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of a sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0015] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is before one or more random access decodable pre-sub-pictures associated with the clean random access sub-picture in an order.

[0016] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is after one or more intra-random access point sub-pictures before the clean random access sub-picture in a second order after the one or more intra-random access point sub-pictures in a first order.

[0017] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a syntax element indicating that a coded layer video sequence conveys a picture representing a frame, and (2) a current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in a decoding order.

[0018] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video coding layer (VCL) network abstraction layer (NAL) units in the picture include RADL_NUT or RASL_NUT.

[0019] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to: (1) at least one sub-picture being before a progressive decoding refresh sub-picture in a second order, (2) the at least one sub-picture and the progressive decoding refresh sub-picture having the same first value for a layer to which network abstraction layer (NAL) units of the at least one sub-picture and the progressive decoding refresh sub-picture belong, and (3) the at least one sub-picture and the progressive decoding refresh picture having the same second value of sub-picture index, the at least one sub-picture is before the progressive decoding refresh sub-picture and one or more sub-pictures associated with the progressive decoding refresh sub-picture in a first order.

[0020] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that in response to: (a) a first picture having the same temporal identifier and the same layer identifier of network abstraction layer (NAL) units as the current sub-picture, and (b) the current sub-picture being after a stepwise temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the stepwise temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, the format rule does not allow an active entry in a reference picture list of the current strip to include the first picture that is before a second picture in decoding order, the second picture including the stepwise temporal sub-layer access sub-picture.

[0021] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that in response to the current sub-picture not being a specific type of sub-picture, the format rule does not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0022] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the sub-picture including a current strip, wherein the bitstream conforms to format rules, and in response to the current sub-picture not being a sub-picture of a specific type, the format rules do not allow an entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0023] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current strip, wherein the bitstream conforms to format rules, and in response to (a) a first picture including a previous intra-random access point sub-picture, the previous intra-random access point sub-picture being before the current sub-picture in a second order, (b) the previous intra-random access point sub-picture having the same layer identifier of a network abstraction layer (NAL) unit and the same sub-picture index as the current sub-picture, and (c) the current sub-picture being a pure random access sub-picture, the format rules do not allow an entry in a reference picture list of the current strip to include the first picture that is before the current picture in a first order or a second order.

[0024] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current strip, wherein the bitstream conforms to format rules, and in response to (a) the current sub-picture being associated with an intra-random access point sub-picture, and (b) the current sub-picture being after the intra-random access point sub-picture in a first order, the format rules do not allow an active entry in a reference picture list of the current strip to include the first picture that is before the current picture in a first order or a second order.

[0025] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current strip, wherein the bitstream conforms to format rules, and in response to: (a) the current sub-picture being after an intra-random access point sub-picture in a first order, and (b) the current sub-picture being after one or more preceding sub-pictures associated with an IRAP sub-picture in a first order and a second order, the format rules do not allow an entry in a reference picture list of the current strip to include the first picture that is before the current picture including the intra-random access point sub-picture associated with the current sub-picture in a first order or a second order.

[0026] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture and a bitstream of the video, where the current sub-picture includes a current strip, and the bitstream conforms to format rules that specify that in response to the current sub-picture being a random access decodable prefix sub-picture, the reference picture list of the current strip does not include active entries of any one or more of the following: a first picture including a random access skipped prefix sub-picture and a second picture that is before a third picture including an intra random access point sub-picture in decoding order.

[0027] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video. The codec representation conforms to format rules that specify that one or more pictures including one or more sub-pictures are included in the codec representation according to network abstraction layer (NAL) units, where a type of NAL unit indicated in the codec representation includes a coded strip of a specific type of picture or a coded strip of a specific type of sub-picture.

[0028] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that specify that two adjacent sub-pictures having different network abstraction layer unit types will have the same indication of the sub-picture regarded as a picture flag.

[0029] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that define an order of a first type of sub-picture and a second type of sub-picture, where the first sub-picture is a postfix sub-picture or a prefix sub-picture or a random access skipped prefix (RASL) sub-picture type, and the second sub-picture is a RASL type or a random access decodable prefix (RADL) type or an instant decoding refresh (IDR) type or a gradual decoding refresh (GDR) type sub-picture.

[0030] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that define conditions for allowing or not allowing a first type of sub-picture to coexist with a second type of sub-picture.

[0031] In yet another example aspect, a video encoder device is disclosed. The video encoder includes a processor configured to implement the above-described method.

[0032] In yet another example aspect, a video decoder device is disclosed. The video decoder includes a processor configured to implement the above-described method.

[0033] In yet another example aspect, a computer-readable medium having code stored thereon is disclosed. The code embodies one of the methods described herein in the form of processor-executable code.

[0034] These and other features will be described throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An example of raster scan strip partitioning of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0036] Figure 2 An example of rectangular strip partitioning of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0037] Figure 3 An example of a picture divided into slices and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0038] Figure 4 A picture divided into 15 slices, 24 strips, and 24 sub-pictures is shown.

[0039] Figure 5 is a block diagram of an example video processing system.

[0040] Figure 6 is a block diagram of a video processing device.

[0041] Figure 7 is a flowchart of an example method of video processing.

[0042] Figure 8 is a block diagram showing a video codec system according to some embodiments of the present disclosure.

[0043] Figure 9 is a block diagram showing an encoder according to some embodiments of the present disclosure.

[0044] Figure 10 is a block diagram showing a decoder according to some embodiments of the present disclosure.

[0045] Figures 11 to 31 is a flowchart of an example method of video processing. DETAILED DESCRIPTION

[0046] The use of section headings in this document is for ease of understanding and does not limit the applicability of the technologies and embodiments disclosed in each section to that section. Additionally, the use of H.266 terms in some descriptions is merely for ease of understanding and does not limit the scope of the disclosed technologies. Therefore, the technologies described herein are also applicable to other video codec protocols and designs. In this document, editorial changes to the text are shown by left and right square brackets (e.g., []) with respect to the current draft of the VVC specification. The deleted text within the square brackets represents the stricken text, and represents the added text.

[0047] 1. Initial Discussion

[0048] This document is related to video coding and decoding technologies. Specifically, it is about the definition of sub-picture types and the relationships in terms of decoding order, output order, and prediction relationships between different types of sub-pictures in single-layer and multi-layer contexts. The key is to clearly specify the meaning of the mixed sub-picture types within a picture through a set of constraints on decoding order, output order, and prediction relationships. These ideas can be applied individually or in various combinations to any video coding standard or non-standard video codec that supports multi-layer video coding and decoding, such as the Versatile Video Coding (VVC) being developed.

[0049] 2. Abbreviations

[0050] APS (Adaptation Parameter Set) Adaptive Parameter Set

[0051] AU (Access Unit) Access Unit

[0052] AUD (Access Unit Delimiter) Access Unit Delimiter

[0053] AVC (Advanced Video Coding) Advanced Video Coding

[0054] CLVS (Coded Layer Video Sequence) Coded Layer Video Sequence

[0055] CPB (Coded Picture Buffer) Coded Picture Buffer

[0056] CRA (Clean Random Access) Clean Random Access

[0057] CTU (Coding Tree Unit) Coding Tree Unit

[0058] CVS (Coded Video Sequence) Coded Video Sequence

[0059] DCI (Decoding Capability Information) Decoding Capability Information

[0060] DPB (Decoded Picture Buffer) Decoded Picture Buffer

[0061] EOB (End Of Bitstream) End Of Bitstream

[0062] EOS (End Of Sequence) End Of Sequence

[0063] GDR (Gradual Decoding Refresh) Gradual Decoding Refresh

[0064] HEVC (High Efficiency Video Coding) High Efficiency Video Coding

[0065] HRD (Hypothetical Reference Decoder) Hypothetical Reference Decoder

[0066] IDR (Instantaneous Decoding Refresh) Instantaneous Decoding Refresh

[0067] JEM (Joint Exploration Model) Joint Exploration Model

[0068] MCTS (Motion-Constrained Tile Sets) Motion-Constrained Tile Sets

[0069] NAL (Network Abstraction Layer) Network Abstraction Layer

[0070] OLS (Output Layer Set) Output Layer Set

[0071] PH (Picture Header) Picture Header

[0072] PPS (Picture Parameter Set) Picture Parameter Set

[0073] PTL (Profile, Tier and Level) Profile, Tier and Level

[0074] PU (Picture Unit) Picture Unit

[0075] RADL (Random Access Decodable Leading (Picture)) Random Access Decodable Leading (Picture)

[0076] RAP (Random Access Point) Random Access Point

[0077] RASL (Random Access Skipped Leading (Picture)) Random Access Skipped Leading (Picture)

[0078] RBSP (Raw Byte Sequence Payload) Raw Byte Sequence Payload

[0079] RPL (Reference Picture List) Reference Picture List

[0080] SEI (Supplemental Enhancement Information) Supplemental Enhancement Information

[0081] SPS (Sequence Parameter Set) Sequence Parameter Set

[0082] STSA (Step - wise Temporal Sublayer Access) Step - wise Temporal Sublayer Access

[0083] SVC (Scalable Video Coding) Scalable Video Coding

[0084] VCL (Video Coding Layer) Video Coding Layer

[0085] VPS (Video Parameter Set) Video Parameter Set

[0086] VTM (VVC Test Model) VVC Test Model

[0087] VUI (Video Usability Information) Video Usability Information

[0088] VVC (Versatile Video Coding) Versatile Video Coding

[0089] 3. Introduction to Video Coding

[0090] Video coding standards have evolved mainly through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly developed H.262 / MPEG-2 video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure, in which temporal prediction plus transform coding is utilized. To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to a reference software called the Joint Exploration Model (JEM). At the same time, JVET meetings are held quarterly, and the goal of the new coding standard is to reduce the bit rate by 50% compared to HEVC. The new video coding standard was officially named Versatile Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. Due to the continuous efforts in VVC standardization, new coding technologies have been adopted into the VVC standard at each JVET meeting. Then, the working draft of VVC and the test model VTM are updated after each meeting. The VVC project is now striving to be technically completed (FDIS) at the meeting in July 2020.

[0091] 3.1. Picture segmentation schemes in HEVC

[0092] HEVC includes four different picture segmentation schemes, namely regular slices, dependent slices, tiles, and Wavefront Parallel Processing (WPP), which can be applied to Maximum Transfer Unit (MTU) size matching, parallel processing, and reduced end-to-end latency.

[0093] Regular slices are similar to those in H.264 / AVC. Each regular slice is encapsulated in its own NAL unit, and picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are disabled. Therefore, a regular slice can be reconstructed independently of other regular slices within the same picture (although there may still be dependencies due to loop filter operations).

[0094] Regular slices are the only tool that can be used for parallelization and are also available in H.264 / AVC in a nearly identical form. Parallelization based on regular slices does not require much inter-processor or inter-core communication (except for the inter-processor or inter-core data sharing for motion compensation when decoding predicted coded pictures, which is usually much heavier due to intra-picture prediction than inter-processor or inter-core data sharing). However, for the same reason, the use of regular slices incurs a large amount of coding and decoding overhead due to the bit cost of the slice header and due to the lack of prediction across slice boundaries. In addition, due to the intra-picture independence of regular slices and each regular slice being encapsulated in its own NAL unit, regular slices (compared to other tools mentioned below) also serve as a key mechanism for bitstream segmentation to match the MTU size requirements. In many cases, the goals of parallelization and MTU size matching pose conflicting requirements for the slice layout in the picture. The recognition of this situation led to the development of the parallelization tools mentioned below.

[0095] Dependent slices have short slice headers and allow the bitstream to be segmented at tree block boundaries without breaking any intra-picture prediction. Basically, dependent slices provide fragmentation of regular slices into multiple NAL units to provide reduced end-to-end latency by allowing a part of a regular slice to be sent out before the encoding of the entire regular slice is completed.

[0096] In WPP, a picture is segmented into single-row coded tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other segments. Through parallel decoding of CTB rows, parallel processing is possible, where the start of decoding of a CTB row is delayed by two CTBs to ensure that data related to CTBs above and to the right of the main CTB can be obtained before the main CTB being decoded. Using this staggered start (which looks like a wavefront when represented graphically), as many processors / cores as there are CTB rows in the picture can be parallelized. Since intra-picture prediction between neighboring tree block rows within a picture is permitted, the inter-processor / inter-core communication required to enable intra-picture prediction can be substantial. Compared to when not applied, WPP segmentation does not result in the generation of additional NAL units, so WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular slices can be used together with WPP with a certain degree of coding and decoding overhead.

[0097] Slices define the horizontal and vertical boundaries that divide a picture into slice columns and slice rows. Slice columns extend from the top to the bottom of the picture. Similarly, slice rows extend from the left to the right of the picture. The number of slices in a picture can be simply derived by multiplying the number of slice columns by the number of slice rows.

[0098] Before decoding the top-left CTB of the next slice in the raster scan order of the picture's slices, the scan order of CTBs is changed to be local within the slice (in the raster scan order of the slice's CTBs). Similar to regular stripes, slices break prediction dependencies and entropy decoding dependencies within the picture. However, they do not need to be included in separate NAL units (the same as WPP in this regard); thus, slices cannot be used for MTU size matching. Each slice can be processed by one processor / core, and in the case where a stripe spans multiple slices, the inter-processor / inter-core communication required for intra-picture prediction between processing units decoding adjacent slices is limited to transmitting the shared stripe header and loop filtering related to the sharing of reconstructed samples and metadata. When a stripe contains more than one slice or WPP segment, the byte offset of the entry point of each slice or WPP segment except the first in the stripe is signaled in the stripe header.

[0099] For simplicity, HEVC has specified restrictions on the application of four different picture segmentation schemes. A given coded video sequence cannot include both slices and wavefronts for most profiles specified in HEVC. For each stripe and slice, one or both of the following conditions must be met: 1) all coded tree blocks in the stripe belong to the same slice; 2) all coded tree blocks in the slice belong to the same stripe. Finally, a wavefront segment exactly contains one CTB row, and when WPP is being used, if a stripe starts within a CTB row, it must end in the same CTB row.

[0100] The latest revision to HEVC is specified in the JCT-VC output file JCTVC-AC1005, J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, Y.-K. Wang (editors), "HEVC Additional Supplemental Enhancement Information (Draft 4)", publicly released on October 24, 2017: http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. By including this amendment, HEVC specifies three MCTS-related SEI messages, namely the Temporal MCTS SEI message, the MCTS Extraction Information Set SEI message, and the MCTS Extraction Information Nested SEI message.

[0101] The time-domain MCTS SEI message indicates the presence of MCTS in the bitstream and signals the MCTS. For each MCTS, the motion vectors are restricted to point to full-sample positions within the MCTS and fractional-sample positions of full-sample positions within the MCTS that are only needed for interpolation, and the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is not allowed. In this way, each MCTS can be independently decoded in the absence of slices not included in the MCTS.

[0102] The MCTS extraction information set SEI message provides supplementary information (specified as part of the semantics of the SEI message) that can be used in MCTS sub-bitstream extraction to generate a compliant bitstream for an MCTS group. This information consists of a number of extraction information sets, each defining a number of MCTS groups and containing the RBSP bytes that will replace the VPS, SPS, and PPS during the MCTS sub-bitstream extraction process. When extracting the sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all of the syntax elements related to the slice address (including first_slice_segment_in_pic_flag and slice_segment_address) usually need to have different values.

[0103] 3.2. Picture segmentation in VVC

[0104] In VVC, a picture is divided into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs that cover a rectangular region of the picture. The CTUs within a slice are scanned in raster scan order within the slice.

[0105] A strip consists of an integer number of complete slices or an integer number of consecutive complete CTU rows within a slice of the picture.

[0106] Two strip modes are supported, namely the raster scan strip mode and the rectangular strip mode. In the raster scan strip mode, a strip contains the complete sequence of slices in the slice raster scan of the picture. In the rectangular strip mode, a strip contains a number of complete slices that together form a rectangular region of the picture, or a number of consecutive complete CTU rows of a single slice that together form a rectangular region of the picture. The slices within a rectangular strip are scanned in slice raster scan order within the rectangular region corresponding to the strip.

[0107] A sub-picture contains one or more strips that together cover a rectangular region of the picture.

[0108] Figure 1 An example of the raster scan strip segmentation of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0109] Figure 2 An example of rectangular strip segmentation of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0110] Figure 3 An example of a picture segmented into slices and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0111] Figure 4 An example of sub-picture segmentation of a picture is shown, where the picture is segmented into 18 slices, 12 slices on the left (each covering a strip with 4x4 CTUs) and 6 slices on the right (each covering 2 vertically stacked strips with 2x2 CTUs), resulting in a total of 24 strips and 24 sub-pictures of different dimensions (each strip is a sub-picture).

[0112] 3.3. Change of Picture Resolution within a Sequence

[0113] In AVC and HEVC, unless a new sequence using a new SPS starts with an IRAP picture, the spatial resolution of a picture cannot be changed. VVC enables changing the picture resolution within a sequence at positions where IRAP pictures are not encoded, and IRAP pictures are always intra-coded and decoded. This feature is sometimes referred to as reference picture resampling (RPR) because when the reference picture has a different resolution from the current picture being decoded, this feature requires resampling of the reference pictures used for inter prediction.

[0114] The scaling ratio is restricted to be greater than or equal to 1 / 2 (2x downsampling from the reference picture to the current picture) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cut-offs are specified to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, the same as in the case of motion compensation interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process with a scaling ratio range from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height and the left, right, up, and down scaling offsets specified for the reference picture and the current picture.

[0115] Other aspects of the VVC design that support features different from HEVC include: i) Picture resolution and the corresponding consistency window are signaled in the PPS rather than in the SPS, while the maximum picture resolution is signaled in the SPS. ii) For a single-layer bitstream, each picture buffer (the storage slot for a decoded picture in the DPB) occupies the buffer size required to store a decoded picture with the maximum picture resolution.

[0116] 3.4. In general and scalable video coding (SVC) in VVC

[0117] Scalable video coding (SVC, sometimes also referred to as scalability in video coding) refers to video coding in which a base layer (BL), sometimes referred to as a reference layer (RL), and one or more scalable enhancement layers (ELs) are used. In SVC, the base layer can carry video data with a base-level quality. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as the BL, and the top layer can be used as the EL. Intermediate layers can act as an EL or an RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of the layer below (e.g., the base layer or any intermediate enhancement layer) and at the same time an RL for one or more enhancement layers above the intermediate layer. Similarly, in the multi-view or 3D extension of the HEVC standard, there can be multiple views, and the information of one view can be used for coding (e.g., encoding or decoding) the information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancies).

[0118] In SVC, parameters used by the encoder or decoder are grouped into parameter sets based on the coding level (e.g., video level, sequence level, picture level, slice level, etc.) where they can be utilized. For example, parameters that can be utilized by one or more coded video sequences of different layers in the bitstream can be included in the video parameter set (VPS), and parameters utilized by one or more pictures in a coded video sequence can be included in the sequence parameter set (SPS). Similarly, parameters utilized by one or more slices in a picture can be included in the picture parameter set (PPS), and other parameters specific to a single slice can be included in the slice header. Similarly, an indication of which (which) parameter set a particular layer uses at a given time can be provided at various coding levels.

[0119] Due to the support for reference picture resampling (RPR) in VVC, the support for bitstreams containing multiple layers, e.g., two layers with SD and HD resolutions in VVC, can be designed without any additional signaling processing level codec tools because the upsampling required for spatial scalability support can use only the RPR upsampling filter. However, for scalability support, a high level of syntax change is required (compared to non - scalability support). The scalability support is specified in VVC version 1. Different from the scalability support in any early video codec standard (including the extensions of AVC and HEVC), the VVC scalability is designed to be as friendly as possible to the single - layer decoder design. The decoding capabilities of the multi - layer bitstream are specified in a way as if there were only a single layer in the bitstream. For example, the decoding capabilities such as the DPB size are specified in a way independent of the number of layers in the bitstream to be decoded. Basically, a decoder designed for a single - layer bitstream can decode a multi - layer bitstream with little change. Compared with the multi - layer extension design of AVC and HEVC, the HLS aspect has been significantly simplified at the cost of some flexibility. For example, an IRAP AU needs to include pictures of each layer present in the CVS.

[0120] 3.5. Random Access and Its Support in HEVC and VVC

[0121] Random access means accessing and decoding a bitstream starting from a picture that is not the first picture of the bitstream in decoding order. To support tuning and channel switching in broadcast / multicast and multiparty video conferencing, search in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points, which are usually intra - coded pictures but can also be inter - coded pictures (e.g., in the case of progressive decoding refresh).

[0122] HEVC includes signaling of Intra Random Access Point (IRAP) pictures in the NAL unit header by NAL unit type. Three types of IRAP pictures are supported, namely Instantaneous Decoder Refresh (IDR), Clean Random Access (CRA), and Broken Link Access (BLA) pictures. IDR pictures constrain the inter-picture prediction structure to not reference any pictures prior to the current group-of-pictures (GOP), which is commonly referred to as a closed-GOP random access point. By allowing certain pictures to reference pictures prior to the current GOP, CRA pictures are less restrictive and in the case of random access, all pictures are discarded. CRA pictures are commonly referred to as open-GOP random access points. BLA pictures typically result from the concatenation of two bitstreams or parts thereof in a CRA picture, for example during stream switching. To enable better system use of IRAP pictures in the system, a total of six different NAL units are defined to signal the properties of IRAP pictures, which can be used to better match the stream access point types defined in the ISO base media file format (ISOBMFF), which is used for random access support in Dynamic Adaptive Streaming over HTTP (DASH).

[0123] VVC supports three types of IRAP pictures, two types of IDR pictures (one type with or the other type without associated RADL pictures), and one type of CRA picture. These are basically the same as in HEVC. The BLA picture type in HEVC is not included in VVC, mainly for two reasons: i) The basic function of BLA pictures can be achieved by CRA pictures plus the sequence end NAL unit, the presence of which indicates that the subsequent pictures start a new CVS in a single-layer bitstream. ii) During the development of VVC, it was desired to specify fewer NAL unit types than in HEVC, such as by using five bits instead of six bits for the NAL unit type field in the NAL unit header to indicate.

[0124] Another key difference in random access support between VVC and HEVC is that GDR is supported in a more standardized way in VVC. In GDR, the decoding of the bitstream can start from an inter-coded picture, and although not the entire picture area can be correctly decoded at the beginning, after multiple pictures, the entire picture area will be correct. AVC and HEVC also support GDR, using recovery point SEI messages to signal GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and the recovery point is signaled in the picture header syntax structure. Starting with a GDR picture allows for CVS and the bitstream. This means that the entire bitstream is allowed to contain only inter-coded pictures without a single intra-coded picture. The main benefit of specifying GDR support in this way is to provide consistent behavior for GDR. GDR-enabled encoders smooth the bitrate of the bitstream by distributing intra-coded stripes or blocks across multiple pictures instead of intra-coding the entire picture, allowing for a significant reduction in end-to-end latency, which is considered more important today as ultra-low latency applications such as wireless displays, online games, and drone-based applications become increasingly popular.

[0125] Another GDR-related feature in VVC is virtual boundary signaling. The boundary between the refreshed area (i.e., the correctly decoded area) and the non-refreshed area at the pictures between a GDR picture and its recovery point can be signaled as a virtual boundary, and when signaled, the loop filter across the boundary will not be applied, so there will be no decoding mismatch for some samples at or near the boundary. This can be useful when the application determines to display the correctly decoded area during GDR processing.

[0126] IRAP pictures and GDR pictures can be collectively referred to as random access point (RAP) pictures.

[0127] 3.6. Reference Picture Management and Reference Picture List (RPL)

[0128] Reference picture management is a core function essential for any video coding scheme that uses inter-prediction. It manages the storage and removal of reference pictures in the decoded picture buffer (DPB) and places the reference pictures in the RPL in their correct order.

[0129] The reference picture management of HEVC, including reference picture marking and removal from the decoded picture buffer (DPB) as well as reference picture list construction (RPLC), is different from that of AVC. Instead of the reference picture marking mechanism based on a sliding window plus adaptive memory management control operation (MMCO) in AVC, HEVC specifies a reference picture management and marking mechanism based on the so-called reference picture set (RPS), and thus RPLC is based on the RPS mechanism. The RPS consists of a set of reference pictures associated with a picture, including all reference pictures in decoding order before the associated picture, which can be used for inter prediction of the associated picture or any picture after the associated picture in decoding order. The reference picture set includes five reference picture lists. The first three lists contain all reference pictures that can be used for inter prediction of the current picture and can be used for inter prediction of one or more pictures after the current picture in decoding order. The other two lists consist of all reference pictures that are not used in the inter prediction of the current picture but can be used for inter prediction of one or more pictures after the current picture in decoding order. The RPS provides "intra-coded" signaling notification of the DPB state, rather than "inter-coded" signaling notification as in AVC, mainly to improve fault tolerance. The RPLC process in HEVC is based on the RPS, by signaling an index for each reference index to an RPS subgroup; this process is simpler than the RPLC process in AVC.

[0130] The reference picture management in VVC is more similar to HEVC than to AVC, but is simpler and more robust. As in those standards, two RPLs, list 0 and list 1, are derived, but they are not based on the reference picture set concept used in HEVC or the automatic sliding window process used in AVC; instead, they are signaled more directly. The reference pictures of the RPL are listed as active and inactive entries, and only active entries can be used as reference indices for inter prediction of the CTUs of the current picture. Inactive entries indicate other pictures to be hosted in the DPB for reference of other pictures arriving later in the bitstream.

[0131] 3.7. Parameter Sets

[0132] AVC, HEVC, and VVC specify parameter sets. The types of parameter sets include SPS, PPS, APS, and VPS. AVC, HEVC, and VVC all support SPS and PPS. VPS was introduced starting from HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0133] The SPS is designed to carry sequence-level header information, while the PPS is designed to carry picture-level header information that does not change frequently. By using the SPS and PPS, it is not necessary to repeat the information that does not change frequently for each sequence or picture, thus avoiding redundant signaling of this information. In addition, the use of the SPS and PPS enables out-of-band transmission of important header information, which not only avoids the need for redundant transmission but also improves fault tolerance.

[0134] The VPS is introduced to carry sequence-level header information common to all layers in a multi-layer bitstream.

[0135] The APS is introduced to carry picture-level or slice-level information that requires a significant number of bits to encode and decode, can be shared by multiple pictures, and can have a significant number of different variations in the sequence.

[0136] 3.8. Related Definitions in VVC

[0137] The related definitions in the latest VVC text (JVET-Q2001-vE / v15) are as follows.

[0138] Associated IRAP picture (of a specific picture): The previous IRAP picture (when present) in decoding order has the same value as the nuh_layer_id of the specific picture.

[0139] Clean Random Access (CRA) PU: A PU for which the decoded picture is a CRA picture.

[0140] Clean Random Access (CRA) picture: An IRAP picture in which each VCL NAL unit includes a nal_unit_type equal to CRA_NUT.

[0141] Coded Video Sequence (CVS): A sequence of AUs consisting of CVSS AUs in decoding order, followed by zero or more AUs that are not CVSS AUs, including all subsequent AUs but excluding any subsequent AUs that are CVSS AUs.

[0142] Coded Video Sequence Start (CVSS) AU: An AU in which each layer in the CVS has a PU and the decoded picture in each PU is a CLVSS picture.

[0143] Gradual Decoding Refresh (GDR) AU: An AU in which the decoded picture in each current PU is a GDR picture.

[0144] Gradual Decoding Refresh (GDR) PU: A PU for which the decoded picture is a GDR picture.

[0145] Gradual Decoding Refresh (GDR) Picture: Each VCL NAL unit includes a picture with a nal_unit_type equal to GDR_NUT.

[0146] Instantaneous Decoding Refresh (IDR) PU: A PU whose decoded picture is an IDR picture.

[0147] Instantaneous Decoding Refresh (IDR) Picture: Each VCL NAL unit includes an IRAP picture with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP.

[0148] Intra Random Access Point (IRAP) AU: An AU in which each layer in the CVS has a PU and the decoded picture in each PU is an IRAP picture.

[0149] Intra Random Access Point (IRAP) PU: A PU whose decoded picture is an IRAP picture.

[0150] Intra Random Access Point (IRAP) Picture: All VCL NAL units include decoded pictures with the same value of nal_unit_type in the range from IDR_W_RADL to CRA_NUT (inclusive).

[0151] Preceding Picture: A picture in the same layer as the associated IRAP picture and preceding the associated IRAP picture in output order.

[0152] Output Order: The order in which decoded pictures are output from the DPB (for decoded pictures to be output from the DPB).

[0153] Random Access Decodable Leading (RADL) PU: A PU whose decoded picture is a RADL picture.

[0154] Random Access Decodable Leading (RADL) Picture: Each VCL NAL unit includes a picture with a nal_unit_type equal to RADL_NUT.

[0155] Random Access Skippable Leading (RASL) PU: A PU whose decoded picture is a RASL picture.

[0156] Random Access Skippable Leading (RASL) Picture: Each VCL NAL unit includes a picture with a nal_unit_type equal to RASL_NUT.

[0157] Stepwise Temporal Sub-layer Access (STSA) PU: A PU whose decoded picture is a STSA picture.

[0158] Stepwise Temporal Sub-layer Access (STSA) Picture: Each VCL NAL unit includes a picture with nal_unit_type equal to STSA_NUT.

[0159] Note – An STSA picture does not use pictures with the same TemporalId as the STSA picture for inter-prediction reference. A picture with the same TemporalId as the STSA picture that comes after the STSA picture in decoding order does not use a picture with the same TemporalId as the STSA picture that comes before the STSA picture in decoding order for inter-prediction reference. An STSA picture enables switching at the STSA picture from the immediately lower sub-layer to the sub-layer containing the STSA picture. The TemporalId of an STSA picture must be greater than 0.

[0160] Sub-picture: A rectangular region of one or more slices within a picture.

[0161] Trailing Picture: A non-IRAP picture that comes after the associated IRAP picture in output order and is not an STSA picture.

[0162] Note – A trailing picture associated with an IRAP picture also comes after the IRAP picture in decoding order. A picture that comes after the associated IRAP picture in output order and before the associated IRAP picture in decoding order is not allowed.

[0163] 3.9 NAL Unit Header Syntax and Semantics in VVC

[0164] In the latest VVC text (in JVET-Q2001-vE / v15), the NAL unit header syntax and semantics are as follows.

[0165] 7.3.1.2 NAL Unit Header Syntax

[0166]

[0167]

[0168] 7.4.2.2 NAL Unit Header Semantics

[0169] forbidden_zero_bit shall be equal to 0.

[0170] nuh_reserved_zero_bit shall be equal to 0. The value 1 of nuh_reserved_zero_bit may be specified by ITU-T|ISO / IEC in the future. The decoder shall ignore (i.e., remove and discard from the bitstream) NAL units with nuh_reserved_zero_bit equal to 1.

[0171] The nuh_layer_id specifies the identifier of the layer to which the VCL NAL unit belongs or the identifier of the layer to which the non-VCL NAL unit applies. The value of nuh_layer_id shall be in the range of 0 to 55 (inclusive). Other values of nuh_layer_id shall be reserved for future use by ITU-T|ISO / IEC.

[0172] The value of nuh_layer_id shall be the same for all VCL NAL units of a coded picture. The value of nuh_layer_id of a coded picture or PU is the value of nuh_layer_id of the VCL NAL units of the coded picture or PU.

[0173] The nuh_layer_id values for AUD, PH, EOS, and FD NAL units are subject to the following constraints:

[0174] – If nal_unit_type is equal to AUD_NUT, then nuh_layer_id shall be equal to vps_layer_id[0].

[0175] – Otherwise, when nal_unit_type is equal to PH_NUT, EOS_NUT, or FD_NUT, nuh_layer_id shall be equal to the nuh_layer_id of the associated VCL NAL unit.

[0176] Note 1 – The nuh_layer_id values for DCI, VPS, and EOB NAL units are not constrained.

[0177] The value of nal_unit_type shall be the same for all pictures of a CVSS AU.

[0178] nal_unit_type specifies the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0179] NAL units with unspecified semantics in the range of UNSPEC_28..UNSPEC_31 (inclusive) shall not affect the decoding process specified in this specification.

[0180] Annotation 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 can be used as determined by the application. The decoding process for these values of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, special care must be taken in practice when designing an encoder that generates NAL units with these nal_unit_type values, and when designing a decoder that interprets the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values may only apply in contexts where "conflicts" (i.e., different definitions of the meaning of NAL unit content with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in a control application or transport specification, or through the context of a controlled bitstream distribution).

[0181] For purposes other than determining the amount of data in the DU of a bitstream (as specified in Annex C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units that use the reserved values of nal_unit_type.

[0182] Annotation 3 – This requirement allows for future definition of compatible extensions to this specification.

[0183] Table 5 – NAL unit type codes and NAL unit type classifications

[0184]

[0185]

[0186] Annotation 4 – A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream.

[0187] Annotation 5 – An instant decoding refresh (IDR) picture with a nal_unit_type equal to IDR_N_LP has no associated preceding picture in the bitstream. An IDR picture with a nal_unit_type equal to IDR_W_RADL has no associated RASL picture present in the bitstream, but may have associated RADL pictures in the bitstream.

[0188] The nal_unit_type values of all VCL NAL units in a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0189] For the VCL NAL units of any particular picture, the following applies:

[0190] – If mixed_nalu_types_in_pic_flag equals 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0191] – Otherwise (mixed_nalu_types_in_pic_flag equals 1), the picture shall have at least two sub - pictures, and the VCL NAL units of the picture shall have exactly two different values of nal_unit_type as follows: The VCL NAL units of at least one sub - picture of the picture shall all have a nal_unit_type with a specific value equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP or CRA_NUT, while the VCL NAL units of the other sub - pictures in the picture shall all have a different specific value equal to TRAIL_NUT, RADL_NUT, RASL_NUT.

[0192] For a single - layer bitstream, the following constraints apply:

[0193] – Except for the first picture in the bitstream in decoding order, each picture is considered to be associated with the previous IRAP picture in decoding order.

[0194] – When a picture is a pre - picture of an IRAP picture, it shall be a RADL or RASL picture.

[0195] – When a picture is a post - picture of an IRAP picture, it shall not be a RADL or RASL picture.

[0196] – There shall be no RASL pictures associated with IDR pictures in the bitstream.

[0197] – There shall be no RADL pictures associated with IDR pictures having a nal_unit_type equal to IDR_N_LP in the bitstream.

[0198] Note 6 – Random access can be performed at the location of an IRAP PU by discarding all PUs before the IRAP PU (and decoding the IRAP picture and all subsequent non - RASL pictures correctly in decoding order), provided that each parameter set is available when referenced (either in the bitstream or by an external means not specified in this specification).

[0199] – Any picture before the IRAP picture in decoding order shall be before the IRAP picture in output order and shall be before any RADL picture associated with the IRAP picture in output order.

[0200] – Any RASL picture associated with a CRA picture in output order shall be before any RADL picture associated with the CRA picture.

[0201] – Any RASL picture associated with a CRA picture in output order shall be after any IRAP picture before the CRA picture in decoding order.

[0202] – If field_seq_flag equals 0 and the current picture is a prefix picture associated with an IRAP picture, then in decoding order it shall be before all non-prefix pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last prefix pictures associated with the IRAP picture in decoding order, there shall be at most one non-prefix picture before picA in decoding order, and there shall be no non-prefix pictures between picA and picB in decoding order.

[0203] nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit.

[0204] The value of nuh_temporal_id_plus1 shall not be equal to 0.

[0205] The derivation of the variable TemporalId is as follows:

[0206] TemporalId = nuh_temporal_id_plus1 - 1 (36)

[0207] When nal_unit_type is in the range from IDR_W_RADL to RSV_IRAP_12 (inclusive), TemporalId shall be equal to 0.

[0208] When nal_unit_type equals STSA_NUT and vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] equals 1, TemporalId shall not be equal to 0.

[0209] For all VCL NAL units of an AU, the value of TemporalId shall be the same. The value of TemporalId for a coded picture, PU, or AU is the value of TemporalId for the VCL NAL units of the coded picture, PU, or AU. The value of TemporalId for a sublayer representation is the maximum value of TemporalId for all VCL NAL units in the sublayer representation.

[0210] The value of TemporalId for non-VCL NAL units is subject to the following constraints:

[0211] – If nal_unit_type is equal to DCI_NUT, VPS_NUT, or SPS_NUT, then TemporalId shall be equal to 0, and the TemporalId of the AU containing the NAL unit shall be equal to 0.

[0212] – Otherwise, if nal_unit_type is equal to PH_NUT, then TemporalId shall be equal to the TemporalId of the PU containing the NAL unit.

[0213] – Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, then TemporalId shall be equal to 0.

[0214] – Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, then TemporalId shall be equal to the TemporalId of the AU containing the NAL unit.

[0215] – Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId shall be greater than or equal to the TemporalId of the PU containing the NAL unit.

[0216] Note 7 – When the NAL unit is a non-VCL NAL unit, the value of TemporalId is equal to the minimum value of the TemporalId values for all AUs applicable to the non-VCL NAL unit. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId may be greater than or equal to the TemporalId of the containing AU because all PPSs and APSs can be included at the beginning of the bitstream (e.g., when they are transmitted out-of-band and the receiver places them at the beginning of the bitstream), where the TemporalId of the first coded picture is equal to 0.

[0217] 3.10. Mixed NAL Unit Types in Picture

[0218] 7.4.3.4 Picture Parameter Set Semantics ...

[0220] When mixed_nalu_types_in_pic_flag equals 1, it is specified that each picture that refers to the PPS has more than one VCL NAL unit, the VCL NAL units do not have the same nal_unit_type value, and the picture is not an IRAP picture. When mixed_nalu_types_in_pic_flag equals 0, it is specified that each picture that refers to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture that refers to the PPS have the same nal_unit_type value.

[0221] When no_mixed_nalu_types_in_pic_constraint_flag equals 1, the value of mixed_nalu_types_in_pic_flag shall equal 0.

[0222] For each slice with a nal_unit_type value nalUnitTypeA in the range from IDR_W_RADL to CRA_NUT (inclusive), in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag of picture picA equals 1), the following applies:

[0223] – The slice shall belong to a sub-picture subpicA for which the value of subpic_treated_as_pic_flag[i] equals 1.

[0224] – The slice shall not belong to a sub-picture of picA that contains VCL NAL units and for which nal_unit_type is not equal to nalUnitTypeA.

[0225] – If nalUnitTypeA equals CRA, for all subsequent PUs in CLVS after the current picture in decoding order and in output order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.

[0226] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS that are after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that is prior to picA in decoding order in the active entries.

[0227] Annotation 1 – mixed_nalu_types_in_pic_flag being equal to 1 indicates that the pictures referring to the PPS contain slices with different NAL unit types, e.g., coded pictures resulting from a sub-picture bitstream merge operation, where the encoder must ensure the bitstream structure match and further alignment of the original bitstream parameters. An example of such alignment is as follows: when the value of sps_idr_rpl_present_flag is equal to 0 and mixed_nalu_types_in_pic_flag is equal to 1, the pictures referring to the PPS shall not have slices with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0229] 3.11. Picture Header Structure Syntax and Semantics in VVC

[0230] In the latest VVC text (in JVET-Q2001-vE / v15), the picture header structure syntax and semantics most relevant to the technical solution of this article are as follows.

[0231] 7.3.2.7 Picture Header Structure Syntax

[0232]

[0233]

[0234] 7.4.3.7 Picture Header Structure Semantics

[0235] The PH syntax structure contains information that is common to all slices of the coded picture associated with the PH syntax structure.

[0236] gdr_or_irap_pic_flag being equal to 1 specifies that the current picture is a GDR or IRAP picture. gdr_or_irap_pic_flag being equal to 0 specifies that the current picture may or may not be a GDR or IRAP picture.

[0237] When gdr_pic_flag equals 1, it is specified that the picture associated with PH is a GDR picture. When gdr_pic_flag equals 0, it is specified that the picture associated with PH is not a GDR picture. When not present, the value of gdr_pic_flag is inferred to be equal to 0. When gdr_enabled_flag equals 0, the value of gdr_pic_flag shall be equal to 0.

[0238] Note 1 – When gdr_or_irap_pic_flag equals 1 and gdr_pic_flag equals 0, the picture associated with PH is an IRAP picture. ...

[0240] ph_pic_order_cnt_lsb specifies the picture order count modulo MaxPicOrderCntLsb of the current picture. The length of the ph_pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4 + 4 bits. The value of ph_pic_order_cnt_lsb shall be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0241] no_output_of_prior_pics_flag affects the output of previously decoded pictures in the DPB after decoding a CLVSS picture that is not the first picture in the bitstream specified in Annex C.

[0242] recovery_poc_cnt specifies the recovery point for decoding pictures in output order. If the current picture is a GDR picture associated with PH, and in the CLVS with PicOrderCntVal there is a picture picA that is in decoding order after the current GDR picture and PicOrderCntVal is equal to the value of the PicOrderCntVal of the current GDR picture plus the value of recovery_poc_cnt, then picture picA is called a recovery point picture. Otherwise, the first picture whose PicOrderCntVal in output order is greater than the value of the PicOrderCntVal of the current picture plus the value of recovery_poc_cnt is called a recovery point picture. The recovery point picture shall not be in decoding order before the current GDR picture. The value of recovery_poc_cnt shall be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0243] When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0244] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)

[0245] Annotation 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to the RpPicOrderCntVal of the associated GDR picture, the current and subsequent decoded pictures in the output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when it exists) before the associated GDR picture in the decoding order. ...

[0247] 3.12 Constraints on RPL in VVC

[0248] In the latest VVC text (in JVET-Q2001-vE / v15), the constraints on RPL in VVC are as follows (as part of the decoding process of VVC clause 8.3.2 Reference Picture List construction).

[0249] 8.3.2 Decoding process of Reference Picture List construction ...

[0251] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called the active entries in RefPicList[i], and the other entries in RefPicList[i] are called the non-active entries in RefPicList[i].

[0252] Annotation 2 – A particular picture may be involved in entries in both RefPicList[0] and RefPicList[1]. It is also possible for a particular picture to be involved in more than one entry in RefPicList[0] or more than one entry in RefPicList[1].

[0253] Annotation 3 – The active entries in RefPicList[0] and the active entries in RefPicList[1] together refer to all the reference pictures that can be used for inter prediction of the current picture and one or more pictures after the current picture in the decoding order. The non-active entries in RefPicList[0] and the non-active entries in RefPicList[1] together refer to all the reference pictures that are not used for inter prediction of the current picture but can be used for inter prediction of one or more pictures after the current picture in the decoding order.

[0254] Note 4 – One or more entries in RefPicList[0] or RefPicList[1] may be equal to "no reference picture" because the corresponding picture does not exist in the DPB. Each inactive entry in RefPicList[0] or RefPicList[0] that is equal to "no reference picture" shall be ignored. An unexpected picture loss shall be inferred for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference picture".

[0255] One requirement for bitstream conformance is to apply the following constraints:

[0256] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].

[0257] – The picture referred to by each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB and the TemporalId shall be less than or equal to the TemporalId of the current picture.

[0258] – The picture referred to by each entry in RefPicList[0] or RefPicList[1] shall not be the current picture and the non_reference_picture_flag shall be equal to 0.

[0259] – The STRP entry in RefPicList[0] or RefPicList[1] for a strip of a picture and the LTRP entry in RefPicList[0] or RefPicList[1] for the same strip or a different strip of the same picture shall not refer to the same picture.

[0260] – There shall not be an LTRP entry in RefPicList[0] or RefPicList[1] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referred to by that entry is greater than or equal to 2 24 .

[0261] – Let setOfRefPics be the unique set of pictures involved by all entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize-1 (inclusive), where MaxDpbSize is specified in clause A.4.2 and setOfRefPics shall be the same for all slices of the picture.

[0262] – When the nal_unit_type of the current slice is equal to STSA_NUT, there shall be no active entry in RefPicList[0] or RefPicList[1] whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0263] – When the current picture is a picture that comes after the STSA picture in decoding order and the STSA picture has the same TemporalId as the current picture and the same nuh_layer_id as the current picture, there shall be no picture that comes before the STSA picture in decoding order and has the same TemporalId as the current picture and the same nuh_layer_id as the current picture, which is included as an active entry in RefPicList[0] or RefPicList[1].

[0264] – When the current picture is a CRA picture, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] that comes before any previous IRAP picture (when present) in decoding order, either in output order or in decoding order.

[0265] – When the current picture is a post picture, there shall be no picture involved by an active entry in RefPicList[0] or RefPicList[1] that is generated by the decoding process of an unavailable reference picture that generates an IRAP picture associated with the current picture.

[0266] – When the current picture is a post picture that is after one or more pre - pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no picture involved in the entry in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures associated with the current picture.

[0267] – When the current picture is a recovery point picture or a picture that is after the recovery point picture in output order, there shall be no entry in RefPicList[0] or RefPicList[1] that contains a picture generated by the decoding process of generating unavailable reference pictures for the GDR picture of the recovery point picture.

[0268] – When the current picture is a post picture, there shall be no picture involved in the active entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture either in output order or in decoding order.

[0269] – When the current picture is a post picture that is after one or more pre - pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no picture involved in the entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture either in output order or in decoding order.

[0270] – When the current picture is a RADL picture, there shall not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0271] ο RASL picture

[0272] ο A picture generated by the decoding process of generating unavailable reference pictures

[0273] ο A picture that is before the associated IRAP picture in decoding order

[0274] – The picture involved in each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall be in the same AU as the current picture.

[0275] – The picture involved in each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall exist in the DPB, and the nuh_layer_id shall be less than the nuh_layer_id of the current picture.

[0276] – Each ILRP entry in RefPicList[0] or RefPicList[1] of a slice shall be an active entry. ...

[0278] 3.13. Setting of PictureOutputFlag

[0279] In the latest VVC text (in JVET-Q2001-vE / v15), the specification for setting the value of the variable PictureOutputFlag is as follows (as part of the decoding process of Clause 8.1.2 for coded pictures).

[0280] 8.1.2 Decoding process of coded pictures

[0281] The decoding process specified in this clause applies to each coded picture in BitstreamToDecode, referred to as the current picture and represented by the variable CurrPic.

[0282] Depending on the value of chroma_format_idc, the number of sample arrays of the current picture is as follows:

[0283] – If chroma_format_idc is equal to 0, the current picture consists of 1 sample array S L .

[0284] – Otherwise (chroma_format_idc is not equal to 0), the current picture consists of 3 sample arrays S L , S Cb , S Cr .

[0285] The decoding process of the current picture takes the syntax elements from Clause 7 and the uppercase variables as input. When interpreting the semantics of each NAL unit and each syntax element in the remainder of Clause 8, the term "bitstream" (or a part thereof, such as the CVS of the bitstream) refers to BitstreamToDecode (or a part thereof).

[0286] Depending on the value of separate_colour_plane_flag, the structure of the decoding process is as follows:

[0287] – If separate_colour_plane_flag is equal to 0, the decoding process is called once and the current picture is output.

[0288] – Otherwise (separate_colour_plane_flag equals 1), call the decoding process three times. The input to the decoding process is all the NAL units of the coded and decoded pictures with the same colour_plane_id value. Specifying the decoding process for NAL units with a specific colour_plane_id value is as if only the CVS of the monochrome colour format with that specific colour_plane_id value existed in the bitstream. The output of each of the three decoding processes is assigned to one of the 3 sample arrays of the current picture, and the NAL units with colour_plane_id equal to 0, 1, and 2 are assigned to S L , S Cb and S Cr .

[0289] Note – When separate_colour_plane_flag equals 1 and chroma_format_idc equals 3, the derived variable ChromaArrayType equals 0. In the decoding process, the value of this variable is evaluated, resulting in the same operations as for a monochrome picture (when chroma_format_idc equals 0).

[0290] For the current picture CurrPic, the decoding process operates as follows:

[0291] 1. Decode the NAL units as specified in Clause 8.2.

[0292] 2. The processing in Clause 8.3 specifies the following decoding process using the syntax elements in the slice header layer and above:

[0293] – Derive the variables and functions related to picture order counting as specified in Clause 8.3.1. It only needs to be called for the first slice of the picture.

[0294] – At the start of the decoding process for each slice of a non-IDR picture, call the decoding process for reference picture list construction specified in Clause 8.3.2 to derive reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]).

[0295] – Call the decoding process for reference picture marking in Clause 8.3.3, where the reference picture can be marked as "not used for reference" or "used for long-term reference". It only needs to be called for the first slice of the picture.

[0296] – When the current picture is a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, call the decoding process for generating unavailable reference pictures as specified in Subclause 8.3.4, which needs to be called only for the first slice of the picture.

[0297] – PictureOutputFlag is set as follows:

[0298] – If one of the following conditions is true, then PictureOutputFlag is set to be equal to 0:

[0299] – The current picture is a RASL picture and the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1.

[0300] – gdr_enabled_flag is equal to 1 and the current picture is a GDR picture with NoOutputBeforeRecoveryFlag equal to 1.

[0301] – gdr_enabled_flag is equal to 1, the current picture is associated with a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current picture is less than the RpPicOrderCntVal of the associated GDR picture.

[0302] – sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions:

[0303] – The PictureOutputFlag of PicA is equal to 1.

[0304] – The nuh_layer_id nuhLid of PicA is greater than the nuh_layer_id of the current picture.

[0305] – PicA belongs to the output layer of OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0306] – The sps_video_parameter_set_id is greater than 0, the ols_mode_idc is equal to 2, and the ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0307] – Otherwise, PictureOutputFlag is set to be equal to pic_output_flag.

[0308] 3. The processing regulations in Clauses 8.4, 8.5, 8.6, 8.7, and 8.8 use the decoding process of the syntax elements in all syntax structure layers. The requirement for bitstream consistency is that the coded strip of a picture will contain the strip data of each CTU of the picture, such that dividing the picture into strips and dividing the strips into CTUs each forms a segmentation of the picture.

[0309] 4. After all strips of the current picture have been decoded, the currently decoded picture is marked as "for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "for short-term reference".

[0310] 4. Technical problems solved by the disclosed technical solution

[0311] The existing designs in the latest VVC text (JVET-Q2001-vE / v15) have the following problems:

[0312] 1) Since different types of sub-pictures are allowed to be mixed in a picture, it is confusing to refer to the content of the NAL unit of the VCL NAL unit type as the coded strip of a specific type of picture. For example, an NAL unit with a nal_unit_type equal to CRA_NUT is the coded strip of a CRA picture only when the nal_unit_type of all strips of the picture is equal to CRA_NUT; when the nal_unit_type of one strip of the picture is not equal to CRA_NUT, then the picture is not a CRA picture.

[0313] 2) Currently, if a sub-picture contains VCL NAL units with nal_unit_type in the range from IDR_W_RADL to CRA_NUT (inclusive), the value of subpic_treated_as_pic_flag[] for the sub-picture needs to be equal to 1, and the mixed_nalu_types_in_pic_flag for the picture is equal to 1. In other words, for an IRAP sub-picture mixed with another type of sub-picture in a picture, the value of subpic_treated_as_pic_flag[] needs to be equal to 1. However, with more support for the mixing of VCL NAL unit types, this requirement is insufficient.

[0314] 3) Currently, at most two different types of VCL NAL units (and two different types of sub-pictures) are allowed in a picture.

[0315] 4) In single-layer and multi-layer contexts, there is a lack of constraints on the output order of post sub-pictures relative to associated IRAP or GDR sub-pictures.

[0316] 5) Currently, it is stipulated that when a picture is a pre-picture of an IRAP picture, it should be a RADL or RASL picture. This constraint, together with the definition of pre- / RADL / RASL pictures, does not allow the mixing of RADL and RASL NAL unit types within a picture generated by the mixing of two CRA pictures and their associated RADL and RASL pictures that are not AU-aligned.

[0317] 6) In single-layer and multi-layer contexts, there is a lack of constraints on the sub-picture type of pre-sub-pictures (i.e., the NAL unit type of VCL NAL units in the sub-picture).

[0318] 7) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RASL sub-picture can exist and be associated with an IDR sub-picture.

[0319] 8) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RADL sub-picture can exist and be associated with an IDR sub-picture with a nal_unit_type equal to IDR_N_LP.

[0320] 9) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between a sub-picture before an IRAP sub-picture in decoding order and a RADL sub-picture associated with the IRAP sub-picture.

[0321] 10) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between a sub-picture before a GDR sub-picture in decoding order and a sub-picture associated with the GDR sub-picture.

[0322] 11) In single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with a CRA sub-picture and the RADL sub-pictures associated with the CRA sub-picture.

[0323] 12) In single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with a CRA sub-picture and the IRAP sub-pictures that are before the CRA sub-picture in decoding order.

[0324] 13) In single-layer and multi-layer contexts, there is a lack of constraint on the relative decoding order between the associated non-preceding pictures and the preceding pictures of an IRAP picture.

[0325] 14) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are after a STSA sub-picture in decoding order.

[0326] 15) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of a CRA sub-picture.

[0327] 16) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that refer to pictures generated by a decoding process for generating unavailable reference pictures.

[0328] 17) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that refer to pictures generated by a decoding process for generating unavailable reference pictures.

[0329] 18) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are associated with an IRAP picture and are after the IRAP picture in output order.

[0330] 19) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that are associated with an IRAP picture and are after the IRAP picture in output order.

[0331] 20) In single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of a RADL sub-picture.

[0332] 5. Examples of Technical Solutions and Embodiments

[0333] To solve the above problems and other problems, the following summarized methods are disclosed. These items should be considered as examples for explaining general concepts and should not be interpreted in a narrow way. In addition, these items can be used alone or combined in any way.

[0334] 1) To solve Problem 1, instead of designating the content of a NAL unit with a VCL NAL unit type as a "coding strip of a specific type of picture", it is designated as a "coding strip of a specific type of picture or sub - picture". For example, the content of a NAL unit with a nal_unit_type equal to CRA_NUT is designated as a "coding strip of a CRA picture or sub - picture".

[0335] a. In addition, one or more of the following terms are defined: associated GDR sub - picture, associated IRAP sub - picture, CRA sub - picture, GDR sub - picture, IDR sub - picture, IRAP sub - picture, pre - sub - picture, RADL sub - picture, RASL sub - picture, STSA sub - picture, post - sub - picture.

[0336] 2) To solve Problem 2, a constraint is added to require that subpic_treated_as_pic_flag[] of any two adjacent sub - pictures of different NAL unit types should be equal to 1.

[0337] a. In one example, the constraint is specified as follows: for any two adjacent sub - pictures in a picture with sub - picture indices i and j, when subpic_treated_as_pic_flag[i] or subpic_treated_as_pic_flag[j] is equal to 0, the two sub - pictures should have the same NAL unit type.

[0338] a. Optionally, it is required that when subpic_treated_as_pic_flag[i] of any sub - picture with sub - picture index i is equal to 0, all sub - pictures in the picture should have the same NAL unit type (i.e., all VCL NAL units in the picture should have the same NAL unit type, i.e., the value of mixed_nalu_types_in_pic_flag should be equal to 0). And this means that when all sub - pictures make their corresponding subpic_treated_as_pic_flag[] equal to 1, mixed_nalu_types_in_pic_flag can only be equal to 1.

[0339] 3) To solve Problem 3, when mixed_nalu_types_in_pic_flag is equal to 1, a picture can be allowed to contain more than two different types of VCL NAL units.

[0340] 4) To solve Problem 4, it is specified that the post - sub - picture should be after the associated IRAP or GDR sub - picture in the output order.

[0341] 5) To solve Problem 5, in order to allow the mixing of RADL and RASL NAL unit types in pictures resulting from the mixing of two CRA pictures and their non-AU-aligned associated RADL and RASL pictures, the existing constraint that the preceding picture specifying an IRAP picture should be a RADL or RASL picture is changed as follows: When a picture is a preceding picture of an IRAP picture, the nal_unit_type value of all VCL NAL units in the picture should be equal to RADL_NUT or RASL_NUT. In addition, in the decoding process of pictures using a mixed nal_unit_type value of RADL_NUT and RASL_NUT, when the layer containing the picture is the output layer, the PictureOutputFlag of the picture is set to be equal to pic_output_flag.

[0342] Thus, although when the NoOutputBeforeRecoveryFlag associated with the CRA picture is equal to 1, the guarantee of the "correctness" of the RASL sub-pictures of the "intermediate values" in such pictures is also appropriate, but it is actually not necessary. However, by having all output pictures meet the correct constraints of the decoder, the RADL sub-pictures in such pictures can be guaranteed. The unnecessary part of the guarantee is insignificant and does not increase the complexity of implementing a conforming encoder or decoder. In this case, it would be useful to add a NOTE to clarify that although such RASL sub-pictures associated with CRA pictures with a NoOutputBeforeRecoveryFlag equal to 1 can be output by the decoding process, they are not intended for display and thus are not applied to display.

[0343] 6) To solve Problem 6, it is stipulated that when a sub-picture is a preceding sub-picture of an IRAP sub-picture, it should be a RADL or RASL sub-picture.

[0344] 7) To solve Problem 7, it is stipulated that there should be no RASL sub-pictures associated with IDR sub-pictures in the bitstream.

[0345] 8) To solve Problem 8, it is stipulated that there should be no RADL sub-pictures associated with IDR sub-pictures with a nal_unit_type equal to IDR_N_LP in the bitstream.

[0346] 9) To solve problem 9, it is stipulated that any sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx shall, in the output order, be before the IRAP sub-picture and all its associated RADL sub-pictures, and this any sub-picture shall, in the decoding order, be before the IRAP sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx.

[0347] 10) To solve problem 10, it is stipulated that any sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx shall, in the output order, be before the GDR sub-picture and all its associated sub-pictures, and this any sub-picture shall, in the decoding order, be before the GDR sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx.

[0348] 11) To solve problem 11, it is stipulated that any RASL sub-picture associated with a CRA sub-picture shall, in the output order, be before any RADL sub-picture associated with the CRA sub-picture.

[0349] 12) To solve problem 12, it is stipulated that any RASL sub-picture associated with a CRA sub-picture shall, in the output order, be after any IRAP sub-picture that is before the CRA sub-picture in the decoding order.

[0350] 13) To solve problem 13, it is stipulated that if field_seq_flag is equal to 0, and the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is a pre-sub-picture associated with an IRAP sub-picture, then it shall, in the decoding order, be before all non-pre-sub-pictures associated with the same IRAP sub-picture; otherwise, let subpicA and subpicB be the first and last pre-sub-pictures associated with the IRAP sub-picture in the decoding order respectively. There shall be at most one non-pre-sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx before subpicA in the decoding order, and there shall be no non-pre-pictures with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx between picA and picB in the decoding order.

[0351] 14) To solve problem 14, it is stipulated that when the current sub-picture with a TemporalId equal to a specific value tId, a nuh_layer_id equal to a specific value layerId, and a sub-picture index equal to a specific value subpicIdx is a sub-picture that comes after the STSA sub-picture with a TemporalId equal to tId, a nuh_layer_id equal to layerId, and a sub-picture index equal to subpicIdx in decoding order, there should be no picture with a TemporalId equal to tId and a nuh_layer_id equal to layerId that comes before the picture containing the STSA sub-picture in decoding order and is included as an active entry in RefPicList[0] or RefPicList[1].

[0352] 15) To solve problem 15, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is a CRA sub-picture, there should be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that comes before any picture (when it exists) of the previous IRAP sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx in output order or in decoding order.

[0353] 16) To solve problem 16, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is not a sub-picture of a RASL sub-picture associated with a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR sub-picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved in an active entry in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures.

[0354] 17) To solve problem 17, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is not a CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a sub-picture before the same CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a pre-sub-picture associated with the CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR sub-picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a sub-picture of a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates an unavailable reference picture.

[0355] 18) To solve problem 18, it is stipulated that when the current sub-picture is associated with an IRAP sub-picture and is after the IRAP sub-picture in output order, there should be no picture involved by an active entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub-picture in output order or in decoding order.

[0356] 19) To solve problem 19, it is stipulated that when the current sub-picture is associated with an IRAP sub-picture, after the IRAP sub-picture in output order, and after the pre-sub-picture associated with the same IRAP sub-picture (if any) in both decoding order and output order, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub-picture in output order or in decoding order.

[0357] 20) To solve problem 20, it is stipulated that when the current sub-picture is a RADL sub-picture, there should not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0358] a. A picture containing a RASL sub-picture

[0359] b. A picture before the picture containing the associated IRAP sub-picture in decoding order

[0360] 6. Embodiments

[0361] The following are some example embodiments in terms of the technical solutions summarized in Section 5 above, which can be applied to the VVC specification. The modified text is based on the latest VVC text in JVET-Q2001-vE / v15. Most of the relevant parts added or modified are highlighted, and some of the deleted parts are highlighted with double brackets on the left and right (e.g., [[ ]]), and the deleted text is between the double brackets. There are also some other changes that are editorial or not part of this technical solution and are thus not highlighted.

[0362] 6.1. First Embodiment

[0363] This embodiment is for items 1, 1a, 2, 2a, 4, and 6 to 20.

[0364] 3 Definitions ...

[0366] The previous GDR with nuh_layer_id equal to layerId (when present), there is no IRAP with nuh_layer_id equal to layerId between this picture and a specific one.

[0367]

[0368] The previous IRAP with nuh_layer_id equal to layerId (when present), there is no GDR with nuh_layer_id equal to layerId between this picture and a specific one.

[0369]

[0370] An IRAP picture with nal_unit_type equal to CRA_NUT for each VCL NAL unit.

[0371]

[0372] Each layer in the CVS has one PU and in each current PU​​​ All are AUs of GDR pictures.

[0373] Each VCL NAL unit includes a .

[0374]

[0375]

[0376] Each VCL NAL includes an IRAP with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP .

[0377]

[0378] All VCL NALs include the same value of nal_unit_type within the range from IDR_W_RADL to CRA_NUT (including the endpoints) .

[0379]

[0380]

[0381]

[0382] sequence, from the output from the DPB

[0383] Each VCL NAL includes a .

[0384]

[0385] Each VCL NAL includes a .

[0386]

[0387] Each VCL NAL includes the one with nal_unit_type equal to STSA_NUT .

[0388]

[0389] Each VCL NAL includes the one with nal_unit_type equal to TRAIL_NUT .

[0390] Note – The post pictures associated with IRAP or GDR pictures are also after the IRAP or GDR pictures in decoding order. Pictures that are after the associated IRAP or GDR pictures in output order and before the associated IRAP or GDR pictures in decoding order are not allowed.

[0391]

[0392] ...

[0394] 7.4.2.2 NAL Unit Header Semantics ...

[0396] nal_unit_type specifies the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0397] NAL units with unspecified semantics within the range UNSPEC_28..UNSPEC_31 (inclusive) of nal_unit_type shall not affect the decoding process specified in this specification.

[0398] Note 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding process for these values of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, care must be taken in practice when designing an encoder that produces NAL units with these nal_unit_type values, and when designing a decoder that interprets the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values may only be applicable in contexts where "conflicts" (i.e., different definitions of the meaning of the content of NAL units with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in a control application or transport specification, or by the context of the control bitstream distribution).

[0399] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Annex C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units that use reserved values of nal_unit_type.

[0400] Note 3 – This requirement allows for the future definition of compatible extensions to this specification.

[0401] Table 5 – NAL unit type codes and NAL unit type classifications

[0402]

[0403]

[0404]

[0405] Note 4 – A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream.

[0406] Note 5 – An instantaneous decoding refresh (IDR) picture with a nal_unit_type equal to IDR_N_LP has no associated preceding pictures in the bitstream. An IDR picture with a nal_unit_type equal to IDR_W_RADL has no associated RASL pictures present in the bitstream, but may have associated RADL pictures in the bitstream.

[0407] The nal_unit_type values of all VCL NAL units within a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0408]

[0409] For the VCL NAL units of any particular picture, the following applies:

[0410] – If the mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the VCL NAL units of the picture or PU.

[0411] – Otherwise (mixed_nalu_types_in_pic_flag is equal to 1), the picture shall have at least two sub - pictures, and the VCL NAL units of the picture shall have exactly two different nal_unit_type values as follows: the VCL NAL units of at least one sub - picture of the picture shall all have a nal_unit_type with a specific value equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP or CRA_NUT, while the VCL NAL units of the other sub - pictures in the picture shall all have a different specific value equal to TRAIL_NUT, RADL_NUT, RASL_NUT.

[0412] One requirement for bitstream conformance is to apply the following constraints:

[0413] – Post - pictures in output order shall be after the associated IRAP or GDR pictures.

[0414]

[0415] – When a picture is a pre - picture of an IRAP picture, it shall be a RADL or RASL picture.

[0416]

[0417] – There shall be no RASL pictures associated with IDR pictures in the bitstream.

[0418]

[0419] – There shall be no RADL pictures associated with IDR pictures having a nal_unit_type equal to IDR_N_LP in the bitstream.

[0420] Annotation 6 – Random access can be performed at the position of the IRAP PU by discarding all PUs before the IRAP PU (and the IRAP picture and all subsequent non-RASL pictures can be correctly decoded in decoding order), provided that each parameter set is available when referenced (either in the bitstream or by an external means not specified in this specification).

[0421]

[0422] – Any picture (with nuh_layer_id equal to a specific value layerId) before the IRAP picture with nuh_layer_id equal to layerId in decoding order shall be before the IRAP picture and all its associated RADL pictures in output order.

[0423]

[0424] – Any picture (with nuh_layer_id equal to a specific value layerId) before the GDR picture with nuh_layer_id equal to layerId in decoding order shall be before the GDR picture and all its associated pictures in output order.

[0425]

[0426] – Any RASL picture associated with a CRA picture in output order shall be before any RADL picture associated with the CRA picture.

[0427]

[0428] – Any RASL picture associated with a CRA picture in output order shall be after any IRAP picture before the CRA picture in decoding order.

[0429]

[0430] – If field_seq_flag is equal to 0 and the current picture with nuh_layer_id equal to a specific value layerId is a reference picture associated with an IRAP picture, it shall be before all non-reference pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last reference pictures associated with the IRAP picture in decoding order. There shall be at most one non-reference picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be no non-reference pictures with nuh_layer_id equal to layerId between picA and picB in decoding order.

[0431]

[0432] ...

[0434] 7.4.3.4 Picture Parameter Set Semantics ...

[0436] When mixed_nalu_types_in_pic_flag equals 1, it is specified that each picture referring to the PPS has more than one VCL NAL unit and the VCL NAL units do not have the same nal_unit_type value [[, and the picture is not an IRAP picture]]. When mixed_nalu_types_in_pic_flag equals 0, it is specified that each picture referring to the PPS has one or more VCL NAL units and the VCL NAL units of each picture referring to the PPS have the same nal_unit_type value.

[0437] When no_mixed_nalu_types_in_pic_constraint_flag equals 1, the value of mixed_nalu_types_in_pic_flag shall equal 0.

[0438] [[For each slice with a nal_unit_type value nalUnitTypeA in the range of IDR_W_RADL to CRA_NUT (inclusive), in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag of picture picA equals 1), the following applies:

[0439] – The slice shall belong to a sub-picture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] equals 1.

[0440] – The slice shall not belong to a sub-picture of picA that contains VCL NAL units and has a nal_unit_type not equal to nalUnitTypeA.

[0441] – If nalUnitTypeA equals CRA, for all subsequent PUs in the CLVS after the current picture in decoding order and output order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.

[0442] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS that are after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture in the active list that is before picA in decoding order.]]

[0443] Note 1 – When mixed_nalu_types_in_pic_flag is equal to 1, it indicates that the picture that references the PPS contains slices with different NAL unit types, e.g., a coded picture resulting from a sub-picture bitstream merge operation. In this case, the encoder must ensure that the bitstream structure is matched and further alignment of the original bitstream parameters is performed. An example of such alignment is as follows: when the value of sps_idr_rpl_present_flag is equal to 0 and mixed_nalu_types_in_pic_flag is equal to 1, the picture that references the PPS shall not have slices with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0445] 7.4.3.7 Picture Header Structure Semantics ...

[0447] recovery_poc_cnt specifies the recovery point for pictures decoded in output order.

[0448]

[0449] If the current picture is a GDR picture [[associated with the PH]], and there is a picture picA in the CLVS that is after the current GDR picture in decoding order and has a PicOrderCntVal equal to [[the value of PicOrderCntVal of the current GDR picture plus recovery_poc_cnt]], then the picture picA is called a recovery point picture. Otherwise, the first picture with a PicOrderCntVal greater than [[the value of PicOrderCntVal of the current picture plus recovery_poc_cnt]] is called a recovery point picture. The recovery point picture shall not be before the current GDR picture in decoding order. The value of recovery_poc_cnt shall be in the range of 0 to MaxPicOrderCntLsb - 1, inclusive.

[0450] [[When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0451] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)]

[0452] Note 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to [[RpPicOrderCntVal]] of the associated GDR picture, the current and subsequent decoded pictures in output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when present) before the associated GDR picture in decoding order. ...

[0454] 8.3.2 Decoding process for reference picture list construction ...

[0456] A requirement for bitstream consistency is to apply the following constraints:

[0457] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].

[0458] – The pictures involved in each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB, and the TemporalId shall be less than or equal to the TemporalId of the current picture.

[0459] – The pictures involved in each entry in RefPicList[0] or RefPicList[1] shall not be the current picture, and non_reference_picture_flag shall be equal to 0.

[0460] – The STRP entries in RefPicList[0] or RefPicList[1] for a slice of a picture and the LTRP entries in RefPicList[0] or RefPicList[1] for the same slice or different slices of the same picture shall not involve the same picture.

[0461] – There shall not exist in RefPicList[0] or RefPicList[1] a difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture involved in that entry that is greater than or equal to 224 LTRP entry.

[0462] – Let setOfRefPics be the unique set of pictures involved by all the entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all the entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize - 1 (inclusive), where MaxDpbSize is specified in Clause A.4.2 and setOfRefPics shall be the same for all slices of the picture.

[0463] – When the nal_unit_type of the current slice is equal to STSA_NUT, there shall be no active entry in RefPicList[0] or RefPicList[1] whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0464] – When the current picture is a picture after the STSA picture in decoding order and the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, there shall be no picture before the STSA picture in decoding order whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture and which is included as an active entry in RefPicList[0] or RefPicList[1].

[0465]

[0466]

[0467] – When the current picture with nuh_layer_id equal to a specific value layerId is a CRA picture, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] that is before any previous IRAP picture (if any) in decoding order with nuh_layer_id equal to layerId either in output order or in decoding order.

[0468]

[0469] – When the current picture with nuh_layer_id equal to a specific value layerId is not a RASL picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by an active entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates unavailable reference pictures.

[0470]

[0471] – When the current picture with nuh_layer_id equal to a specific value layerId is not a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a picture before the preceding picture associated with the same CRA picture with NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a preceding picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates unavailable reference pictures.

[0472]

[0473] – When the current picture is associated with an IRAP picture and is after the IRAP picture in output order, there shall be no picture involved by an active entry in RefPicList[0] or RefPicList[1] before the associated IRAP picture in output order or in decoding order.

[0474]

[0475] – When the current picture is associated with an IRAP picture, it shall be after the IRAP picture in output order, and after the preceding pictures associated with the same IRAP picture (if any) both in decoding order and in output order. There shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture either in output order or in decoding order.

[0476]

[0477]

[0478] – When the current picture is a RADL picture, then there shall not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0479] ο RASL pictures

[0480] ο Pictures that are before the associated IRAP picture in decoding order

[0481] ο Pictures that are before the associated IRAP picture in decoding order

[0482]

[0483]

[0484]

[0485] – The pictures referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall be in the same AU as the current picture.

[0486] – The pictures referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall exist in the DPB, and the nuh_layer_id shall be less than the nuh_layer_id of the current picture.

[0487] – Each ILRP entry in RefPicList[0] or RefPicList[1] of the slice shall be an active entry.

[0488] Figure 5FIG. 0 is a block diagram showing an example video processing system 1900 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8- or 10-bit multi-component pixel values), or may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0489] System 1900 may include a codec component 1904 that may implement the various codec or encoding methods described in this document. The codec component 1904 may reduce the average bit rate of the video from the input 1902 to the output of the codec component 1904 to produce a coded representation of the video. Thus, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of the codec component 1904 may be stored or transmitted via a connection represented by component 1906. Component 1908 may use the stored or communicated bitstream (or coded) representation of the video received at input 1902 to generate pixel values or a displayable video to be sent to the display interface 1910. The process of generating a user-visible video from the bitstream representation is sometimes referred to as video decompression. Additionally, although certain video processing operations are referred to as "codec" operations or tools, it should be understood that encoding tools or operations are used at the encoder, and the corresponding decoding tools or operations to reverse the encoding results will be performed by the decoder.

[0490] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Displayport, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in various electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0491] Figure 6is a block diagram of a video processing apparatus 3600. The apparatus 3600 can be used to implement one or more methods described herein. The apparatus 3600 can be implemented in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 3600 can include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The (multiple) processors 3602 can be configured to implement one or more methods described in this document. The (multiple) memories 3604 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuitry.

[0492] Figure 8 is a block diagram showing an example video codec system 100 that can utilize the techniques of the present invention.

[0493] As Figure 8 shown, the video codec system 100 can include a source device 110 and a destination device 120. The source device 110 generates encoded video data that can be referred to as a video encoding device. The destination device 120 can decode the encoded video data generated by the source device 110, which can be referred to as a video decoding device.

[0494] The source device 110 can include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0495] The video source 112 can include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data can include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form a codec representation of the video data. The bitstream can include coded pictures and associated data. A coded picture is a codec representation of a picture. The associated data can include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be directly sent to the destination device 120 via the I / O interface 116 over a network 130a. The encoded video data can also be stored on a storage medium / server 130b for access by the destination device 120.

[0496] The destination device 120 can include an I / O interface 126, a video decoder 124, and a display device 122.

[0497] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the target device 120 or may be external to the target device 120, which is configured to connect to an external display device.

[0498] The video encoder 114 and the video decoder 124 may operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0499] Figure 9 is a block diagram showing an example of a video encoder 200, and the video encoder 200 may be Figure 8 the video encoder 114 in the system 100 shown in

[0500] The video encoder 200 may be configured to perform any or all of the techniques of the present invention. In Figure 9 the example of, the video encoder 200 includes a plurality of functional components. The techniques described in the present invention may be shared among various components of the video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0501] The functional components of the video encoder 200 may include a splitting unit 201, a prediction unit 202 including a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, an intra prediction unit 206, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0502] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in the IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0503] In addition, some components (e.g., the motion estimation unit 204 and the motion compensation unit 205) may be highly integrated but are shown separately in Figure 9 the example of for explanatory purposes.

[0504] The splitting unit 201 can split an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0505] The mode selection unit 203 can select (e.g., based on the error result) one of the encoding / decoding modes (intra or inter), and provide the resulting intra or inter encoded / decoded blocks to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the encoded blocks for use as reference pictures. In some examples, the mode selection unit 203 can select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 can also select the resolution of the motion vector for the block (e.g., sub-pixel or integer pixel precision).

[0506] To perform inter prediction on the current video block, the motion estimation unit 204 can generate the motion information of the current video block by comparing one or more reference frames from the buffer 213 with the current video block. The motion compensation unit 205 can determine the predicted video block of the current video block based on the motion information of pictures other than the picture associated with the current video block from the buffer 213 and the decoded samples.

[0507] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, e.g., depending on whether the current video block is in an I-slice, a P-slice, or a B-slice.

[0508] In some examples, the motion estimation unit 204 can perform uni-directional prediction on the current video block, and the motion estimation unit 204 can search for the reference video block of the current video block in the reference pictures of list 0 or list 1. The motion estimation unit 204 can then generate a reference index indicating the reference picture in list 0 or list 1 that contains the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 can output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 can generate the predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0509] In other examples, the motion estimation unit 204 may perform bidirectional prediction on a current video block. The motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate a reference index and a motion vector. The reference index indicates the reference pictures in list 0 and list 1 that contain the reference video block, and the motion vector indicates the spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0510] In some examples, the motion estimation unit 204 may output a set of complete motion information for the decoding process of the decoder.

[0511] In some examples, the motion estimation unit 204 may not output a set of complete motion information of the current video. Instead, the motion estimation unit 204 may signal the motion information of the current video block by referring to the motion information of another video block. For example, the motion estimation unit 204 may determine that the motion information of the current video block is similar enough to the motion information of an adjacent video block.

[0512] In one example, the motion estimation unit 204 may indicate a value in the syntax structure associated with the current video block, and this value indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0513] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0514] As described above, the video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include advanced motion vector predication (AMVP) and merge mode signaling.

[0515] The intra prediction unit 206 may perform intra prediction on a current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0516] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0517] In other examples, the current video block may not have residual data for the current video block, such as in a skip mode, and the residual generation unit 207 may not perform the subtraction operation.

[0518] The transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0519] After the transform processing unit 208 generates the transform coefficient video block associated with the current video block, the quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0520] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0521] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce the block effect in the video block.

[0522] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0523] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or mode. In an example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of video blocks, but may not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on a decision or determination, the conversion from the video blocks to the bitstream (or bitstream representation) of the video will use the video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion from the bitstream of the video to the blocks of the video will be performed using the video processing tool or mode enabled based on the decision or determination.

[0524] Figure 10 is a block diagram showing an example of a video decoder 300, which may be Figure 8 the video decoder 114 in the system 100 shown.

[0525] The video decoder 300 may be configured to perform any or all of the techniques of the present invention. In Figure 10 an example, the video decoder 300 includes a plurality of functional components. The techniques described in this disclosure may be shared among the various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0526] In Figure 10 an example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 may perform a decoding pipeline generally opposite to the encoding pipeline described for the video encoder 200 ( Figure 9 ).

[0527] The entropy decoding unit 301 may retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 may decode the entropy-encoded video data, and from the entropy-decoded video data, the motion compensation unit 302 may determine motion information, including motion vectors, motion vector precision, reference picture list indices, and other motion information. The motion compensation unit 302 may determine such information (e.g.,) by performing AMVP and merge mode.

[0528] The motion compensation unit 302 may produce motion-compensated blocks, possibly performing interpolation based on an interpolation filter. The identifier of the interpolation filter to be used with sub-pixel precision may be included in the syntax elements.

[0529] The motion compensation unit 302 may use an interpolation filter such as that used by the video encoder 200 during the encoding of video blocks to calculate the interpolation of sub-integer pixels of a reference block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 according to the received syntax information, and use the interpolation filter to generate a prediction block.

[0530] The motion compensation unit 302 may use some syntax information to determine the size of the blocks for encoding the (multiple) frames and / or (multiple) slices of an encoded video sequence, the partitioning information describing how each macroblock of a picture of the encoded video sequence is partitioned, the mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame encoded block, and other information for decoding the encoded video sequence.

[0531] The intra prediction unit 303 may form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0532] The reconstruction unit 306 may add a residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in the buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also generates the decoded video for presentation on a display device.

[0533] Next, a list of technical solutions preferred by some embodiments is provided.

[0534] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 1).

[0535] 1. A video processing method (e.g., Figure 7 the method 700 shown in), including: performing (702) a conversion between a video including one or more pictures including one or more sub-pictures and an encoded and decoded representation of the video, where the encoded and decoded representation format rules, the format rules stipulating that one or more pictures including one or more sub-pictures are included in the encoded and decoded representation according to network abstraction layer (NAL) units, and where one type of NAL unit indicated in the encoded and decoded representation includes an encoded slice of a specific type picture or an encoded slice of a specific type sub-picture.

[0536] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 2).

[0537] 2. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a coded - decoded representation of the video, wherein the coded - decoded representation conforms to format rules, and the format rules stipulate that two adjacent sub - pictures having different network abstraction layer unit types will have the same indication of the sub - picture regarded as a picture flag.

[0538] The following technical solutions illustrate example embodiments of the technologies discussed in the previous sections (e.g., items 4, 5, 6, 7, 9, 1, 11, 12).

[0539] 3. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a coded - decoded representation of the video, wherein the coded - decoded representation conforms to format rules, and the format rules define an order of a first type of sub - picture and a second type of sub - picture, wherein the first sub - picture is a post - sub - picture or a pre - sub - picture or a random - access - skipped - pre (RASL) sub - picture type, and the second sub - picture is a RASL type or a random - access - decodable - pre (RADL) type or an instantly - decodable - refresh (IDR) type or a gradually - decodable - refresh (GDR) type sub - picture.

[0540] 4. The method according to technical solution 3, wherein the rule stipulates that the post - sub - picture is after the random - access point or the GDR sub - picture within the associated frame in the output order.

[0541] 5. The method according to technical solution 3, wherein the rule stipulates that when the picture is a pre - picture of an intra - random - access - point picture, the nal_unit_type value of all network abstraction layer units in the picture is equal to RADL_NUT or RASL_NUT.

[0542] 6. The method according to technical solution 3, wherein the rule stipulates that a given sub - picture that is a pre - sub - picture of an IRAP sub - picture must also be a RADL or a RASL sub - picture.

[0543] 7. The method according to technical solution 3, wherein the rule stipulates that a given sub - picture that is a RASL sub - picture is not allowed to be associated with an IDR sub - picture.

[0544] 8. The method according to technical solution 3, wherein the rule stipulates that a given sub - picture having the same layer id and sub - picture index as an IRAP sub - picture must be before the IRAP sub - picture and all its associated RADL sub - pictures in the output order.

[0545] 9. The method according to aspect 3, wherein the rule stipulates that a given sub-picture having the same layer id and sub-picture index as the GDR sub-picture must be before the GDR sub-picture and all its associated RADL sub-pictures in the output order.

[0546] 10. The method according to aspect 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all RADL sub-pictures associated with the CRA sub-picture in the output order.

[0547] 11. The method according to aspect 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all IRAP sub-pictures associated with the CRA sub-picture in the output order.

[0548] 12. The method according to aspect 3, wherein the rule stipulates that if a given sub-picture is a pre-sub-picture of an IRAP sub-picture, then the given sub-picture is before all non-pre-sub-pictures associated with the IRAP picture in the decoding order.

[0549] The following aspects illustrate example embodiments of the techniques discussed in previous sections (e.g., items 8, 14, 15).

[0550] 13. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub-pictures and a codec representation of the video, wherein the codec representation complies with a format rule that defines conditions for allowing or not allowing a first type of sub-picture to co-exist with a second type of sub-picture.

[0551] 14. The method according to aspect 13, wherein the rule stipulates that if there is an IDR sub-picture of network abstraction layer type IDR_N_LP, the codec representation is not allowed to have a RADP sub-picture.

[0552] 15. The method according to aspect 13, wherein the rule does not allow a picture to be included in a reference list of a picture that includes a stepwise temporal sub-layer access (STSA) sub-picture such that the picture is before the picture that includes the STSA sub-picture.

[0553] 16. The method according to aspect 13, wherein the rule does not allow a picture to be included in a reference list of a picture that includes an intra random access point (IRAP) sub-picture such that the picture is before the picture that includes the IRAP sub-picture.

[0554] 17. The method according to any one of aspects 1 to 16, wherein the conversion includes encoding the video into a codec representation.

[0555] 18. According to the method described in any one of Technical Solutions 1 to 16, wherein the conversion includes decoding the codec representation to generate pixel values of the video.

[0556] 19. A video decoding device, comprising a processor configured to implement the method described in one or more of Technical Solutions 1 to 18.

[0557] 20. A video encoding device, comprising a processor configured to implement the method described in one or more of Technical Solutions 1 to 18.

[0558] 21. A computer program product having computer code stored thereon, which when run by a processor causes the processor to implement the method described in any one of Technical Solutions 1 to 18.

[0559] 22. The method, device or system described in this document.

[0560] In the technical solutions described herein, the encoder can conform to the format rules by generating a codec representation according to the format rules. In the technical solutions described herein, the decoder can use the format rules to parse the syntax elements in the codec representation, and understand the presence and absence of the syntax elements according to the format rules to generate the decoded video.

[0561] Figure 11 is a flowchart of an example method 1100 for video processing. Operation 1102 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to format rules that govern the syntax of network abstraction layer (NAL) units in the specified bitstream, and wherein the format rules specify that NAL units of video coding layer (VCL) NAL unit types include content associated with a particular type of picture or a particular type of sub-picture.

[0562] In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with a pure random access picture or a pure random access sub-picture. In some embodiments of method 1100, the pure random access sub-picture is an intra random access point sub-picture of a type where each VCL NAL unit has a pure random access type. In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an associated progressive decoding refresh picture or an associated progressive decoding refresh sub-picture. In some embodiments of method 1100, the associated progressive decoding refresh sub-picture is, in decoding order, a previous progressive decoding refresh sub-picture having an identifier of the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous progressive decoding refresh sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no intra random access point sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index.

[0563] In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an associated intra random access point picture or an associated intra random access point sub-picture. In some embodiments of method 1100, the associated intra random access point sub-picture is, in decoding order, a previous intra random access point sub-picture having an identifier of the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous intra random access point sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no progressive decoding refresh sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index. In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an instant decoding refresh picture or an instant decoding refresh sub-picture. In some embodiments of method 1100, the instant decoding refresh sub-picture is an intra random access point sub-picture where each VCL NAL unit has an instant decoding refresh type.

[0564] In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a preceding picture or a preceding sub-picture. In some embodiments of method 1100, a preceding sub-picture is a sub-picture that is, in output order, before the intra-random access point sub-picture within an associated frame. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a random access decodable preceding picture or a random access decodable preceding sub-picture. In some embodiments of method 1100, a random access decodable preceding sub-picture is a sub-picture in which each VCL NAL unit has a random access decodable preceding type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a random access skipped preceding picture or a random access skipped preceding sub-picture. In some embodiments of method 1100, a random access skipped preceding sub-picture is a sub-picture in which each VCL NAL unit has a random access skipped preceding type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a progressive temporal sub-layer access picture or a progressive temporal sub-layer access sub-picture.

[0565] In some embodiments of method 1100, a progressive temporal sub-layer access sub-picture is a sub-picture in which each VCL NAL unit has a progressive temporal sub-layer access type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a following picture or a following sub-picture. In some embodiments of method 1100, a following sub-picture is a sub-picture in which each VCL NAL unit has a following type.

[0566] Figure 12 is a flow chart of an example method 1200 of video processing. Operation 1202 includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that, in response to a sub-picture being a preceding sub-picture of an intra-random access point sub-picture, the sub-picture is a sub-picture of a random access type.

[0567] In some embodiments of method 1200, the random access type of the sub-picture is a random access decodable preceding sub-picture. In some embodiments of method 1200, the random access type of the sub-picture is a random access skipped preceding sub-picture.

[0568] Figure 13FIG. 1300 is a flow diagram of an example method of video processing. Operation 1302 includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that in response to one or more random access skipped pre-sub-pictures being associated with an instant decoding refresh sub-picture, one or more random access skipped pre-sub-pictures are not present in the bitstream.

[0569] Figure 14 FIG. 1400 is a flow diagram of an example method of video processing. Operation 1402 includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that specify that in response to one or more random access decodable pre-sub-pictures being associated with an instant decoding refresh sub-picture having a type of network abstraction layer (NAL) unit that indicates that the instant decoding refresh sub-picture is not associated with a pre-picture, one or more random access decodable pre-sub-pictures are not present in the bitstream.

[0570] Figure 15 FIG. 1500 is a flow diagram of an example method of video processing. Operation 1502 includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that specify that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value that indicates whether each of the two adjacent sub-pictures in a coded video sequence is considered a picture.

[0571] In some embodiments of method 1500, the format rules specify that the syntax element of the two adjacent sub-pictures indicates that each of the two adjacent sub-pictures in the coded video sequence is considered a picture.

[0572] Figure 16 FIG. 1600 is a flow diagram of an example method of video processing. Operation 1602 includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the format rules specify that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and where the format rules specify that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0573] In some embodiments of method 1600, a picture includes a plurality of sub-pictures including two adjacent sub-pictures, and wherein the formatting rule provides that in response to a sub-picture from the plurality of sub-pictures having a syntax element indicating that the sub-picture is not considered a picture, the plurality of sub-pictures have NAL units of the same type. In some embodiments of method 1600, a picture includes a plurality of sub-pictures including two adjacent sub-pictures, and wherein the formatting rule provides that in response to the plurality of sub-pictures having a corresponding syntax element indicating that each of the plurality of sub-pictures in the CLVS is considered a picture, the syntax element indicates that each picture of the video with a reference picture parameter set (PPS) has a plurality of VCL NAL units that do not have VCL NAL units of the same type.

[0574] Figure 17 is a flowchart of an example method 1700 of video processing. Operation 1702 includes performing a conversion between a video including a picture that includes one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a formatting rule that provides that in response to a syntax element indicating that each picture of the video with a reference picture parameter set (PPS) has a plurality of VCL network abstraction layer (NAL) units that do not have VCL NAL units of the same type, allowing the picture to include more than two different types of VCL NAL units.

[0575] Figure 18 is a flowchart of an example method 1800 of video processing. Operation 1802 includes performing a conversion between a video including one or more pictures that include one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a formatting rule that provides that a post sub-picture associated in sequence with an intra random access point sub-picture or a progressive decode refresh sub-picture is after the intra random access point sub-picture or the progressive decode refresh sub-picture.

[0576] In some embodiments of method 1800, the sequence is an output sequence.

[0577] Figure 19It is a flowchart of an example method 1900 for video processing. Operation 1902 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to: (1) the sub-picture being before an intra-random access point sub-picture in a second order, (2) the sub-picture and the intra-random access point sub-picture having the same first value for a layer to which a network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of a sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0578] In some embodiments of method 1800, the first order is the output order. In some embodiments of method 1800, the second order is the decoding order.

[0579] Figure 20 It is a flowchart of an example method 2000 for video processing. Operation 2002 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that a random access skip pre-sub-picture associated with a pure random access sub-picture is before one or more random access decodable pre-sub-pictures associated with the pure random access sub-picture in an order.

[0580] In some embodiments of method 2000, the order is the output order.

[0581] Figure 21 It is a flowchart of an example method 2100 for video processing. Operation 2102 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that a random access skip pre-sub-picture associated with a pure random access sub-picture is after one or more intra-random access point sub-pictures before the pure random access sub-picture in a second order in a first order.

[0582] In some embodiments of method 2100, the first order is the output order. In some embodiments of method 2100, the second order is the decoding order.

[0583] Figure 22is a flowchart of an example method 2200 for video processing. Operation 2202 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a frame, and (2) the current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in decoding order.

[0584] In some embodiments of method 2200, in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a field, and (2) the current sub-picture not being a pre-sub-picture, the format rule specifies that there is at most one non-pre-sub-picture before a first pre-sub-picture associated with an intra-random access point sub-picture in decoding order, and there is no non-pre-picture between the first pre-sub-picture and the last pre-sub-picture associated with the intra-random access point sub-picture in decoding order, where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same first value for a layer to which network abstraction layer (NAL) units of one or more types of all video coding layer (VCL) NAL units in the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture belong, and where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same second value of a sub-picture index.

[0585] Figure 23 is a flowchart of an example method 2300 for video processing. Operation 2302 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video coding layer (VCL) network abstraction layer (NAL) units in the picture include RADL_NUT or RASL_NUT.

[0586] In some embodiments of method 2300, the format rule specifies that in response to: (1) one or more types of NAL units of all VCL NAL units in a picture include RADL_NUT and RASL_NUT, and (2) a layer including the picture is an output layer, a variable of the picture is set to be equal to a value of a picture output flag.

[0587] Figure 24It is a flowchart of an example method 2400 for video processing. Operation 2402 includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to: (1) at least one sub-picture being before a gradually decoded refresh sub-picture in a second order, (2) the at least one sub-picture and the gradually decoded refresh sub-picture having the same first value for a layer to which NAL units of the at least one sub-picture and the gradually decoded refresh sub-picture belong, and (3) the at least one sub-picture and the gradually decoded refresh picture having the same second value of sub-picture index, the at least one sub-picture is before the gradually decoded refresh sub-picture and one or more sub-pictures associated with the gradually decoded refresh sub-picture in a first order.

[0588] In some methods of embodiment 2400, the first order is the output order. In some methods of embodiment 2400, the second order is the decoding order.

[0589] Figure 25 It is a flowchart of an example method 2500 for video processing. Operation 2502 includes performing a conversion between a video including a current picture including a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to a format rule that in response to: (a) a first picture having the same temporal identifier and the same layer identifier of a network abstraction layer (NAL) unit as the current sub-picture, and (b) the current sub-picture being after a stepwise temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the stepwise temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, the format rule does not allow an active entry in a reference picture list of the current strip to include the first picture that is before a second picture in decoding order, and the second picture includes the stepwise temporal sub-layer access sub-picture.

[0590] In some methods of embodiment 2500, the reference picture list includes a list 0 reference picture list. In some methods of embodiment 2500, the reference picture list includes a list 1 reference picture list.

[0591] Figure 26 It is a flowchart of an example method 2600 for video processing. Operation 2602 includes performing a conversion between a video including a current picture including a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to a format rule that in response to the current sub-picture not being a sub-picture of a specific type, the format rule does not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0592] In some embodiments of method 2600, the current sub - picture is not a random - access skip pre - sub - picture associated with a pure random - access sub - picture of a pure random - access picture, where the pure random - access sub - picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub - picture is not a progressive - decoding refresh sub - picture of a progressive - decoding refresh picture, where the progressive - decoding refresh sub - picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub - picture is not a sub - picture of a recovery picture of a progressive - decoding refresh picture, where the sub - picture of the recovery picture of the progressive - decoding refresh picture has a flag value equal to 1 indicating no output before recovery and has the same layer identifier of the network abstraction layer (NAL) unit as the current sub - picture. In some embodiments of method 2600, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2600, the reference picture list includes the list 1 reference picture list.

[0593] Figure 27 is a flowchart of an example method 2700 for video processing. Operation 2702 includes performing a conversion between a video including a current picture that includes a current sub - picture and a bitstream of the video, where the current sub - picture includes a current slice, and where the bitstream conforms to formatting rules that, in response to the current sub - picture not being a particular type of sub - picture, do not allow an entry in the reference picture list of the current slice to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0594] In some embodiments of method 2700, the current picture is not a pure random - access sub - picture of a pure random - access picture, where the pure random - access sub - picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub - picture is not a sub - picture that is before one or more pre - sub - pictures associated with a pure random - access sub - picture of a pure random - access picture in decoding order, where the one or more pre - sub - pictures have a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub - picture is not a pre - sub - picture associated with a pure random - access sub - picture of a pure random - access picture, where the pre - sub - picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub - picture is not a progressive - decoding refresh sub - picture of a progressive - decoding refresh picture, where the progressive - decoding refresh sub - picture has a flag value equal to 1 indicating no output before recovery.

[0595] In some embodiments of method 2700, the current sub - picture is not a sub - picture of a recovery picture that is gradually decoded and refreshed. The sub - picture has a flag value indicating no output before recovery equal to 1 and has the same layer identifier of the network abstraction layer (NAL) unit as the current sub - picture. In some embodiments of method 2700, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2700, the reference picture list includes the list 1 reference picture list.

[0596] Figure 28 FIG. is a flowchart of an example method 2800 of video processing. Operation 2802 includes performing a conversion between a video including a current picture that contains a current sub - picture and a bit - stream of the video, where the current sub - picture includes a current slice, and where the bit - stream conforms to format rules. In response to (a) a first picture including a previous intra - random - access - point sub - picture, the previous intra - random - access - point sub - picture being before the current sub - picture in a second order, (b) the previous intra - random - access - point sub - picture having the same layer identifier of the network abstraction layer (NAL) unit and the same sub - picture index as the current sub - picture, and (c) the current sub - picture being a pure random - access sub - picture, the format rules do not allow an entry in the reference picture list of the current slice to include the first picture that is before the current picture in the first order or the second order.

[0597] In some embodiments of method 2800, the first order includes the output order. In some embodiments of method 2800, the second order includes the decoding order. In some embodiments of method 2800, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2800, the reference picture list includes the list 1 reference picture list.

[0598] Figure 29 FIG. is a flowchart of an example method 2900 of video processing. Operation 2902 includes performing a conversion between a video including a current picture that contains a current sub - picture and a bit - stream of the video, where the current sub - picture includes a current slice, and where the bit - stream conforms to format rules. In response to (a) the current sub - picture being associated with an intra - random - access - point sub - picture, and (b) the current sub - picture being after the intra - random - access - point sub - picture in a first order, the format rules do not allow an active entry in the reference picture list of the current slice to include the first picture that is before the current picture in the first order or the second order.

[0599] In some embodiments of method 2900, the first order includes the output order. In some embodiments of method 2900, the second order includes the decoding order. In some embodiments of method 2900, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2900, the reference picture list includes the list 1 reference picture list.

[0600] Figure 30 It is a flowchart of an example method 3000 for video processing. Operation 3002 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, where the current sub-picture includes a current slice, and the bitstream conforms to format rules, in response to: (a) the current sub-picture being after an intra-random access point sub-picture in a first order, and (b) the current sub-picture being after one or more pre-sub-pictures associated with an IRAP sub-picture in a first order and a second order, and the format rules do not allow an entry in the reference picture list of the current slice to include a first picture that is before the current picture including the intra-random access point sub-picture associated with the current sub-picture in the first order or the second order.

[0601] In some embodiments of method 3000, the first order includes the output order. In some embodiments of method 3000, the second order includes the decoding order. In some embodiments of method 3000, the reference picture list includes the list 0 reference picture list. In some embodiments of method 3000, the reference picture list includes the list 1 reference picture list.

[0602] Figure 31 It is a flowchart of an example method 3100 for video processing. Operation 3102 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, where the current sub-picture includes a current slice, and the bitstream conforms to format rules, and the format rules specify that in response to the current sub-picture being a random access decodable pre-sub-picture, the reference picture list of the current slice does not include any active entry of any one or more of the following: a first picture including a random access skipped pre-sub-picture and a second picture that is before a third picture including an associated intra-random access point sub-picture in the decoding order.

[0603] In some embodiments of method 3100, the reference picture list includes the list 0 reference picture list. In some embodiments of method 3100, the reference picture list includes the list 1 reference picture list.

[0604] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from the pixel representation of a video to the corresponding bitstream representation, and vice versa. For example, the bitstream representation of a current video block may correspond to bits co-located or propagated at different positions in the bitstream defined by the syntax. For example, a macroblock may be encoded based on the transformed and coded / decoded error residual values and also using bits in the headers and other fields in the bitstream. Additionally, during the conversion, the decoder may parse the bitstream based on this determination, knowing the possible presence or absence of some fields, as described in the above technical solution. Similarly, the encoder may determine to include or not include certain syntax fields and generate the coded / decoded representation accordingly by including or not including the syntax fields from the coded / decoded representation.

[0605] The disclosed and other technical solutions, examples, embodiments, modules, and functional operations described herein may be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances affecting a machine-readable propagated signal, or one or more of them. The term "data processing apparatus" includes all apparatuses, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver apparatus.

[0606] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0607] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuitry, and the apparatus can also be implemented as special-purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0608] For example, processors suitable for executing a computer program include general and special-purpose microprocessors, and any one or more of any type of digital computer. In general, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. In general, a computer will also include one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable hard disks; magneto-optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0609] Although this patent document contains many details, it should not be construed as limiting any subject matter or the scope of any claims, but rather as a description of the features of particular embodiments of a particular technology. Certain features that are described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, the various functions that are described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable sub-combination. In addition, although the above features may be described as acting in certain combinations, and even initially claimed as such, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0610] Similarly, although the operations are described in a particular order in the figures, this should not be understood to mean that such operations must be performed in the particular order or sequence shown to obtain the desired result, or that all of the illustrated operations must be performed. In addition, the separation of various system components in the embodiments of this patent document should not be understood to be required in all embodiments.

[0611] Only some implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

Claims

1. A method for processing video data, comprising: Performing a conversion between a video including one or more pictures each including one or more sub - pictures and a bitstream of the video, wherein the bitstream conforms to format rules, the format rules stipulating that a syntax element of a network abstraction layer (NAL) unit type is included in the bitstream, and wherein the format rules stipulate that a NAL unit is allowed to include content associated with an encoding / decoding strip of a specific type of sub - picture, the NAL unit having a first NAL unit type belonging to a category of NAL unit types including a video coding layer (VCL), wherein the first NAL unit type has a name of clean random access NAL unit type (CRA_NUT), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with a clean random access sub - picture, or wherein the first NAL unit type has a name of instant decoding refresh with random access decodable prefix (IDR_W_RADL) or instant decoding refresh without prefix picture (IDR_N_LP), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with an instant decoding refresh sub - picture, wherein the format rules further stipulate that there is no RADL sub - picture associated with an IDR sub - picture having a type equal to IDR_N_LP in the bitstream, or wherein the first NAL unit type has a name of gradual decoding refresh NAL unit type (GDR_NUT), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with a gradual decoding refresh sub - picture, or, wherein the first NAL unit type has a name of random access decodable prefix NAL unit type (RADL_NUT), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with a random access decodable prefix sub - picture, or wherein the first NAL unit type has a name of step - by - step temporal sub - layer access NAL unit type (STSA_NUT), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with a step - by - step temporal sub - layer access sub - picture, or wherein the first NAL unit type has a name of trailing NAL unit type (TRAIL_NUT), and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding strip is allowed to be associated with a trailing sub - picture.

2. The method according to claim 1, wherein, The clean random access sub - picture is an intra - random access point sub - picture in which each VCL NAL unit has a type equal to CRA_NUT.

3. The method according to claim 1, wherein, The instant decoding refresh sub - picture is an intra - random access point sub - picture in which each VCL NAL unit has a type equal to IDR_W_RADL or IDR_N_LP.

4. The method according to claim 1, wherein, The gradual decoding refresh sub - picture is a sub - picture in which each VCL NAL unit has a type equal to GDR_NUT.

5. The method according to claim 1, wherein, The randomly accessible decodable pre-subpicture is a subpicture in which each VCL NAL unit has a type equal to RADL_NUT.

6. The method according to claim 1, wherein The randomly accessible skipped pre-subpicture is a subpicture in which each VCL NAL unit has a type equal to RASL_NUT.

7. The method according to claim 6, wherein The format rule also stipulates that there is no randomly accessible skipped pre-subpicture associated with the instantaneously decodable refresh subpicture in the bitstream.

8. The method according to claim 1, wherein, The progressive temporal sublayer access subpicture is a subpicture in which each VCL NAL unit has a type equal to STSA_NUT.

9. The method according to claim 1, wherein The post-subpicture is a subpicture in which each VCL NAL unit has a type equal to TRAIL_NUT.

10. The method according to claim 1, wherein, The format rule also stipulates that when the subpicture is a pre-subpicture of an intra random access point IRAP subpicture, the subpicture is a randomly accessible decodable pre RADL or randomly accessible skipped pre RASL subpicture.

11. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

12. The method according to claim 1, wherein The conversion includes decoding the video from the bitstream.

13. The method according to claim 1, further comprising: The format rule also stipulates that NAL units of the video coding layer VCL NAL unit type are allowed to include content associated with a specific type of subpicture.

14. The method according to claim 13, wherein, The content of the NAL unit of the VCL NAL unit type indicates that the coded strip is allowed to be associated with an associated gradually decodable refresh subpicture.

15. The method according to claim 14, Among them, The associated gradually decodable refresh subpicture is, in decoding order, a previous gradually decodable refresh subpicture having an identifier of the layer to which the VCL NAL unit belongs or an identifier of the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of the subpicture index, and wherein, in the decoding order, between the previous gradually decodable refresh subpicture and a specific subpicture having the first specific value of the identifier and the second specific value of the subpicture index, there is no intra random access point subpicture having the first specific value of the identifier and the second specific value of the subpicture index.

16. The method according to claim 13, wherein, The content of the NAL unit of the VCL NAL unit type indicates that the coded strip is allowed to be associated with an associated random access point subpicture.

17. The method according to claim 16, Among them, The associated random access point subpicture is, in decoding order, a previous intra random access point subpicture having an identifier of the layer to which the VCL NAL unit belongs or an identifier of the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of the subpicture index, and wherein, in the decoding order, between the previous intra random access point subpicture and a specific subpicture having the first specific value of the identifier and the second specific value of the subpicture index, there is no gradually decodable refresh subpicture having the first specific value of the identifier and the second specific value of the subpicture index.

18. The method according to claim 13, wherein, The content of the NAL unit of the VCL NAL unit type indicates that the coded strip is allowed to be associated with a pre-subpicture.

19. The method according to claim 18, wherein, The preposed sub-picture is the sub-picture before the random access point sub-picture within the associated frame in the output order.

20. A device for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, The instruction, when executed by the processor, causes the processor to: perform a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies a syntax element including a network abstraction layer NAL unit type in the bitstream, and wherein the format rule specifies that a NAL unit is allowed to include content associated with an encoding / decoding slice of a specific type of sub-picture, and the NAL unit has a first NAL unit type belonging to a NAL unit type category including a video coding layer VCL, wherein the first NAL unit type has a name of clean random access NAL unit type CRA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with a clean random access sub-picture, or wherein the first NAL unit type has a name of instantly decodable refresh with preposed pictures IDR_W_RADL or instantly decodable refresh without preposed pictures IDR_N_LP, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with an instantly decodable refresh sub-picture, and wherein the format rule further specifies that there is no RADL sub-picture associated with an IDR sub-picture having a type equal to IDR_N_LP in the bitstream, or wherein the first NAL unit type has a name of gradually decodable refresh NAL unit type GDR_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with a gradually decodable refresh sub-picture, or, wherein the first NAL unit type has a name of randomly decodable preposed NAL unit type RADL_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with a randomly decodable preposed sub-picture, or wherein the first NAL unit type has a name of stepwise temporal sub-layer access NAL unit type STSA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with a stepwise temporal sub-layer access sub-picture, or wherein the first NAL unit type has a name of trailing NAL unit type TRAIL_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that an encoding / decoding slice is allowed to be associated with a trailing sub-picture.

21. The apparatus according to claim 20, wherein, The instantly decodable refresh sub-picture is an intra random access point sub-picture where each VCL NAL unit has a type equal to IDR_W_RADL or IDR_N_LP.

22. The apparatus according to claim 20, wherein, The random access skip preposed sub-picture is a sub-picture where each VCL NAL unit has a type equal to RASL_NUT.

23. The device according to claim 22, wherein, The format rule also specifies that there is no random access skip prefix sub - picture associated with an instant decoding refresh sub - picture in the bitstream.

24. A non - transitory computer - readable storage medium storing instructions that cause a processor to: perform a conversion between a video including one or more pictures each including one or more sub - pictures and a bitstream of the video, Among them, wherein the bitstream conforms to a format rule that specifies a syntax element of a network abstraction layer NAL unit type in the bitstream, and wherein the format rule specifies that a NAL unit is allowed to include content associated with a coding / decoding strip of a particular type of sub - picture, and the NAL unit has a first NAL unit type belonging to a NAL unit type category including a video coding / decoding layer VCL, wherein the first NAL unit type has a name of clean random access NAL unit type CRA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with a clean random access sub - picture, or wherein the first NAL unit type has a name of instant decoding refresh with random access decodable prefix IDR_W_RADL or instant decoding refresh without prefix picture IDR_N_LP, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with an instant decoding refresh sub - picture, and wherein the format rule also specifies that there is no RADL sub - picture associated with an IDR sub - picture having a type equal to IDR_N_LP in the bitstream, or wherein the first NAL unit type has a name of gradual decoding refresh NAL unit type GDR_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with a gradual decoding refresh sub - picture, or, wherein the first NAL unit type has a name of random access decodable prefix NAL unit type RADL_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with a random access decodable prefix sub - picture, or wherein the first NAL unit type has a name of step - by - step temporal sub - layer access NAL unit type STSA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with a step - by - step temporal sub - layer access sub - picture, or wherein the first NAL unit type has a name of trailing NAL unit type TRAIL_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that a coding / decoding strip is allowed to be associated with a trailing sub - picture.

25. A non-transitory computer-readable recording medium stores a bitstream of a video generated by a method executed by a video processing device, wherein, The method includes: generating the bitstream for the video including one or more pictures each including one or more sub - pictures, wherein the bitstream conforms to a format rule that specifies a syntax element of a network abstraction layer NAL unit type in the bitstream, and Among them, the format rule stipulates that a NAL unit is allowed to include content associated with an encoded / decoded slice of a specific type of sub-picture, and the NAL unit has a first NAL unit type belonging to the category of NAL unit types including the video coding layer VCL. Among them, the first NAL unit type has the name of the clean random access NAL unit type CRA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with a clean random access sub-picture, or Among them, the first NAL unit type has the name of the instantly decodable refresh with random access decodable prefix IDR_W_RADL or the instantly decodable refresh without prefix picture IDR_N_LP. The content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with an instantly decodable refresh sub-picture. Among them, the format rule also stipulates that there is no RADL sub-picture associated with an IDR sub-picture having a type equal to IDR_N_LP in the bitstream, or Among them, the first NAL unit type has the name of the gradual decoding refresh NAL unit type GDR_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with a gradual decoding refresh sub-picture, or Among them, the first NAL unit type has the name of the random access decodable prefix NAL unit type RADL_NUT. The content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with a random access decodable prefix sub-picture, or Among them, the first NAL unit type has the name of the stepwise temporal sub-layer access NAL unit type STSA_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with a stepwise temporal sub-layer access sub-picture, or Among them, the first NAL unit type has the name of the trailing NAL unit type TRAIL_NUT, and the content indication of the NAL unit having the first NAL unit type indicates that the encoded / decoded slice is allowed to be associated with a trailing sub-picture.

26. A method for storing a bitstream of video, including: Generating the bitstream for the video including one or more pictures each containing one or more sub-pictures; And Storing the bitstream in a non-transitory computer-readable recording medium, Among them, the bitstream conforms to a format rule, and the format rule stipulates that a syntax element of the network abstraction layer NAL unit type is included in the bitstream, and Among them, the format rule stipulates that a NAL unit is allowed to include content associated with an encoded / decoded slice of a specific type of sub-picture, and the NAL unit has a first NAL unit type belonging to the category of NAL unit types including the video coding layer VCL. wherein, the first NAL unit type has a name of the clean random access NAL unit type CRA_NUT, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with clean random access sub-pictures, or wherein, the first NAL unit type has a name of the instantly decodable refresh with random access decodable prefix IDR_W_RADL or the instantly decodable refresh without prefix picture IDR_N_LP, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with instantly decodable refresh sub-pictures, wherein, the format rule also stipulates that there is no RADL sub-picture associated with an IDR sub-picture having a type equal to IDR_N_LP in the bitstream, or wherein, the first NAL unit type has a name of the gradual decoding refresh NAL unit type GDR_NUT, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with gradual decoding refresh sub-pictures, or, wherein, the first NAL unit type has a name of the random access decodable prefix NAL unit type RADL_NUT, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with random access decodable prefix sub-pictures, or wherein, the first NAL unit type has a name of the stepwise temporal sub-layer access NAL unit type STSA_NUT, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with stepwise temporal sub-layer access sub-pictures, or wherein, the first NAL unit type has a name of the trailing NAL unit type TRAIL_NUT, and for the content indication of the NAL unit having the first NAL unit type, coded strips are allowed to be associated with trailing sub-pictures.

27. A video decoding apparatus, comprising a processor configured to implement the method according to any one of claims 13 to 19.

28. A video encoding apparatus, comprising a processor configured to implement the method according to any one of claims 13 to 19.

29. A non-transitory computer-readable storage medium storing a bitstream, the bitstream being generated by a video processing apparatus executing the method according to any one of claims 13 to 19.

30. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to implement the method according to any one of claims 13 to 19.

31. A method for generating a bitstream, comprising: generating a bitstream of a video according to the method according to any one of claims 13 to 19, and storing the bitstream on a computer-readable program medium.

Images