The sequential relationship between sub-images

Abstract

Methods and apparatuses for processing video are described. The processing may include video encoding, decoding, or transcoding. An example video processing method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that specify that post-sub-pictures associated in sequence with an intra-random access point sub-picture or a progressive decoding refresh sub-picture are after the intra-random access point sub-picture or the progressive decoding refresh sub-picture.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority and benefit of U.S. Provisional Patent Application No. 62 / 992,724, filed on Mar. 20, 2020, and is based on International Patent Application No. PCT / US2021 / 022982, filed on Mar. 18, 2021. All of the above - mentioned patent applications are hereby incorporated by reference in their entirety. Technical Field

[0003] This patent document relates to image and video encoding, decoding, and transcoding. Background Art

[0004] In the Internet and other digital communication networks, digital video occupies the largest bandwidth. With the increase in the number of connected user devices capable of receiving and displaying video, the bandwidth demand for digital video use is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process the encoded and decoded representations of video using various syntax rules.

[0006] In one example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures that include one or more sub - pictures and a bitstream of the video, where the bitstream conforms to format rules for the syntax of network abstraction layer (NAL) units in the specified bitstream, and where the format rules specify that NAL units of video coding layer (VCL) NAL unit types include content associated with a particular type of picture or a particular type of sub - picture.

[0007] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes sub - pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that a sub - picture is a random - access - type sub - picture in response to the sub - picture being a pre - sub - picture of an intra - random - access - point sub - picture.

[0008] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes sub - pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that in response to one or more random - access - skip pre - sub - pictures being associated with an instantly decoded refresh sub - picture, the one or more random - access - skip pre - sub - pictures do not exist in the bitstream.

[0009] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to one or more random access decodable pre-sub-pictures being associated with an instantly decodable refresh sub-picture having a type of network abstraction layer (NAL) unit indicating that the instantly decodable refresh sub-picture is not associated with a pre-picture, there are no one or more random access decodable pre-sub-pictures in the bitstream.

[0010] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value that indicates whether each of the two adjacent sub-pictures in a coded layer video sequence is considered a picture.

[0011] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, wherein the format rule specifies that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and wherein the format rule specifies that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0012] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture that includes one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to a syntax element indicating that each picture of a video with a reference picture parameter set (PPS) has multiple video coding layer (VCL) network abstraction layer (NAL) units that do not have the same type of VCL NAL unit, a picture is allowed to include more than two different types of VCL NAL units.

[0013] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures that include one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a post-sub-picture associated in sequence with an intra random access point sub-picture or a gradual decoding refresh sub-picture is after the intra random access point sub-picture or the gradual decoding refresh sub-picture.

[0014] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a second-order sub-picture being before an intra-random access point sub-picture, (2) the sub-picture and the intra-random access point sub-picture having the same first value for a layer to which a network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of a sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0015] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is before one or more random access decodable pre-sub-pictures associated with the clean random access sub-picture in an order.

[0016] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is after one or more intra-random access point sub-pictures before the clean random access sub-picture in a second order after the one or more intra-random access point sub-pictures in a first order.

[0017] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a syntax element indicating that a coded layer video sequence conveys a picture representing a frame, and (2) a current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in a decoding order.

[0018] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video coding layer (VCL) network abstraction layer (NAL) units in the picture include RADL_NUT or RASL_NUT.

[0019] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that in response to: (1) at least one sub-picture being before a progressive decoding refresh sub-picture in a second order, (2) the at least one sub-picture and the progressive decoding refresh sub-picture having the same first value for a layer to which network abstraction layer (NAL) units of the at least one sub-picture and the progressive decoding refresh sub-picture belong, and (3) the at least one sub-picture and the progressive decoding refresh picture having the same second value of sub-picture index, the at least one sub-picture is before the progressive decoding refresh sub-picture and one or more sub-pictures associated with the progressive decoding refresh sub-picture in a first order.

[0020] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that in response to: (a) a first picture having the same temporal identifier and the same layer identifier of network abstraction layer (NAL) units as the current sub-picture, and (b) the current sub-picture being after a stepwise temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the stepwise temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, the format rule does not allow an active entry in a reference picture list of the current strip to include the first picture before a second picture in decoding order, the second picture including the stepwise temporal sub-layer access sub-picture.

[0021] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that in response to the current sub-picture not being a specific type of sub-picture, the format rule does not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0022] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture, where the sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules, and in response to the current sub-picture not being a specific type of sub-picture, the format rules do not allow an entry in the reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0023] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules, and in response to (a) a first picture including a previous intra random access point sub-picture, the previous intra random access point sub-picture being before the current sub-picture in a second order, (b) the previous intra random access point sub-picture having the same layer identifier of a network abstraction layer (NAL) unit and the same sub-picture index as the current sub-picture, and (c) the current sub-picture being a pure random access sub-picture, the format rules do not allow an entry in the reference picture list of the current strip to include the first picture before the current picture in a first order or a second order.

[0024] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules, and in response to (a) the current sub-picture being associated with an intra random access point sub-picture, and (b) the current sub-picture being after the intra random access point sub-picture in a first order, the format rules do not allow an active entry in the reference picture list of the current strip to include the first picture before the current picture in a first order or a second order.

[0025] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules, and in response to: (a) the current sub-picture being after an intra random access point sub-picture in a first order, and (b) the current sub-picture being after one or more pre-sub-pictures associated with an IRAP sub-picture in a first order and a second order, the format rules do not allow an entry in the reference picture list of the current strip to include the first picture before the current picture including the intra random access point sub-picture associated with the current sub-picture in a first order or a second order.

[0026] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture and a bitstream of the video, where the current sub-picture includes a current strip, and the bitstream conforms to format rules that specify that in response to the current sub-picture being a random access decodable prefix sub-picture, the reference picture list of the current strip does not include active entries of any one or more of the following: a first picture including a random access skipped prefix sub-picture and a second picture that is before a third picture including an intra random access point sub-picture in decoding order.

[0027] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video. The codec representation conforms to format rules that specify that one or more pictures including one or more sub-pictures are included in the codec representation according to network abstraction layer (NAL) units, where a type of NAL unit indicated in the codec representation includes a coded strip of a specific type of picture or a coded strip of a specific type of sub-picture.

[0028] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that specify that two adjacent sub-pictures having different network abstraction layer unit types will have the same indication of the sub-picture regarded as a picture flag.

[0029] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that define an order of a first type of sub-picture and a second type of sub-picture, where the first sub-picture is a postfix sub-picture or a prefix sub-picture or a random access skipped prefix (RASL) sub-picture type, and the second sub-picture is a RASL type or a random access decodable prefix (RADL) type or an instant decoding refresh (IDR) type or a gradual decoding refresh (GDR) type sub-picture.

[0030] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, where the codec representation conforms to format rules that define conditions for allowing or not allowing a first type of sub-picture to coexist with a second type of sub-picture.

[0031] In yet another exemplary aspect, a video encoder device is disclosed. The video encoder includes a processor configured to implement the above-described method.

[0032] In yet another exemplary aspect, a video decoder device is disclosed. The video decoder includes a processor configured to implement the above-described method.

[0033] In yet another exemplary aspect, a computer-readable medium having code stored thereon is disclosed. The code embodies one of the methods described herein in the form of processor-executable code.

[0034] These and other features will be described throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An example of raster scan strip segmentation of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0036] Figure 2 An example of rectangular strip segmentation of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0037] Figure 3 An example of a picture segmented into slices and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0038] Figure 4 A picture divided into 15 slices, 24 strips, and 24 sub-pictures is shown.

[0039] Figure 5 is a block diagram of an example video processing system.

[0040] Figure 6 is a block diagram of a video processing device.

[0041] Figure 7 is a flowchart of an example method of video processing.

[0042] Figure 8 is a block diagram showing a video coding / decoding system according to some embodiments of the present disclosure.

[0043] Figure 9 is a block diagram showing an encoder according to some embodiments of the present disclosure.

[0044] Figure 10 is a block diagram showing a decoder according to some embodiments of the present disclosure.

[0045] Figures 11 to 31 is a flowchart of an example method of video processing. DETAILED DESCRIPTION

[0046] The use of chapter headings in this document is for ease of understanding and does not limit the applicability of the technologies and embodiments disclosed in each chapter to that chapter. Additionally, the use of H.266 terms in some descriptions is merely for ease of understanding and does not limit the scope of the disclosed technologies. Therefore, the technologies described herein are also applicable to other video codec protocols and designs. In this document, with respect to the current draft of the VVC specification, editorial changes to the text are shown by left and right square brackets (e.g., [[]]), where the deleted text within the brackets represents the superseded text, and bold italic represents the added text.

[0047] 1. Initial Discussion

[0048] This document is related to video coding and decoding technologies. Specifically, it is about the definition of sub-picture types and the relationships in terms of decoding order, output order, and prediction relationships among different types of sub-pictures in single-layer and multi-layer contexts. The key is to clearly specify the meaning of the hybrid sub-picture types within a picture through a set of constraints on decoding order, output order, and prediction relationships. These ideas can be applied alone or in various combinations to any video coding standard or non-standard video codec that supports multi-layer video coding and decoding, such as the Versatile Video Coding (VVC) being developed.

[0049] 2. Abbreviations

[0050] APS (Adaptation Parameter Set) Adaptive Parameter Set

[0051] AU (Access Unit) Access Unit

[0052] AUD (Access Unit Delimiter) Access Unit Delimiter

[0053] AVC (Advanced Video Coding) Advanced Video Coding

[0054] CLVS (Coded Layer Video Sequence) Coded Layer Video Sequence

[0055] CPB (Coded Picture Buffer) Coded Picture Buffer

[0056] CRA (Clean Random Access) Clean Random Access

[0057] CTU (Coding Tree Unit) Coding Tree Unit

[0058] CVS (Coded Video Sequence) Coded Video Sequence

[0059] DCI (Decoding Capability Information) Decoding Capability Information

[0060] DPB (Decoded Picture Buffer) Decoded Picture Buffer

[0061] EOB (End Of Bitstream) End Of Bitstream

[0062] EOS (End Of Sequence) End Of Sequence

[0063] GDR (Gradual Decoding Refresh) Gradual Decoding Refresh

[0064] HEVC (High Efficiency Video Coding) High Efficiency Video Coding

[0065] HRD (Hypothetical Reference Decoder) Hypothetical Reference Decoder

[0066] IDR (Instantaneous Decoding Refresh) Instantaneous Decoding Refresh

[0067] JEM (Joint Exploration Model) Joint Exploration Model

[0068] MCTS (Motion-Constrained Tile Sets) Motion-Constrained Tile Sets

[0069] NAL (Network Abstraction Layer) Network Abstraction Layer

[0070] OLS (Output Layer Set) Output Layer Set

[0071] PH (Picture Header) Picture Header

[0072] PPS (Picture Parameter Set) Picture Parameter Set

[0073] PTL (Profile, Tier and Level) Profile, Tier and Level

[0074] PU (Picture Unit) Picture Unit

[0075] RADL (Random Access Decodable Leading (Picture)) Random Access Decodable Leading (Picture)

[0076] RAP (Random Access Point) Random Access Point

[0077] RASL (Random Access Skipped Leading (Picture)) Random Access Skipped Leading (Picture)

[0078] RBSP (Raw Byte Sequence Payload) Raw Byte Sequence Payload

[0079] RPL (Reference Picture List) Reference Picture List

[0080] SEI (Supplemental Enhancement Information) Supplemental Enhancement Information

[0081] SPS (Sequence Parameter Set) Sequence Parameter Set

[0082] STSA (Step - wise Temporal Sublayer Access) Step - wise Temporal Sublayer Access

[0083] SVC (Scalable Video Coding) Scalable Video Coding

[0084] VCL (Video Coding Layer) Video Coding Layer

[0085] VPS (Video Parameter Set) Video Parameter Set

[0086] VTM (VVC Test Model) VVC Test Model

[0087] VUI (Video Usability Information) Video Usability Information

[0088] VVC (Versatile Video Coding) Versatile Video Coding

[0089] 3. Introduction to Video Coding

[0090] Video coding standards have evolved mainly through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly developed H.262 / MPEG-2 video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure, in which temporal prediction plus transform coding is utilized. To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to a reference software called the Joint Exploration Model (JEM). At the same time, JVET meetings are held quarterly, and the goal of the new coding standard is to reduce the bit rate by 50% compared to HEVC. The new video coding standard was officially named Versatile Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. Due to the continuous efforts in VVC standardization, new coding technologies have been adopted into the VVC standard at each JVET meeting. Then, the working draft of VVC and the test model VTM are updated after each meeting. The VVC project is now striving to be technically completed (FDIS) at the meeting in July 2020.

[0091] 3.1. Picture segmentation schemes in HEVC

[0092] HEVC includes four different picture segmentation schemes, namely regular slices, dependent slices, tiles, and Wavefront Parallel Processing (WPP), which can be applied to Maximum Transfer Unit (MTU) size matching, parallel processing, and reduced end-to-end latency.

[0093] Regular slices are similar to those in H.264 / AVC. Each regular slice is encapsulated in its own NAL unit, and picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are disabled. Therefore, a regular slice can be reconstructed independently of other regular slices within the same picture (although there may still be dependencies due to loop filter operations).

[0094] Regular slices are the only tool that can be used for parallelization and are also available in almost the same form in H.264 / AVC. Parallelization based on regular slices does not require much inter-processor or inter-core communication (except for the inter-processor or inter-core data sharing for motion compensation when decoding predicted coded pictures, which is usually much heavier than inter-processor or inter-core data sharing due to intra-picture prediction). However, for the same reason, the use of regular slices incurs a large amount of coding and decoding overhead due to the bit cost of slice headers and the lack of prediction across slice boundaries. In addition, due to the intra-picture independence of regular slices and each regular slice being encapsulated in its own NAL unit, regular slices (compared to other tools mentioned below) also serve as a key mechanism for bitstream segmentation to match MTU size requirements. In many cases, the goals of parallelization and MTU size matching pose conflicting requirements for the slice layout in the picture. The recognition of this situation led to the development of the parallelization tools mentioned below.

[0095] Dependent slices have short slice headers and allow the bitstream to be segmented at tree block boundaries without breaking any intra-picture prediction. Basically, dependent slices provide the fragmentation of regular slices into multiple NAL units to provide reduced end-to-end latency by allowing a part of a regular slice to be sent out before the coding of the entire regular slice is completed.

[0096] In WPP, a picture is segmented into single-row coded tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other segments. Through the parallel decoding of CTB rows, parallel processing is possible, where the start of the decoding of a CTB row is delayed by two CTBs to ensure that data related to CTBs above and to the right of the main CTB can be obtained before the main CTB being decoded. Using this staggered start (which looks like a wavefront when graphically represented), as many processors / cores can be parallelized as there are CTB rows in the picture. Since intra-picture prediction between adjacent tree block rows within a picture is permitted, the inter-processor / inter-core communication required to enable intra-picture prediction can be substantial. Compared to when not applied, WPP segmentation does not result in the generation of additional NAL units, so WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular slices can be used together with WPP with a certain degree of coding and decoding overhead.

[0097] Slices define the horizontal and vertical boundaries that divide a picture into slice columns and slice rows. Slice columns extend from the top to the bottom of the picture. Similarly, slice rows extend from the left to the right of the picture. The number of slices in a picture can be simply derived by multiplying the number of slice columns by the number of slice rows.

[0098] Before decoding the top-left CTB of the next slice in the raster scan order of the picture's slices, the scan order of CTBs is changed to be local within the slice (in the raster scan order of the slice's CTBs). Similar to conventional strips, slices break the prediction dependencies and entropy decoding dependencies within the picture. However, they do not need to be included in separate NAL units (the same as WPP in this regard); thus, slices cannot be used for MTU size matching. Each slice can be processed by one processor / core, and in the case where a strip spans multiple slices, the inter-processor / inter-core communication required for intra-picture prediction between processing units decoding adjacent slices is limited to transmitting the shared strip header and loop filtering related to the sharing of reconstructed samples and metadata. When a strip contains more than one slice or WPP segment, the byte offset of the entry point of each slice or WPP segment other than the first in the strip is signaled in the strip header.

[0099] For simplicity, HEVC has specified restrictions on the application of four different picture partitioning schemes. A given coded video sequence cannot include both slices and wavefronts for most profiles specified in HEVC. For each strip and slice, one or both of the following conditions must be met: 1) all coded tree blocks in the strip belong to the same slice; 2) all coded tree blocks in the slice belong to the same strip. Finally, a wavefront segment exactly contains one CTB row, and when WPP is being used, if a strip starts within a CTB row, it must end within the same CTB row.

[0100] The latest revision to HEVC is specified in the JCT-VC output file JCTVC-AC1005, J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, Y.-K. Wang (editors), "HEVC Additional Supplementary Enhancement Information (Draft 4)", publicly released on October 24, 2017:

[0101] http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. By including this amendment, HEVC specifies three MCTS-related SEI messages, namely the Temporal MCTS SEI message, the MCTS Extraction Information Set SEI message, and the MCTS Extraction Information Nested SEI message.

[0102] The time-domain MCTS SEI message indicates the presence of MCTS in the bitstream and signals the MCTS. For each MCTS, the motion vectors are restricted to point to full-sample positions within the MCTS and fractional-sample positions of full-sample positions within the MCTS that are only required for interpolation, and the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is not allowed. In this way, each MCTS can be independently decoded in the absence of slices not included in the MCTS.

[0103] The MCTS extraction information set SEI message provides supplementary information (specified as part of the semantics of the SEI message) that can be used in MCTS sub-bitstream extraction to generate a compliant bitstream for the MCTS group. This information consists of a number of extraction information sets, each defining a number of MCTS groups and containing the RBSP bytes that will replace the VPS, SPS, and PPS during the MCTS sub-bitstream extraction process. When extracting the sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all of the syntax elements related to the slice address (including first_slice_segment_in_pic_flag and slice_segment_address) usually need to have different values.

[0104] 3.2. Picture segmentation in VVC

[0105] In VVC, a picture is divided into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs that covers a rectangular area of the picture. The CTUs in a slice are scanned in raster scan order within the slice.

[0106] A strip consists of an integer number of complete slices or an integer number of consecutive complete CTU rows within a slice of the picture.

[0107] Two strip modes are supported, namely the raster scan strip mode and the rectangular strip mode. In the raster scan strip mode, a strip contains a complete sequence of slices in the slice raster scan of the picture. In the rectangular strip mode, a strip contains a number of complete slices that together form a rectangular area of the picture, or a number of consecutive complete CTU rows of a single slice that together form a rectangular area of the picture. The slices within the rectangular strip are scanned in slice raster scan order within the rectangular area corresponding to the strip.

[0108] A sub-picture contains one or more strips that together cover a rectangular area of the picture.

[0109] Figure 1 An example of the raster scan strip segmentation of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0110] Figure 2 Shows an example of rectangular strip segmentation of a picture, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0111] Figure 3 Shows an example of a picture segmented into slices and rectangular strips, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0112] Figure 4 Shows an example of sub - picture segmentation of a picture, where the picture is segmented into 18 slices, 12 slices on the left (each covering a strip with 4x4 CTUs) and 6 slices on the right (each covering 2 vertically stacked strips with 2x2 CTUs), resulting in a total of 24 strips and 24 sub - pictures of different dimensions (each strip is a sub - picture).

[0113] 3.3. Change of Picture Resolution within a Sequence

[0114] In AVC and HEVC, unless a new sequence using a new SPS starts with an IRAP picture, the spatial resolution of a picture cannot be changed. VVC enables changing the picture resolution within a sequence at positions where IRAP pictures are not encoded. IRAP pictures are always intra - coded and decoded. This feature is sometimes referred to as reference picture resampling (RPR) because when the reference picture has a different resolution from the current picture being decoded, this feature requires resampling of the reference pictures used for inter - prediction.

[0115] The scaling ratio is limited to be greater than or equal to 1 / 2 (2 - fold downsampling from the reference picture to the current picture) and less than or equal to 8 (8 - fold upsampling). Three sets of resampling filters with different frequency cut - offs are specified to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8 respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, the same as in the case of motion - compensation interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process with a scaling ratio range from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height and the left, right, up, and down scaling offsets specified for the reference picture and the current picture.

[0116] Other aspects of the VVC design that support features different from HEVC include: i) Picture resolution and the corresponding consistency window are signaled in the PPS rather than in the SPS, while the maximum picture resolution is signaled in the SPS. ii) For a single-layer bitstream, each picture buffer (the storage slot for a decoded picture in the DPB) occupies the buffer size required to store a decoded picture with the maximum picture resolution.

[0117] 3.4. Scalable Video Coding (SVC) in General and in VVC

[0118] Scalable Video Coding (SVC, sometimes also referred to as scalability in video coding) refers to video coding in which a base layer (BL), sometimes referred to as a reference layer (RL), and one or more scalable enhancement layers (ELs) are used. In SVC, the base layer can carry video data with a base-level quality. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as the BL, and the top layer can be used as the EL. Intermediate layers can act as an EL or an RL, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of the layer below it (e.g., the base layer or any intervening enhancement layer) and at the same time an RL for one or more enhancement layers above it. Similarly, in the multi-view or 3D extension of the HEVC standard, there can be multiple views, and the information of one view can be used to code (e.g., encode or decode) the information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancies).

[0119] In SVC, parameters used by the encoder or decoder are grouped into parameter sets based on the coding level (e.g., video level, sequence level, picture level, slice level, etc.) where they can be utilized. For example, parameters that can be utilized by one or more coded video sequences in different layers of the bitstream can be included in the Video Parameter Set (VPS), and parameters that are utilized by one or more pictures in a coded video sequence can be included in the Sequence Parameter Set (SPS). Similarly, parameters that are utilized by one or more slices in a picture can be included in the Picture Parameter Set (PPS), and other parameters specific to a single slice can be included in the slice header. Similarly, an indication of which (which) parameter set a particular layer uses at a given time can be provided at various coding levels.

[0120] Due to the support for reference picture resampling (RPR) in VVC, the support for bitstreams containing multiple layers, e.g., in VVC with two layers of SD and HD resolutions, can be designed to require no additional signaling processing level codec tools because the upsampling required for spatial scalability support can be achieved using only the RPR upsampling filter. However, for scalability support, a high level of syntax changes is required (compared to non - scalability support). Scalability support is specified in VVC version 1. Different from the scalability support in any early video codec standard (including extensions of AVC and HEVC), the VVC scalability is designed to be as friendly as possible to single - layer decoder designs. The decoding capabilities of multi - layer bitstreams are specified in a way as if there were only a single layer in the bitstream. For example, the decoding capabilities such as the DPB size are specified in a way independent of the number of layers in the bitstream to be decoded. Basically, a decoder designed for single - layer bitstreams can decode multi - layer bitstreams with little change. Compared with the multi - layer extension designs of AVC and HEVC, the HLS aspect has been significantly simplified at the cost of some flexibility. For example, an IRAP AU needs to contain pictures of each layer present in the CVS.

[0121] 3.5. Random Access and Its Support in HEVC and VVC

[0122] Random access means accessing and decoding a bitstream starting from a picture that is not the first picture of the bitstream in decoding order. To support tuning and channel switching in broadcast / multicast and multiparty video conferencing, seeking in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points, which are usually intra - coded pictures but can also be inter - coded pictures (e.g., in the case of progressive decoding refresh).

[0123] HEVC includes signaling of Intra Random Access Point (IRAP) pictures in the NAL unit header by means of NAL unit type. Three types of IRAP pictures are supported, namely Instantaneous Decoder Refresh (IDR), Clean Random Access (CRA), and Broken Link Access (BLA) pictures. An IDR picture constrains the inter-picture prediction structure to not reference any pictures prior to the current group-of-pictures (GOP), which is commonly referred to as a closed-GOP random access point. Less restrictive, a CRA picture allows some pictures to reference pictures prior to the current GOP, and in the case of random access, all pictures are discarded. A CRA picture is commonly referred to as an open-GOP random access point. A BLA picture typically results from the concatenation of two bitstreams or parts thereof in a CRA picture, e.g., during stream switching. To enable better system use of IRAP pictures in the system, a total of six different NAL units are defined to signal the attributes of IRAP pictures, which can be used to better match the stream access point types defined in the ISO base media file format (ISOBMFF), which is used for random access support in Dynamic Adaptive Streaming over HTTP (DASH).

[0124] VVC supports three types of IRAP pictures, two types of IDR pictures (one type with or the other type without an associated RADL picture), and one type of CRA picture. These are basically the same as in HEVC. The BLA picture type in HEVC is not included in VVC, mainly for two reasons: i) The basic functionality of a BLA picture can be achieved by a CRA picture plus an end-of-sequence NAL unit, the presence of which indicates that the subsequent pictures start a new CVS in a single-layer bitstream. ii) During the development of VVC, it was desired to specify fewer NAL unit types than in HEVC, e.g., by using five bits instead of six bits for the NAL unit type field in the NAL unit header to indicate.

[0125] Another key difference in random access support between VVC and HEVC is that GDR is supported in a more standardized way in VVC. In GDR, the decoding of the bitstream can start from an inter-coded picture, and although not the entire picture area can be correctly decoded at the beginning, after multiple pictures, the entire picture area will be correct. AVC and HEVC also support GDR, using recovery point SEI messages to signal GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and the recovery point is signaled in the picture header syntax structure. Starting with a GDR picture allows for CVS and the bitstream. This means that it is allowed for the entire bitstream to contain only inter-coded pictures without a single intra-coded picture. The main benefit of specifying GDR support in this way is to provide consistent behavior for GDR. GDR-enabled encoders smooth the bitrate of the bitstream by distributing intra-coded stripes or blocks across multiple pictures instead of intra-coding the entire picture, thus allowing for a significant reduction in end-to-end latency, which is considered more important today as ultra-low latency applications such as wireless displays, online gaming, drone-based applications, etc. become increasingly popular.

[0126] Another GDR-related feature in VVC is virtual boundary signaling. The boundary between the refreshed area (i.e., the correctly decoded area) and the non-refreshed area at the picture between a GDR picture and its recovery point can be signaled as a virtual boundary, and when signaled, the loop filter across the boundary will not be applied, so there will be no decoding mismatch for some samples at or near the boundary. This can be useful when the application determines to display the correctly decoded area during GDR processing.

[0127] IRAP pictures and GDR pictures can be collectively referred to as random access point (RAP) pictures.

[0128] 3.6. Reference Picture Management and Reference Picture List (RPL)

[0129] Reference picture management is a core function essential for any video coding scheme using inter-prediction. It manages the storage and removal of reference pictures in the decoded picture buffer (DPB) and puts the reference pictures into the RPL in their correct order.

[0130] The reference picture management of HEVC, which includes reference picture labeling and removal from the decoded picture buffer (DPB) as well as reference picture list construction (RPLC), is different from that of AVC. Instead of the reference picture labeling mechanism based on a sliding window plus adaptive memory management control operation (MMCO) in AVC, HEVC specifies a reference picture management and labeling mechanism based on the so-called reference picture set (RPS), and thus RPLC is based on the RPS mechanism. The RPS consists of a set of reference pictures associated with a picture, including all reference pictures in decoding order before the associated picture, which can be used for inter prediction of the associated picture or any picture in decoding order after the associated picture. The reference picture set includes five reference picture lists. The first three lists contain all reference pictures that can be used for inter prediction of the current picture and can be used for inter prediction of one or more pictures in decoding order after the current picture. The other two lists consist of all reference pictures that are not used in the inter prediction of the current picture but can be used for inter prediction of one or more pictures in decoding order after the current picture. The RPS provides "intra-coded" signaling notification of the DPB state, rather than "inter-coded" signaling notification as in AVC, mainly to improve fault tolerance. The RPLC process in HEVC is based on the RPS, by signaling an index for each reference index to the RPS subgroup; this process is simpler than the RPLC process in AVC.

[0131] The reference picture management in VVC is more similar to HEVC than to AVC, but is simpler and more robust. As in those standards, two RPLs, list 0 and list 1, are derived, but they are not based on the reference picture set concept used in HEVC or the automatic sliding window process used in AVC; instead, they are signaled more directly. The reference pictures of the RPL are listed as active and inactive entries, and only the active entries can be used as reference indices for inter prediction of the CTUs of the current picture. The inactive entries indicate other pictures to be hosted in the DPB for reference of other pictures arriving later in the bitstream.

[0132] 3.7. Parameter Sets

[0133] AVC, HEVC, and VVC specify parameter sets. The types of parameter sets include SPS, PPS, APS, and VPS. AVC, HEVC, and VVC all support SPS and PPS. VPS was introduced starting from HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC but is included in the latest VVC draft text.

[0134] The SPS is designed to carry sequence-level header information, while the PPS is designed to carry picture-level header information that does not change frequently. By using the SPS and PPS, it is not necessary to repeat the information that does not change frequently for each sequence or picture, thus avoiding redundant signaling of this information. In addition, the use of the SPS and PPS enables out-of-band transmission of important header information, which not only avoids the need for redundant transmission but also improves fault tolerance.

[0135] The VPS is introduced to carry sequence-level header information that is common to all layers in a multi-layer bitstream.

[0136] The APS is introduced to carry such picture-level or slice-level information that requires a significant number of bits to encode and decode, can be shared by multiple pictures, and can have a significant number of different variations in the sequence.

[0137] 3.8. Related Definitions in VVC

[0138] The related definitions in the latest VVC text (JVET-Q2001-vE / v15) are as follows.

[0139] According to previous (when present) has the same value as the nuh_layer_id of a specific picture.

[0140] is of

[0141] each includes the one with a nal_unit_type equal to CRA_NUT

[0142] According to consists of followed by zero or more that are not of includes all subsequent but does not include any subsequent ones that are of

[0143] Each layer in the CVS has and each in is of

[0144] each current in is of

[0145] is of

[0146] Each VCL NAL unit includes one with a nal_unit_type equal to GDR_NUT

[0147] The coded picture is of

[0148] each includes one with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP

[0149] Each layer in the CVS has and each in is of

[0150] is of

[0151] all with the same value of nal_unit_type within the range from IDR_W_RADL to CRA_NUT (inclusive)

[0152] and in the same and in the order output from before

[0153] from output (for that to be output from output ).

[0154] is of

[0155] Each including a nal_unit_type equal to RADL_NUT

[0156] is of

[0157] Each including a nal_unit_type equal to RASL_NUT

[0158] is STSA of

[0159] Each including a nal_unit_type equal to STSA_NUT

[0160] Note – The STSA picture does not use pictures with the same TemporalId as the STSA picture for inter-prediction reference. Pictures with the same TemporalId as the STSA picture that are after the STSA picture in decoding order do not use pictures with the same TemporalId as the STSA picture that are before the STSA picture in decoding order for inter-prediction reference. The STSA picture enables switching from the immediately lower sublayer to the sublayer containing the STSA picture at the STSA picture. The TemporalId of the STSA picture must be greater than 0.

[0161] one or more rectangular regions within.

[0162] According to after and not an IRAP picture that is not an STSA picture

[0163] Note – Post pictures related to an IRAP picture are also after the IRAP picture in decoding order. Pictures that are after the associated IRAP picture in output order and before the associated IRAP picture in decoding order are not allowed.

[0164] 3.9. NAL unit header syntax and semantics in VVC

[0165] In the latest VVC text (in JVET-Q2001-vE / v15), the NAL unit header syntax and semantics are as follows.

[0166] 7.3.1.2 NAL unit header syntax

[0167]

[0168]

[0169] 7.4.2.2 NAL unit header semantics

[0170] shall be equal to 0.

[0171] shall be equal to 0. The value of nuh_reserved_zero_bit being 1 may be specified by ITU-T|ISO / IEC in the future. The decoder shall ignore (i.e., remove and discard from the bitstream) NAL units for which nuh_reserved_zero_bit is equal to 1.

[0172] Specifies the identifier of the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit applies. The value of nuh_layer_id shall be in the range of 0 to 55 (inclusive). Other values of nuh_layer_id are reserved for future use by ITU-T|ISO / IEC.

[0173] The value of nuh_layer_id shall be the same for all VCL NAL units of a coded picture. The value of nuh_layer_id for a coded picture or PU is the value of nuh_layer_id of the VCL NAL units of the coded picture or PU.

[0174] The value of nuh_layer_id for AUD, PH, EOS, and FD NAL units is subject to the following constraints:

[0175] – If nal_unit_type is equal to AUD_NUT, then nuh_layer_id shall be equal to vps_layer_id[0].

[0176] – Otherwise, when nal_unit_type is equal to PH_NUT, EOS_NUT, or FD_NUT, nuh_layer_id shall be equal to the nuh_layer_id of the associated VCL NAL unit.

[0177] Note 1 – The nuh_layer_id values of DCI, VPS, and EOB NAL units are not constrained. The value of nal_unit_type shall be the same for all pictures of a CVSS AU.

[0178] Specify the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0179] NAL units with unspecified semantics in the range UNSPEC_28..UNSPEC_31 (inclusive) shall not affect the decoding process specified in this specification.

[0180] Note 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding process for these values of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, great care must be exercised in the design of encoders that generate NAL units with these nal_unit_type values, and in the design of decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values may only be applicable in contexts where "conflicts" (i.e., different definitions of the meaning of the content of NAL units with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in a control application or transmission specification, or through the context of controlling the distribution of the bitstream).

[0181] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Appendix C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units that use the reserved values of nal_unit_type.

[0182] Note 3 – This requirement allows for future definition of compatible extensions to this specification.

[0183] Table 5 – NAL unit type codes and NAL unit type classifications

[0184]

[0185]

[0186] Note 4 – A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream.

[0187] Annotation 5 – An Instantaneous Decoding Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP has no associated preceding picture in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL has no associated RASL picture present in the bitstream, but may have an associated RADL picture in the bitstream.

[0188] The nal_unit_type value of all VCL NAL units in a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0189] For the VCL NAL units of any particular picture, the following applies:

[0190] – If mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0191] – Otherwise (mixed_nalu_types_in_pic_flag is equal to 1), the picture shall have at least two sub-pictures, and the VCL NAL units of the picture shall have exactly two different nal_unit_type values as follows: The VCL NAL units of at least one sub-picture of the picture shall all have a nal_unit_type with a specific value equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP or CRA_NUT, while the VCL NAL units of the other sub-pictures in the picture shall all have a different specific value equal to TRAIL_NUT, RADL_NUT, RASL_NUT.

[0192] For a single-layer bitstream, the following constraints apply:

[0193] – Except for the first picture in the bitstream in decoding order, each picture is considered to be associated with the previous IRAP picture in decoding order.

[0194] – When a picture is a preceding picture of an IRAP picture, it shall be a RADL or RASL picture.

[0195] – When a picture is a following picture of an IRAP picture, it shall not be a RADL or RASL picture.

[0196] – There shall be no RASL picture associated with an IDR picture in the bitstream.

[0197] – There shall be no RADL pictures associated with an IDR picture with nal_unit_type equal to IDR_N_LP in the bitstream.

[0198] Note 6 – Random access can be performed at the location of an IRAP PU by discarding all PUs before the IRAP PU (and correctly decoding the IRAP picture and all subsequent non-RASL pictures in decoding order), provided that each parameter set is available when referenced (either in the bitstream or by an external means not specified in this specification).

[0199] – Any picture before an IRAP picture in decoding order shall be before the IRAP picture in output order and shall be before any RADL picture associated with the IRAP picture in output order.

[0200] – Any RASL picture associated with a CRA picture in output order shall be before any RADL picture associated with the CRA picture.

[0201] – Any RASL picture associated with a CRA picture in output order shall be after any IRAP picture before the CRA picture in decoding order.

[0202] – If field_seq_flag equals 0 and the current picture is a pre-picture associated with an IRAP picture, then in decoding order it shall be before all non-pre-pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last pre-pictures associated with the IRAP picture in decoding order, respectively. There shall be at most one non-pre-picture before picA in decoding order and no non-pre-pictures between picA and picB in decoding order.

[0203] Minus 1 specifies the temporal identifier of the NAL unit.

[0204] The value of nuh_temporal_id_plus1 shall not be equal to 0.

[0205] The derivation of the variable TemporalId is as follows:

[0206] TemporalId = nuh_temporal_id_plus1 - 1 (36)

[0207] When nal_unit_type is in the range from IDR_W_RADL to RSV_IRAP_12, inclusive, TemporalId shall be equal to 0.

[0208] When nal_unit_type is equal to STSA_NUT and vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, TemporalId shall not be equal to 0.

[0209] For all VCL NAL units of an AU, the value of TemporalId shall be the same. The value of TemporalId for a coded picture, PU, or AU is the value of TemporalId for the VCL NAL units of the coded picture, PU, or AU. The value of TemporalId for a sublayer representation is the maximum value of TemporalId for all VCL NAL units in the sublayer representation.

[0210] The value of TemporalId for non-VCL NAL units is subject to the following constraints:

[0211] – If nal_unit_type is equal to DCI_NUT, VPS_NUT, or SPS_NUT, then TemporalId shall be equal to 0, and the TemporalId of the AU containing the NAL unit shall be equal to 0.

[0212] – Otherwise, if nal_unit_type is equal to PH_NUT, then TemporalId shall be equal to the TemporalId of the PU containing the NAL unit.

[0213] – Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, then TemporalId shall be equal to 0.

[0214] – Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, then TemporalId shall be equal to the TemporalId of the AU containing the NAL unit.

[0215] – Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId shall be greater than or equal to the TemporalId of the PU containing the NAL unit.

[0216] Annotation 7 – When the NAL unit is a non-VCL NAL unit, the value of TemporalId is equal to the minimum of the TemporalId values of all AUs applicable to the non-VCL NAL unit. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId can be greater than or equal to the TemporalId of the containing AU, since all PPSs and APSs can be included at the beginning of the bitstream (e.g., when they are transmitted out-of-band and the receiver places them at the beginning of the bitstream), where the TemporalId of the first decoded picture is equal to 0.

[0217] 3.10. Mixed NAL unit types in picture

[0218] 7.4.3.4 Picture parameter set semantics ...

[0220] Each picture that references a PPS and has a value of 1 for mixed_nalu_types_in_pic_flag has more than one VCL NAL unit, the VCL NAL units do not have the same nal_unit_type value, and the picture is not an IRAP picture. A value of 0 for mixed_nalu_types_in_pic_flag specifies that each picture that references a PPS has one or more VCL NAL units, and the VCL NAL units of each picture that references a PPS have the same nal_unit_type value.

[0221] When no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0222] For each slice with a nal_unit_type value nalUnitTypeA in the range from IDR_W_RADL to CRA_NUT, inclusive, in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag for picture picA is equal to 1), the following applies:

[0223] – The slice shall belong to the subpicture for which the corresponding value of subpic_treated_as_pic_flag[i] is equal to 1

[0224] subpicA.

[0225] – A strip shall not belong to picA sub - picture that contains VCL NAL units and the nal_unit_type is not equal to nalUnitTypeA.

[0226] – If nalUnitTypeA is equal to CRA, for all subsequent PUs in CLVS after the current picture in decoding order and output order, neither RefPicList[0] nor RefPicList[1] of the strips in subpicA of these PUs shall include any picture that is prior to picA in decoding order in the active entries.

[0227] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in CLVS after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the strips in subpicA of these PUs shall include any picture that is prior to picA in decoding order in the active entries.

[0228] Note 1 – The mixed_nalu_types_in_pic_flag being equal to 1 indicates that the picture referring to the PPS contains strips with different NAL unit types, e.g., a coded picture resulting from a sub - picture bit - stream merge operation, where the encoder must ensure that the bit - stream structure matches and further alignment of the original bit - stream parameters. An example of such alignment is as follows: when the value of sps_idr_rpl_present_flag is equal to 0 and the mixed_nalu_types_in_pic_flag is equal to 1, the picture referring to the PPS cannot have strips with nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0230] 3.11. Picture Header Structure Syntax and Semantics in VVC

[0231] In the latest VVC text (in JVET - Q2001 - vE / v15), the picture header structure syntax and semantics most relevant to the technical solution of this article are as follows.

[0232] 7.3.2.7 Picture Header Structure Syntax

[0233]

[0234]

[0235] 7.4.3.7 Picture Header Structure Semantics

[0236] The PH syntax structure contains information common to all slices of the coded picture associated with the PH syntax structure.

[0237] Being equal to 1 specifies that the current picture is a GDR or IRAP picture. gdr_or_irap_pic_flag being equal to 0 specifies that the current picture may or may not be a GDR or IRAP picture.

[0238] Being equal to 1 specifies that the picture associated with PH is a GDR picture. gdr_pic_flag being equal to 0 specifies that the picture associated with PH is not a GDR picture. When absent, the value of gdr_pic_flag is inferred to be equal to 0. When gdr_enabled_flag is equal to 0, the value of gdr_pic_flag shall be equal to 0.

[0239] Note 1 – When gdr_or_irap_pic_flag is equal to 1 and gdr_pic_flag is equal to 0, the picture associated with PH is an IRAP picture. ...

[0241] Specifies the picture order count modulo MaxPicOrderCntLsb of the current picture. The length of the ph_pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4 + 4 bits. The value of ph_pic_order_cnt_lsb shall be in the range of 0 to MaxPicOrderCntLsb-1, inclusive.

[0242] Affects the output of previously decoded pictures in the DPB after decoding a CLVSS picture that is not the first picture in the bitstream specified in Annex C.

[0243] Specify the recovery point for decoding pictures in the output order. If the current picture is a GDR picture associated with PH, and there is a picture picA in the CLVS with PicOrderCntVal that is after the current GDR picture in the decoding order, and PicOrderCntVal is equal to the PicOrderCntVal of the current GDR picture plus the value of recovery_poc_cnt, then the picture picA is called a recovery point picture. Otherwise, the first picture whose PicOrderCntVal in the output order is greater than the PicOrderCntVal of the current picture plus the value of recovery_poc_cnt is called a recovery point picture. The recovery point picture shall not be before the current GDR picture in the decoding order. The value of recovery_poc_cnt shall be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0244] When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0245] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)

[0246] Note 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to the RpPicOrderCntVal of the associated GDR picture, the current and subsequent decoded pictures in the output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when it exists) that is before the associated GDR picture in the decoding order. ...

[0248] 3.12 Constraints on RPL in VVC

[0249] In the latest VVC text (in JVET - Q2001 - vE / v15), the VVC constraints on RPL are as follows (as part of the decoding process of VVC clause 8.3.2 reference picture list construction).

[0250] 8.3.2 Decoding process of reference picture list construction ...

[0252] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called the active entries in RefPicList[i], and the other entries in RefPicList[i] are called the non - active entries in RefPicList[i].

[0253] Annotation 2 – A particular picture may be referred to by an entry in RefPicList[0] and an entry in RefPicList[1] simultaneously. It is also possible that a particular picture is referred to by more than one entry in RefPicList[0] or more than one entry in RefPicList[1].

[0254] Annotation 3 – The active entries in RefPicList[0] and the active entries in RefPicList[1] together refer to all the reference pictures that can be used for inter - prediction of the current picture and one or more pictures following the current picture in decoding order. The non - active entries in RefPicList[0] and the non - active entries in RefPicList[1] together refer to all the reference pictures that are not used for inter - prediction of the current picture but can be used for inter - prediction of one or more pictures following the current picture in decoding order.

[0255] Annotation 4 – There may be one or more entries in RefPicList[0] or RefPicList[1] equal to "no reference picture" because the corresponding picture does not exist in the DPB. Each non - active entry equal to "no reference picture" in RefPicList[0] or RefPicList[0] should be ignored. An unexpected picture loss should be inferred for each active entry equal to "no reference picture" in RefPicList[0] or RefPicList[1].

[0256] One requirement for bit - stream conformance is to apply the following constraints:

[0257] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].

[0258] – The picture referred to by each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB and the TemporalId shall be less than or equal to the TemporalId of the current picture.

[0259] – The picture referred to by each entry in RefPicList[0] or RefPicList[1] shall not be the current picture and the non_reference_picture_flag shall be equal to 0.

[0260] – The STRP entry in RefPicList[0] or RefPicList[1] of the strip of the picture and the LTRP entry in RefPicList[0] or RefPicList[1] of the same strip or different strips of the same picture should not refer to the same picture.

[0261] – There should not be an LTRP entry in RefPicList[0] or RefPicList[1] where the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referred to by this entry is greater than or equal to 2. 24 of the LTRP entry.

[0262] – Let setOfRefPics be the unique set of pictures referred to by all entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics should be less than or equal to MaxDpbSize - 1 (inclusive), where MaxDpbSize is specified in Clause A.4.2 and setOfRefPics should be the same for all strips of the picture.

[0263] – When the nal_unit_type of the current strip is equal to STSA_NUT, there should not be an active entry in RefPicList[0] or RefPicList[1] where the TemporalId is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0264] – When the current picture is a picture after the STSA picture in decoding order and the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, there should not be a picture before the STSA picture in decoding order where the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, and which is included as an active entry in RefPicList[0] or RefPicList[1].

[0265] – When the current picture is a CRA picture, there should not be a picture referred to by an entry in RefPicList[0] or RefPicList[1] in output order or in decoding order that is before any previous IRAP picture (when present) in decoding order.

[0266] – When the current picture is a post picture, there shall be no pictures involved in the active entries in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures associated with the current picture.

[0267] – When the current picture is a post picture that is after one or more pre pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no pictures involved in the entries in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures associated with the current picture.

[0268] – When the current picture is a recovery point picture or a picture after the recovery point picture in output order, there shall be no entries in RefPicList[0] or RefPicList[1] that contain pictures generated by the decoding process of generating unavailable reference pictures of the GDR pictures of the recovery point picture.

[0269] – When the current picture is a post picture, there shall be no pictures involved in the active entries in RefPicList[0] or RefPicList[1] that are before the associated IRAP picture either in output order or in decoding order.

[0270] – When the current picture is a post picture that is after one or more pre pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no pictures involved in the entries in RefPicList[0] or RefPicList[1] that are before the associated IRAP picture either in output order or in decoding order.

[0271] – When the current picture is a RADL picture, there shall not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0272] ο RASL pictures

[0273] ο Pictures generated by the decoding process of generating unavailable reference pictures

[0274] ο Pictures before the associated IRAP picture in decoding order

[0275] – The pictures involved in each ILRP entry in RefPicList[0] or RefPicList[1] of the current picture's slice shall be in the same AU as the current picture.

[0276] – For each ILRP entry in RefPicList[0] or RefPicList[1] of the current picture's strip, the pictures involved should be present in the DPB, and the nuh_layer_id should be less than the nuh_layer_id of the current picture.

[0277] – Each ILRP entry in RefPicList[0] or RefPicList[1] of the strip should be an active entry. ...

[0279] 3.13. Setting of PictureOutputFlag

[0280] In the latest VVC text (in JVET-Q2001-vE / v15), the specification for setting the value of the variable PictureOutputFlag is as follows (as part of the decoding process in Clause 8.1.2 of the coded picture).

[0281] 8.1.2 Decoding Process of Coded Pictures

[0282] The decoding process specified in this clause applies to each coded picture in BitstreamToDecode, referred to as the current picture and represented by the variable CurrPic.

[0283] Depending on the value of chroma_format_idc, the number of sample arrays of the current picture is as follows:

[0284] – If chroma_format_idc is equal to 0, the current picture consists of 1 sample array S L and.

[0285] – Otherwise (chroma_format_idc is not equal to 0), the current picture consists of 3 sample arrays S L , S Cb , S Cr and.

[0286] The decoding process of the current picture takes the syntax elements and uppercase variables from Clause 7 as input. When interpreting the semantics of each NAL unit and each syntax element in the remainder of Clause 8, the term "bitstream" (or a part thereof, such as the CVS of the bitstream) refers to BitstreamToDecode (or a part thereof).

[0287] Depending on the value of separate_colour_plane_flag, the structure of the decoding process is as follows:

[0288] – If separate_colour_plane_flag is equal to 0, call the decoding process once and use the current picture as the output.

[0289] – Otherwise (separate_colour_plane_flag is equal to 1), call the decoding process three times. The input to the decoding process is all the NAL units of the coded picture with the same colour_plane_id value. Specify the decoding process for the NAL units with a specific colour_plane_id value as if there were only CVSs in monochrome colour format with that specific colour_plane_id value in the bitstream. The output of each of the three decoding processes is assigned to one of the three sample arrays of the current picture, and the NAL units with colour_plane_id equal to 0, 1, and 2 are assigned to S L 、S Cb and S Cr .

[0290] Note – When separate_colour_plane_flag is equal to 1 and chroma_format_idc is equal to 3, derive that the variable ChromaArrayType is equal to 0. In the decoding process, the value of this variable is evaluated, resulting in the same operations as for a monochrome picture (when chroma_format_idc is equal to 0).

[0291] For the current picture CurrPic, the decoding process operates as follows:

[0292] 1. Decode the NAL units as specified in Clause 8.2.

[0293] 2. The processing in Clause 8.3 specifies the following decoding processes using the syntax elements in the slice header layer and above:

[0294] – Derive the variables and functions related to picture order count as specified in Clause 8.3.1. It only needs to be called for the first slice of the picture.

[0295] – At the start of the decoding process for each slice of a non-IDR picture, call the decoding process for reference picture list construction specified in Clause 8.3.2 to derive reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]).

[0296] – Call the decoding process for reference picture marking in Clause 8.3.3, where the reference pictures can be marked as "not used for reference" or "used for long-term reference". It only needs to be called for the first slice of the picture.

[0297] – When the current picture is a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, call the decoding process for generating unavailable reference pictures specified in Subclause 8.3.4, which needs to be called only for the first slice of the picture.

[0298] – PictureOutputFlag is set as follows:

[0299] – If one of the following conditions is true, then PictureOutputFlag is set to be equal to 0:

[0300] – The current picture is a RASL picture and the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1.

[0301] – gdr_enabled_flag is equal to 1 and the current picture is a GDR picture with NoOutputBeforeRecoveryFlag equal to 1.

[0302] – gdr_enabled_flag is equal to 1, the current picture is associated with a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current picture is less than the RpPicOrderCntVal of the associated GDR picture.

[0303] – sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions:

[0304] – The PictureOutputFlag of PicA is equal to 1.

[0305] – The nuh_layer_id nuhLid of PicA is greater than the nuh_layer_id of the current picture.

[0306] – PicA belongs to the output layer of OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0307] – The sps_video_parameter_set_id is greater than 0, the ols_mode_idc is equal to 2, and the ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0308] – Otherwise, the PictureOutputFlag is set to be equal to pic_output_flag.

[0309] 3. The processing regulations in Clauses 8.4, 8.5, 8.6, 8.7, and 8.8 use the decoding process of the syntax elements in all syntax structure layers. The requirement for bitstream consistency is that the coded and decoded slices of a picture will contain the slice data of each CTU of the picture, such that the division of the picture into slices and the division of slices into CTUs each form a partitioning of the picture.

[0310] 4. After all slices of the current picture have been decoded, the currently decoded picture is marked as "for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "for short-term reference".

[0311] 4. Technical problems solved by the disclosed technical solutions

[0312] The existing designs in the latest VVC text (JVET-Q2001-vE / v15) have the following problems:

[0313] 1) Since different types of sub-pictures are allowed to be mixed in a picture, it is confusing to refer to the content of NAL units of VCL NAL unit type as the coded and decoded slices of a specific type of picture. For example, an NAL unit with a nal_unit_type equal to CRA_NUT is the coded and decoded slice of a CRA picture only when the nal_unit_type of all slices of the picture is equal to CRA_NUT; when the nal_unit_type of one slice of the picture is not equal to CRA_NUT, then the picture is not a CRA picture.

[0314] 2) Currently, if a sub-picture contains VCL NAL units with nal_unit_type in the range from IDR_W_RADL to CRA_NUT (including the endpoints), the value of subpic_treated_as_pic_flag[] for the sub-picture needs to be equal to 1, and the mixed_nalu_types_in_pic_flag for the picture is equal to 1. In other words, for an IRAP sub-picture mixed with another type of sub-picture in a picture, the value of subpic_treated_as_pic_flag[] needs to be equal to 1. However, with more support for the mixing of VCL NAL unit types, this requirement is insufficient.

[0315] 3) Currently, at most two different types of VCL NAL units (and two different types of sub-pictures) are allowed in a picture.

[0316] 4) In single-layer and multi-layer contexts, there is a lack of constraints on the output order of post sub-pictures relative to associated IRAP or GDR sub-pictures.

[0317] 5) Currently, it is stipulated that when a picture is a pre-picture of an IRAP picture, it should be a RADL or RASL picture. This constraint, together with the definitions of pre- / RADL / RASL pictures, does not allow the mixing of RADL and RASL NAL unit types within a picture generated by the mixing of two CRA pictures and their non-AU-aligned associated RADL and RASL pictures.

[0318] 6) In single-layer and multi-layer contexts, there is a lack of constraints on the sub-picture type of pre-sub-pictures (i.e., the NAL unit type of VCL NAL units in the sub-picture).

[0319] 7) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RASL sub-picture can exist and be associated with an IDR sub-picture.

[0320] 8) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RADL sub-picture can exist and be associated with an IDR sub-picture with nal_unit_type equal to IDR_N_LP.

[0321] 9) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-pictures before an IRAP sub-picture in decoding order and RADL sub-pictures associated with the IRAP sub-picture.

[0322] 10) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-pictures before a GDR sub-picture in decoding order and sub-pictures associated with the GDR sub-picture.

[0323] 11) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with the CRA sub-pictures and the RADL sub-pictures associated with the CRA sub-pictures.

[0324] 12) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with the CRA sub-pictures and the IRAP sub-pictures that are before the CRA sub-pictures in the decoding order.

[0325] 13) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative decoding order between the associated non-preceding pictures and the preceding pictures of the IRAP pictures.

[0326] 14) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are after the STSA sub-pictures in the decoding order.

[0327] 15) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the CRA sub-pictures.

[0328] 16) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that refer to the pictures generated by the decoding process used to generate the unavailable reference pictures.

[0329] 17) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that refer to the pictures generated by the decoding process used to generate the unavailable reference pictures.

[0330] 18) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are associated with the IRAP pictures and are after the IRAP pictures in the output order.

[0331] 19) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that are associated with the IRAP pictures and are after the IRAP pictures in the output order.

[0332] 20) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the RADL sub-pictures.

[0333] 5. Examples of Technical Solutions and Embodiments

[0334] To solve the above problems and other problems, the following summarized methods are disclosed. These items should be considered as examples for explaining general concepts and should not be construed in a narrow manner. In addition, these items can be used individually or combined in any way.

[0335] 1) To solve Problem 1, instead of designating the content of a NAL unit with a VCL NAL unit type as a "coding strip of a specific type of picture", it is designated as a "coding strip of a specific type of picture or sub - picture". For example, the content of a NAL unit with a nal_unit_type equal to CRA_NUT is designated as a "coding strip of a CRA picture or sub - picture".

[0336] a. Additionally, one or more of the following terms are defined: associated GDR sub - picture, associated IRAP sub - picture, CRA sub - picture, GDR sub - picture, IDR sub - picture, IRAP sub - picture, pre - sub - picture, RADL sub - picture, RASL sub - picture, STSA sub - picture, post - sub - picture.

[0337] 2) To solve Problem 2, constraints are added to require that subpic_treated_as_pic_flag[] of any two adjacent sub - pictures of different NAL unit types should be equal to 1.

[0338] a. In one example, the constraint is specified as follows: for any two adjacent sub - pictures in a picture with sub - picture indices i and j, when subpic_treated_as_pic_flag[i] or subpic_treated_as_pic_flag[j] is equal to 0, the two sub - pictures should have the same NAL unit type.

[0339] a. Optionally, it is required that when subpic_treated_as_pic_flag[i] of any sub - picture with sub - picture index i is equal to 0, all sub - pictures in the picture should have the same NAL unit type (i.e., all VCL NAL units in the picture should have the same NAL unit type, i.e., the value of mixed_nalu_types_in_pic_flag should be equal to 0). And this means that when all sub - pictures set their corresponding subpic_treated_as_pic_flag[] equal to 1, mixed_nalu_types_in_pic_flag can only be equal to 1.

[0340] 3) To solve Problem 3, when mixed_nalu_types_in_pic_flag is equal to 1, a picture can be allowed to contain more than two different types of VCL NAL units.

[0341] 4) To solve Problem 4, it is stipulated that the post - sub - picture should be after the associated IRAP or GDR sub - picture in the output order.

[0342] 5) To solve Problem 5, in order to allow mixing of RADL and RASL NAL unit types in pictures resulting from the mixture of two CRA pictures and their non-AU-aligned associated RADL and RASL pictures, the existing constraint that the preceding picture specifying an IRAP picture shall be a RADL or RASL picture is changed as follows: When a picture is a preceding picture of an IRAP picture, the nal_unit_type value of all VCL NAL units in the picture shall be equal to RADL_NUT or RASL_NUT. In addition, in the decoding process of pictures using a mixed nal_unit_type value of RADL_NUT and RASL_NUT, when the layer containing the picture is the output layer, the PictureOutputFlag of the picture is set to be equal to pic_output_flag.

[0343] Thus, although the guarantee of the "correctness" of the RASL sub-pictures of the "intermediate values" in such pictures is also appropriate when the NoOutputBeforeRecoveryFlag associated with the CRA picture is equal to 1, but it is actually not necessary. However, by having all output pictures meet the correct constraints of the decoder, the RADL sub-pictures in such pictures can be guaranteed. The unnecessary part of the guarantee is insignificant and does not increase the complexity of implementing a conforming encoder or decoder. In this case, it would be useful to add a NOTE to clarify that although such RASL sub-pictures associated with a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1 can be output by the decoding process, they are not intended for display and thus are not applied to display.

[0344] 6) To solve Problem 6, it is stipulated that when a sub-picture is a preceding sub-picture of an IRAP sub-picture, it shall be a RADL or RASL sub-picture.

[0345] 7) To solve Problem 7, it is stipulated that there shall be no RASL sub-pictures associated with IDR sub-pictures in the bitstream.

[0346] 8) To solve Problem 8, it is stipulated that there shall be no RADL sub-pictures associated with IDR sub-pictures having a nal_unit_type equal to IDR_N_LP in the bitstream.

[0347] 9) To solve problem 9, it is stipulated that any subpicture with a nuh_layer_id equal to a specific value layerId and a subpicture index equal to a specific value subpicIdx shall, in the output order, be before the IRAP subpicture and all its associated RADL subpictures, and this any subpicture shall, in the decoding order, be before the IRAP subpicture with a nuh_layer_id equal to layerId and a subpicture index equal to subpicIdx.

[0348] 10) To solve problem 10, it is stipulated that any subpicture with a nuh_layer_id equal to a specific value layerId and a subpicture index equal to a specific value subpicIdx shall, in the output order, be before the GDR subpicture and all its associated subpictures, and this any subpicture shall, in the decoding order, be before the GDR subpicture with a nuh_layer_id equal to layerId and a subpicture index equal to subpicIdx.

[0349] 11) To solve problem 11, it is stipulated that any RASL subpicture associated with a CRA subpicture shall, in the output order, be before any RADL subpicture associated with the CRA subpicture.

[0350] 12) To solve problem 12, it is stipulated that any RASL subpicture associated with a CRA subpicture shall, in the output order, be after any IRAP subpicture that is, in the decoding order, before the CRA subpicture.

[0351] 13) To solve problem 13, it is stipulated that if field_seq_flag is equal to 0, and the current subpicture with a nuh_layer_id equal to a specific value layerId and a subpicture index equal to a specific value subpicIdx is a pre-subpicture associated with an IRAP subpicture, then it shall, in the decoding order, be before all non-pre-subpictures associated with the same IRAP subpicture; otherwise, let subpicA and subpicB be the first and last pre-subpictures associated with the IRAP subpicture in the decoding order, respectively. There shall be at most one non-pre-subpicture with a nuh_layer_id equal to layerId and a subpicture index equal to subpicIdx before subpicA in the decoding order, and there shall be no non-pre-picture with a nuh_layer_id equal to layerId and a subpicture index equal to subpicIdx between picA and picB in the decoding order.

[0352] 14) To solve problem 14, it is stipulated that when the current sub-picture with a TemporalId equal to a specific value tId, a nuh_layer_id equal to a specific value layerId, and a sub-picture index equal to a specific value subpicIdx is a sub-picture that comes after the STSA sub-picture with a TemporalId equal to tId, a nuh_layer_id equal to layerId, and a sub-picture index equal to subpicIdx in the decoding order, there should be no picture with a TemporalId equal to tId and a nuh_layer_id equal to layerId that comes before the picture containing the STSA sub-picture in the decoding order and is included as an active entry in RefPicList[0] or RefPicList[1].

[0353] 15) To solve problem 15, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is a CRA sub-picture, there should be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that comes before any picture in the output order or in the decoding order before the previous IRAP sub-picture (when it exists) with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx.

[0354] 16) To solve problem 16, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is not a sub-picture of a RASL sub-picture associated with a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR sub-picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved in the active entries in RefPicList[0] or RefPicList[1] generated by the decoding process that generates unavailable reference pictures.

[0355] 17) To solve problem 17, it is stipulated that when the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is not a CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a sub-picture before the same CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a pre-sub-picture associated with the CRA sub-picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR sub-picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a sub-picture of a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates an unavailable reference picture.

[0356] 18) To solve problem 18, it is stipulated that when the current sub-picture is associated with an IRAP sub-picture and is after the IRAP sub-picture in output order, there should be no picture involved by an active entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub-picture in output order or in decoding order.

[0357] 19) To solve problem 19, it is stipulated that when the current sub-picture is associated with an IRAP sub-picture, after the IRAP sub-picture in output order, and after the pre-sub-picture associated with the same IRAP sub-picture (if any) in both decoding order and output order, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub-picture in output order or in decoding order.

[0358] 20) To solve problem 20, it is stipulated that when the current sub-picture is a RADL sub-picture, there should not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0359] a. A picture containing a RASL sub-picture

[0360] b. A picture before the picture containing the associated IRAP sub-picture in decoding order

[0361] 6. Embodiment

[0362] The following are some example embodiments in terms of the technical solutions summarized in Section 5 above, which can be applied to the VVC specification. The changed text is based on the latest VVC text in JVET-Q2001-vE / v15. Most of the relevant parts added or modified are highlighted in bold italics, and some deleted parts are highlighted in double brackets (e.g., [[ ]]) in a prominent way, and the deleted text is between the double brackets. There are also some other changes that are editorial or not part of this technical solution and thus not highlighted.

[0363] 6.1. First Embodiment

[0364] This embodiment is for Items 1, 1a, 2, 2a, 4, and 6 to 20.

[0365] 3 Definitions ...

[0367] Previous with nuh_layer_id equal to layerId (when present), between this picture and a specific there is no one with nuh_layer_id equal to layerId

[0368]

[0369] Previous with nuh_layer_id equal to layerId Previous (when present), between this picture and a specific there is no one with nuh_layer_id equal to layerId

[0370]

[0371] Each picture with nal_unit_type equal to CRA_NUT .

[0372]

[0373] Each layer in CVS has one and each current in are all of the GDR pictures

[0374] Each VCL NAL unit includes one with a nal_unit_type equal to GDR_NUT

[0375]

[0376] each includes one with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP

[0377]

[0378] all including the same value of nal_unit_type within the range from IDR_W_RADL to CRA_NUT (inclusive)

[0379] before in the

[0380] order, output from the DPB in

[0381] each includes one with a nal_unit_type equal to RADL_NUT

[0382]

[0383] each includes one with a nal_unit_type equal to RASL_NUT

[0384]

[0385] each including those with nal_unit_type equal to STSA_NUT

[0386]

[0387] Each VCL NAL unit includes those with nal_unit_type equal to TRAIL_NUT

[0388] Note – Post pictures associated with IRAP or GDR pictures are also after the IRAP or GDR pictures in decoding order. Pictures that are after the associated IRAP or GDR pictures in output order and before the associated IRAP or GDR pictures in decoding order are not allowed.

[0389]

[0390] ...

[0392] 7.4.2.2 NAL unit header semantics ...

[0394] Specifies the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0395] NAL units with unspecified semantics where nal_unit_type is in the range UNSPEC_28..UNSPEC_31 (inclusive) shall not affect the decoding process specified in this specification.

[0396] Note 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding process for these values of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, care must be taken in practice when designing encoders that produce NAL units with these nal_unit_type values and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values may only be applicable in contexts where "conflicts" (i.e., different definitions of the meaning of the content of NAL units with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in a control application or transport specification, or through the context of controlling bitstream distribution).

[0397] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Annex C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units using the reserved values of nal_unit_type.

[0398] Note 3 – This requirement allows for future definition of compatible extensions to this specification.

[0399] Table 5 – NAL unit type codes and NAL unit type classifications

[0400]

[0401]

[0402]

[0403] Note 4 – A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream.

[0404] Note 5 – An instant decoding refresh (IDR) picture with a nal_unit_type equal to IDR_N_LP has no associated preceding pictures in the bitstream. An IDR picture with a nal_unit_type equal to IDR_W_RADL has no associated RASL pictures present in the bitstream, but may have associated RADL pictures in the bitstream.

[0405] The nal_unit_type values of all VCL NAL units in a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0406]

[0407] For the VCL NAL units of any particular picture, the following applies:

[0408] – If the mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the VCL NAL units of the picture or PU.

[0409] – Otherwise (when mixed_nalu_types_in_pic_flag equals 1), the picture shall have at least two sub - pictures, and the VCL NAL units of the picture shall have exactly two different nal_unit_type values as follows: The VCL NAL units of at least one sub - picture of the picture shall all have a specific value of nal_unit_type equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, or CRA_NUT, while the VCL NAL units of the other sub - pictures in the picture shall all have a different specific value of nal_unit_type equal to TRAIL_NUT, RADL_NUT, RASL_NUT.

[0410] One requirement for bit - stream conformance is to apply the following constraints:

[0411] – The post - picture in output order shall be after the associated IRAP or GDR picture.

[0412] –

[0413] – When the picture is a pre - picture of an IRAP picture, it shall be a RADL or RASL picture.

[0414] –

[0415] – There shall be no RASL picture associated with an IDR picture in the bit - stream.

[0416] –

[0417] – There shall be no RADL picture associated with an IDR picture having a nal_unit_type equal to IDR_N_LP in the bit - stream.

[0418] Note 6 – Random access can be performed at the position of an IRAP PU by discarding all PUs before the IRAP PU (and correctly decoding the IRAP picture and all subsequent non - RASL pictures in decoding order), provided that each parameter set is available when referenced (either in the bit - stream or by an external means not specified in this specification).

[0419] –

[0420] – Any picture (with nuh_layer_id equal to the specific value layerId) that comes before an IRAP picture with nuh_layer_id equal to layerId in decoding order shall come before the IRAP picture and all its associated RADL pictures in output order.

[0421] –

[0422] – Any picture (with nuh_layer_id equal to the specific value layerId) that comes before a GDR picture with nuh_layer_id equal to layerId in decoding order shall come before the GDR picture and all its associated pictures in output order.

[0423] –

[0424] – Any RASL picture associated with a CRA picture in output order shall come before any RADL picture associated with the CRA picture.

[0425] –

[0426] – Any RASL picture associated with a CRA picture in output order shall come after any IRAP picture that comes before the CRA picture in decoding order.

[0427] –

[0428] – If field_seq_flag is equal to 0, and the current picture with nuh_layer_id equal to the specific value layerId is a leading picture associated with an IRAP picture, then it shall come before all non-leading pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last leading pictures associated with the IRAP picture in decoding order. There shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be no non-leading pictures with nuh_layer_id equal to layerId between picA and picB in decoding order.

[0429] – ...

[0431] 7.4.3.4 Picture Parameter Set semantics ...

[0433] Each picture that equals 1 in the specified reference PPS has more than one VCL NAL unit The VCL NAL units do not have the same nal_unit_type value [[, and the picture is not an IRAP picture]]. The mixed_nalu_types_in_pic_flag being equal to 0 specifies that each picture in the reference PPS has one or more VCL NAL units, and that the VCL NAL units of each picture in the reference PPS have the same nal_unit_type value.

[0434] When no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0435] [[For each slice with a nal_unit_type value nalUnitTypeA in the range from IDR_W_RADL to CRA_NUT (inclusive), in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag for picture picA is equal to 1), the following applies:

[0436] – The slice shall belong to a subpicture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] is equal to 1.

[0437] – The slice shall not belong to a subpicture of picA that contains VCL NAL units and has a nal_unit_type different from nalUnitTypeA.

[0438] – If nalUnitTypeA is equal to CRA, for all subsequent PUs in the CLVS after the current picture in decoding order and in output order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active list.

[0439] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS that are after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture in the active entries that is before picA in decoding order.

[0440] Annotation 1 – mixed_nalu_types_in_pic_flag being equal to 1 indicates that the picture referring to the PPS contains slices with different NAL unit types, e.g., coded pictures resulting from sub - picture bitstream merge operations, where the encoder must ensure that the bitstream structure matches and further alignment of the original bitstream parameters. An example of such alignment is as follows: when the value of sps_idr_rpl_present_flag is equal to 0 and mixed_nalu_types_in_pic_flag is equal to 1, the picture referring to the PPS cannot have slices with nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0442] 7.4.3.7 Picture Header Structure Semantics ...

[0444] Specifies the recovery points for decoding pictures in output order.

[0445]

[0446]

[0447] If the current picture is a [[PH - associated]] GDR picture and there is a picture picA in the CLVS that is after the current GDR picture in decoding order and has a PicOrderCntVal equal to [[the value of PicOrderCntVal of the current GDR picture plus recovery_poc_cnt]], then the picture picA is called a recovery - point picture. Otherwise, the first picture in output order that is greater than [[the value of PicOrderCntVal of the current picture plus recovery_poc_cnt]] is called a recovery - point picture. The recovery - point picture shall not be before the current GDR picture in decoding order. The value of recovery_poc_cnt shall be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0448] [[When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0449] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)]

[0450] Annotation 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to [[RpPicOrderCntVal]] of the associated GDR picture, the current and subsequent decoded pictures in output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when present) before the associated GDR picture in decoding order. ...

[0452] 8.3.2 Decoding process for reference picture list construction ...

[0454] One requirement for bitstream conformance is to apply the following constraints:

[0455] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].

[0456] – Each picture involved in an active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB, and the TemporalId shall be less than or equal to the TemporalId of the current picture.

[0457] – Each picture involved in an entry in RefPicList[0] or RefPicList[1] shall not be the current picture, and non_reference_picture_flag shall be equal to 0.

[0458] – The STRP entry in RefPicList[0] or RefPicList[1] of a slice of a picture and the LTRP entry in RefPicList[0] or RefPicList[1] of the same slice or a different slice of the same picture shall not involve the same picture.

[0459] – The difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture involved in the entry in RefPicList[0] or RefPicList[1] should not be greater than or equal to 2. 24 LTRP entry.

[0460] – Let setOfRefPics be the unique set of pictures involved in all entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics should be less than or equal to MaxDpbSize - 1 (inclusive), where MaxDpbSize is specified in Clause A.4.2 and setOfRefPics should be the same for all slices of the pictures.

[0461] – When the nal_unit_type of the current slice is equal to STSA_NUT, there should be no active entry in RefPicList[0] or RefPicList[1] whose TemporalId is equal to the TemporalId of the current picture and nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0462] – When the current picture is a picture after the STSA picture in decoding order and the TemporalId of the STSA picture is equal to the TemporalId of the current picture and nuh_layer_id is equal to the nuh_layer_id of the current picture, there should be no picture before the STSA picture in decoding order whose TemporalId is equal to the TemporalId of the current picture and nuh_layer_id is equal to the nuh_layer_id of the current picture and which is included as an active entry in RefPicList[0] or RefPicList[1].

[0463] –

[0464] – When the current picture with nuh_layer_id equal to a specific value layerId is a CRA picture, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] that, in output order or in decoding order, is before any previous IRAP picture (when present) in decoding order with nuh_layer_id equal to layerId.

[0465] –

[0466] – When the current picture with nuh_layer_id equal to a specific value layerId is not a RASL picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by an active entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates an unavailable reference picture.

[0467] –

[0468] – When the current picture with nuh_layer_id equal to a specific value layerId is not a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a picture before a previous picture associated with the same CRA picture with NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a previous picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates an unavailable reference picture.

[0469] –

[0470] – When the current picture is associated with an IRAP picture and follows that IRAP picture in output order, there shall be no pictures referred to by active entries in RefPicList[0] or RefPicList[1] that precede the associated IRAP picture in output order or in decoding order.

[0471] –

[0472] – When the current picture is associated with an IRAP picture, after the IRAP picture in output order and after the preceding picture associated with the same IRAP picture (if any) in both decoding order and in output order, there shall be no pictures referred to by entries in RefPicList[0] or RefPicList[1] that precede the associated IRAP picture in output order or in decoding order.

[0473] –

[0474] – When the current picture is a RADL picture, it is not in RefPicList[0] or RefPicList[1].

[0475] There should be any one of the following active entries:

[0476] οRASL pictures

[0477] o The picture that precedes the associated IRAP picture in decoding order

[0478] –

[0479] ο

[0480] ο

[0481] – The pictures referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall be in the same AU as the current picture.

[0482] – For each ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture, the picture involved shall exist in the DPB, and the nuh_layer_id shall be less than the nuh_layer_id of the current picture.

[0483] – Each ILRP entry in RefPicList[0] or RefPicList[1] of the slice shall be an active entry.

[0484] Figure 5 FIG. 1900 is a block diagram of an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8- or 10-bit multi-component pixel values), or may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0485] System 1900 may include a codec component 1904 that may implement various codec or encoding methods described in this document. The codec component 1904 may reduce the average bit rate of the video from the input 1902 to the output of the codec component 1904 to produce a coded representation of the video. Thus, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of the codec component 1904 may be stored or transmitted through a connection represented by component 1906. Component 1908 may use the stored or communicated bitstream (or coded) representation of the video received at input 1902 to generate pixel values or a displayable video to be sent to the display interface 1910. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Additionally, although certain video processing operations are referred to as "codec" operations or tools, it should be understood that encoding tools or operations are used at the encoder, and the corresponding decoding tools or operations to reverse the encoding result will be performed by the decoder.

[0486] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in various electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0487] Figure 6 is a block diagram of a video processing apparatus 3600. The apparatus 3600 can be used to implement one or more methods described herein. The apparatus 3600 can be implemented in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 3600 can include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The (multiple) processors 3602 can be configured to implement one or more methods described in this document. The (multiple) memories 3604 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuits.

[0488] Figure 8 is a block diagram showing an example video codec system 100 that can utilize the techniques of the present invention.

[0489] As Figure 8 shown, the video codec system 100 can include a source device 110 and a destination device 120. The source device 110 generates encoded video data that can be referred to as a video encoding device. The destination device 120 can decode the encoded video data generated by the source device 110, which can be referred to as a video decoding device.

[0490] The source device 110 can include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0491] The video source 112 can include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data can include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form a codec representation of the video data. The bitstream can include coded pictures and associated data. The coded pictures are codec representations of pictures. The associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be directly sent to the destination device 120 via the I / O interface 116 over a network 130a. The encoded video data can also be stored on a storage medium / server 130b for access by the destination device 120.

[0492] The destination device 120 can include an I / O interface 126, a video decoder 124, and a display device 122.

[0493] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the target device 120 or may be external to the target device 120, which is configured to connect to an external display device.

[0494] The video encoder 114 and the video decoder 124 may operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0495] Figure 9 is a block diagram showing an example of a video encoder 200, and the video encoder 200 may be Figure 8 the video encoder 114 in the system 100 shown in

[0496] The video encoder 200 may be configured to perform any or all of the techniques of the present invention. In Figure 9 an example, the video encoder 200 includes a plurality of functional components. The techniques described in the present invention may be shared among various components of the video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in the present disclosure.

[0497] The functional components of the video encoder 200 may include a splitting unit 201, a prediction unit 202 including a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0498] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in the IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0499] In addition, some components (e.g., the motion estimation unit 204 and the motion compensation unit 205) may be highly integrated but are shown separately in Figure 9 the example for explanatory purposes.

[0500] The splitting unit 201 can split an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0501] The mode selection unit 203 can select (e.g., based on an error result) one of the encoding / decoding modes (intra or inter), and provide the resulting intra or inter encoded / decoded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the encoded block for use as a reference picture. In some examples, the mode selection unit 203 can select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 can also select the resolution of the motion vector for a block (e.g., sub-pixel or integer-pixel accuracy).

[0502] To perform inter prediction on a current video block, the motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from the buffer 213 with the current video block. The motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information of pictures other than the picture associated with the current video block from the buffer 213 and decoded samples.

[0503] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, e.g., depending on whether the current video block is in an I-slice, a P-slice, or a B-slice.

[0504] In some examples, the motion estimation unit 204 can perform uni-directional prediction on the current video block, and the motion estimation unit 204 can search for a reference video block of the current video block in the reference pictures of list 0 or list 1. The motion estimation unit 204 can then generate a reference index indicating the reference picture in list 0 or list 1 that contains the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 can output the reference index, a prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0505] In other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate a reference index and a motion vector. The reference index indicates the reference pictures in list 0 and list 1 that contain the reference video block, and the motion vector indicates the spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0506] In some examples, the motion estimation unit 204 may output a set of complete motion information for the decoding process of the decoder.

[0507] In some examples, the motion estimation unit 204 may not output a set of complete motion information of the current video. Instead, the motion estimation unit 204 may signal the motion information of the current video block by referring to the motion information of another video block. For example, the motion estimation unit 204 may determine that the motion information of the current video block is similar enough to the motion information of a neighboring video block.

[0508] In one example, the motion estimation unit 204 may indicate a value in the syntax structure associated with the current video block, and this value indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0509] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0510] As described above, the video encoder 200 may predictively signal the motion vector. Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0511] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0512] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0513] In other examples, the current video block may not have residual data for the current video block, such as in the skip mode, and the residual generation unit 207 may not perform the subtraction operation.

[0514] The transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0515] After the transform processing unit 208 generates the transform coefficient video block associated with the current video block, the quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0516] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0517] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce the block effect in the video block.

[0518] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0519] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or mode. In an example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of video blocks, but may not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on a decision or determination, the conversion from video blocks to a bitstream (or bitstream representation) of the video will use the video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion from the bitstream of the video to blocks of the video will be performed using the video processing tool or mode enabled based on the decision or determination.

[0520] Figure 10 is a block diagram showing an example of a video decoder 300, which may be Figure 8 the video decoder 114 in the system 100 shown.

[0521] The video decoder 300 may be configured to perform any or all of the techniques of the present invention. In Figure 10 an example, the video decoder 300 includes a plurality of functional components. The techniques described in this disclosure may be shared among the various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0522] In Figure 10 an example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 may perform a decoding channel that is generally the reverse of the encoding process described for the video encoder 200 ( Figure 9 ).

[0523] The entropy decoding unit 301 may retrieve the encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded video data blocks). The entropy decoding unit 301 may decode the entropy-coded video data, and from the entropy-decoded video data, the motion compensation unit 302 may determine motion information, including motion vectors, motion vector precision, reference picture list indices, and other motion information. The motion compensation unit 302 may determine such information (e.g.) by performing AMVP and merge mode.

[0524] The motion compensation unit 302 may produce motion-compensated blocks, possibly performing interpolation based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in the syntax elements.

[0525] The motion compensation unit 302 may calculate the interpolation of sub-integer pixels of a reference block using an interpolation filter such as that used by the video encoder 200 during the encoding of a video block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 according to the received syntax information, and use the interpolation filter to generate a prediction block.

[0526] The motion compensation unit 302 may use some syntax information to determine the size of the blocks for encoding the (multiple) frames and / or (multiple) slices of an encoded video sequence, the partitioning information describing how each macroblock of a picture of the encoded video sequence is partitioned, the mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame encoded block, and other information for decoding the encoded video sequence.

[0527] The intra prediction unit 303 may form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0528] The reconstruction unit 306 may add a residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video blocks are then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also generates the decoded video for presentation on a display device.

[0529] Next, a list of technical solutions preferred by some embodiments is provided.

[0530] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 1).

[0531] 1. A video processing method (e.g., Figure 7 the method 700 shown in), including: performing (702) a conversion between a video including one or more pictures including one or more sub-pictures and a codec representation of the video, wherein the codec representation format rule stipulates that one or more pictures including one or more sub-pictures are included in the codec representation according to network abstraction layer (NAL) units, and wherein one type of NAL unit indicated in the codec representation includes a codec slice of a specific type picture or a codec slice of a specific type sub-picture.

[0532] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 2).

[0533] 2. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a codec representation of the video, wherein the codec representation conforms to a format rule that specifies that two adjacent sub - pictures having different network abstraction layer unit types will have the same indication of the sub - picture regarded as a picture flag.

[0534] The following technical solutions illustrate example embodiments of the techniques discussed in the previous sections (e.g., items 4, 5, 6, 7, 9, 1, 11, 12).

[0535] 3. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a codec representation of the video, wherein the codec representation conforms to a format rule that defines an order of a first type of sub - picture and a second type of sub - picture, wherein the first sub - picture is a post - sub - picture or a pre - sub - picture or a random access skipped - pre (RASL) sub - picture type, and the second sub - picture is a RASL type or a random access decodable - pre (RADL) type or an instant - decoding refresh (IDR) type or a gradual - decoding refresh (GDR) type sub - picture.

[0536] 4. The method according to technical solution 3, wherein the rule specifies that, in the output order, the post - sub - picture is after a random access point or a GDR sub - picture within an associated frame.

[0537] 5. The method according to technical solution 3, wherein the rule specifies that when the picture is a pre - picture of an intra - frame random access point picture, the nal_unit_type value of all network abstraction layer units in the picture is equal to RADL_NUT or RASL_NUT.

[0538] 6. The method according to technical solution 3, wherein the rule specifies that a given sub - picture that is a pre - sub - picture of an IRAP sub - picture must also be a RADL or a RASL sub - picture.

[0539] 7. The method according to technical solution 3, wherein the rule specifies that a given sub - picture that is a RASL sub - picture is not allowed to be associated with an IDR sub - picture.

[0540] 8. The method according to technical solution 3, wherein the rule specifies that a given sub - picture having the same layer id and sub - picture index as an IRAP sub - picture must be before the IRAP sub - picture and all its associated RADL sub - pictures in the output order.

[0541] 9. The method according to claim 3, wherein the rule stipulates that a given sub-picture having the same layer id and sub-picture index as the GDR sub-picture must be before the GDR sub-picture and all its associated RADL sub-pictures in the output order.

[0542] 10. The method according to claim 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all RADL sub-pictures associated with the CRA sub-picture in the output order.

[0543] 11. The method according to claim 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all IRAP sub-pictures associated with the CRA sub-picture in the output order.

[0544] 12. The method according to claim 3, wherein the rule stipulates that if a given sub-picture is a pre-sub-picture of an IRAP sub-picture, then the given sub-picture is before all non-pre-sub-pictures associated with the IRAP picture in the decoding order.

[0545] The following aspects illustrate example embodiments of the techniques discussed in previous sections (e.g., items 8, 14, 15).

[0546] 13. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub-pictures and a coded representation of the video, wherein the coded representation conforms to format rules that define conditions for allowing or not allowing a first type of sub-picture to co-exist with a second type of sub-picture.

[0547] 14. The method according to claim 13, wherein the rule stipulates that if there is an IDR sub-picture of network abstraction layer type IDR_N_LP, the coded representation is not allowed to have a RADP sub-picture.

[0548] 15. The method according to claim 13, wherein the rule does not allow a picture to be included in a reference list of a picture that includes a sub-picture of stepwise temporal sub-layer access (STSA) such that the picture is before the picture including the STSA sub-picture.

[0549] 16. The method according to claim 13, wherein the rule does not allow a picture to be included in a reference list of a picture that includes an intra random access point (IRAP) sub-picture such that the picture is before the picture including the IRAP sub-picture.

[0550] 17. The method according to any one of claims 1 to 16, wherein the conversion includes encoding the video into a coded representation.

[0551] 18. The method according to any one of technical solutions 1 to 16, wherein the conversion includes decoding an encoded representation to generate pixel values of a video.

[0552] 19. A video decoding apparatus, comprising a processor configured to implement the method described in one or more of technical solutions 1 to 18.

[0553] 20. A video encoding apparatus, comprising a processor configured to implement the method described in one or more of technical solutions 1 to 18.

[0554] 21. A computer program product having computer code stored thereon, the code causing a processor to implement the method according to any one of technical solutions 1 to 18 when run by the processor.

[0555] 22. The method, apparatus or system described in this document.

[0556] In the technical solutions described herein, an encoder may conform to format rules by generating an encoded representation according to the format rules. In the technical solutions described herein, a decoder may use the format rules to parse syntax elements in the encoded representation, and understand the presence and absence of the syntax elements according to the format rules to generate a decoded video.

[0557] Figure 11 is a flowchart of an example method 1100 for video processing. Operation 1102 includes performing a conversion between a video including one or more pictures including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to format rules that govern the syntax of network abstraction layer (NAL) units in the specified bitstream, and wherein the format rules specify that NAL units of a video coding layer (VCL) NAL unit type include content associated with a particular type of picture or a particular type of sub-picture.

[0558] In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a pure random access picture or a pure random access sub-picture. In some embodiments of method 1100, a pure random access sub-picture is an intra random access point sub-picture in which each VCL NAL unit has a pure random access type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with an associated progressive decoding refresh picture or an associated progressive decoding refresh sub-picture. In some embodiments of method 1100, an associated progressive decoding refresh sub-picture is, in decoding order, a previous progressive decoding refresh sub-picture having an identifier of the layer to which the VCL NAL unit belongs or an identifier of the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous progressive decoding refresh sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no intra random access point sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index.

[0559] In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with an associated intra random access point picture or an associated intra random access point sub-picture. In some embodiments of method 1100, an associated intra random access point sub-picture is, in decoding order, a previous intra random access point sub-picture having an identifier of the layer to which the VCL NAL unit belongs or an identifier of the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous intra random access point sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no progressive decoding refresh sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with an instant decoding refresh picture or an instant decoding refresh sub-picture. In some embodiments of method 1100, an instant decoding refresh sub-picture is an intra random access point sub-picture in which each VCL NAL unit has an instant decoding refresh type.

[0560] In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded strip is associated with a previous picture or a previous sub-picture. In some embodiments of method 1100, a previous sub-picture is a sub-picture that is before the intra-random access point sub-picture within the associated frame in output order. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded strip is associated with a random access decodable previous picture or a random access decodable previous sub-picture. In some embodiments of method 1100, a random access decodable previous sub-picture is a sub-picture where each VCL NAL unit has a random access decodable previous type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded strip is associated with a random access skipped previous picture or a random access skipped previous sub-picture. In some embodiments of method 1100, a random access skipped previous sub-picture is a sub-picture where each VCL NAL unit has a random access skipped previous type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded strip is associated with a progressive temporal sub-layer access picture or a progressive temporal sub-layer access sub-picture.

[0561] In some embodiments of method 1100, a progressive temporal sub-layer access sub-picture is a sub-picture where each VCL NAL unit has a progressive temporal sub-layer access type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded strip is associated with a subsequent picture or a subsequent sub-picture. In some embodiments of method 1100, a subsequent sub-picture is a sub-picture where each VCL NAL unit has a subsequent type.

[0562] Figure 12 is a flow chart of an example method 1200 of video processing. Operation 1202 includes performing a conversion between a video including a picture containing a sub-picture and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that, in response to the sub-picture being a previous sub-picture of an intra-random access point sub-picture, the sub-picture is a sub-picture of a random access type.

[0563] In some embodiments of method 1200, the random access type of the sub-picture is a random access decodable previous sub-picture. In some embodiments of method 1200, the random access type of the sub-picture is a random access skipped previous sub-picture.

[0564] Figure 13It is a flowchart of an example method 1300 for video processing. Operation 1302 includes performing a conversion between a video including a picture having sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that in response to one or more random access skip leading sub-pictures being associated with an instant decoding refresh sub-picture, one or more random access skip leading sub-pictures do not exist in the bitstream.

[0565] Figure 14 It is a flowchart of an example method 1400 for video processing. Operation 1402 includes performing a conversion between a video including a picture having sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and the format rules specify that in response to one or more random access decodable leading sub-pictures being associated with an instant decoding refresh sub-picture having a type of network abstraction layer (NAL) unit indicating that the instant decoding refresh sub-picture is not associated with a leading picture, one or more random access decodable leading sub-pictures do not exist in the bitstream.

[0566] Figure 15 It is a flowchart of an example method 1500 for video processing. Operation 1502 includes performing a conversion between a video including a picture having two adjacent sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and the format rules specify that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value, and the syntax element indicates whether each of the two adjacent sub-pictures in a coded video sequence is considered a picture.

[0567] In some embodiments of method 1500, the format rules specify that the syntax element of two adjacent sub-pictures indicates that each of the two adjacent sub-pictures in a coded video sequence is considered a picture.

[0568] Figure 16 It is a flowchart of an example method 1600 for video processing. Operation 1602 includes performing a conversion between a video including a picture having two adjacent sub-pictures and a bitstream of the video, where the format rules specify that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and where the format rules specify that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0569] In some embodiments of method 1600, a picture includes a plurality of sub - pictures including two adjacent sub - pictures, and wherein the formatting rules specify that in response to a sub - picture from the plurality of sub - pictures having a syntax element indicating that the sub - picture is not considered a picture, the plurality of sub - pictures have the same type of NAL unit. In some embodiments of method 1600, a picture includes a plurality of sub - pictures including two adjacent sub - pictures, and wherein the formatting rules specify that in response to the plurality of sub - pictures having a corresponding syntax element indicating that each of the plurality of sub - pictures in the CLVS is considered a picture, the syntax element indicates that each picture of a video with a reference picture parameter set (PPS) has a plurality of VCL NAL units that do not have the same type of video coding layer (VCL) NAL unit.

[0570] Figure 17 is a flowchart of an example method 1700 for video processing. Operation 1702 includes performing a conversion between a video including a picture having one or more sub - pictures and a bitstream of the video, wherein the bitstream conforms to formatting rules that specify that in response to a syntax element indicating that each picture of a video with a reference picture parameter set (PPS) has a plurality of VCL NAL units that do not have the same type of video coding layer (VCL) network abstraction layer (NAL) unit, a picture is allowed to include more than two different types of VCL NAL units.

[0571] Figure 18 is a flowchart of an example method 1800 for video processing. Operation 1802 includes performing a conversion between a video including one or more pictures having one or more sub - pictures and a bitstream of the video, wherein the bitstream conforms to formatting rules that specify that a post - sub - picture associated in sequence with an intra - random access point sub - picture or a progressive decoding refresh sub - picture is after the intra - random access point sub - picture or the progressive decoding refresh sub - picture.

[0572] In some embodiments of method 1800, the sequence is the output sequence.

[0573] Figure 19It is a flowchart of an example method 1900 for video processing. Operation 1902 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to: (1) the second-order sub-picture being before the intra-random access point sub-picture in a frame, (2) the sub-picture and the intra-random access point sub-picture having the same first value for the layer to which the network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0574] In some embodiments of method 1800, the first order is the output order. In some embodiments of method 1800, the second order is the decoding order.

[0575] Figure 20 It is a flowchart of an example method 2000 for video processing. Operation 2002 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that the random access skip pre-sub-pictures associated with a pure random access sub-picture are before one or more random access decodable pre-sub-pictures associated with the pure random access sub-picture in an order.

[0576] In some embodiments of method 2000, the order is the output order.

[0577] Figure 21 It is a flowchart of an example method 2100 for video processing. Operation 2102 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that the random access skip pre-sub-pictures associated with a pure random access sub-picture are after one or more intra-random access point sub-pictures before the pure random access sub-picture in a second order in a first order.

[0578] In some embodiments of method 2100, the first order is the output order. In some embodiments of method 2100, the second order is the decoding order.

[0579] Figure 22It is a flowchart of an example method 2200 for video processing. Operation 2202 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a frame, and (2) the current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in decoding order.

[0580] In some embodiments of method 2200, in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a field, and (2) the current sub-picture not being a pre-sub-picture, the format rules specify that there is at most one non-pre-sub-picture before a first pre-sub-picture associated with an intra-random access point sub-picture in decoding order, and there is no non-pre-picture between the first pre-sub-picture and the last pre-sub-picture associated with the intra-random access point sub-picture in decoding order, where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same first value for the layer to which the network abstraction layer (NAL) units of the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture belong, and where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same second value of sub-picture index.

[0581] Figure 23 It is a flowchart of an example method 2300 for video processing. Operation 2302 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video coding layer (VCL) network abstraction layer (NAL) units in the picture include RADL_NUT or RASL_NUT.

[0582] In some embodiments of method 2300, the format rules specify that, in response to: (1) one or more types of NAL units of all VCL NAL units in a picture include RADL_NUT and RASL_NUT, and (2) the layer including the picture is an output layer, a variable of the picture is set to be equal to the value of a picture output flag.

[0583] Figure 24It is a flowchart of an example method 2400 for video processing. Operation 2402 includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that stipulate that in response to: (1) at least one sub-picture being before a gradually decoded refresh sub-picture in a second order, (2) the at least one sub-picture and the gradually decoded refresh sub-picture having the same first value for a layer to which NAL units of the at least one sub-picture and the gradually decoded refresh sub-picture belong, and (3) the at least one sub-picture and the gradually decoded refresh picture having the same second value of sub-picture index, the at least one sub-picture is before the gradually decoded refresh sub-picture and one or more sub-pictures associated with the gradually decoded refresh sub-picture in a first order.

[0584] In some methods of embodiment 2400, the first order is the output order. In some methods of embodiment 2400, the second order is the decoding order.

[0585] Figure 25 It is a flowchart of an example method 2500 for video processing. Operation 2502 includes performing a conversion between a video including a current picture including a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules in response to: (a) a first picture having the same temporal identifier and the same layer identifier of a network abstraction layer (NAL) unit as the current sub-picture, and (b) the current sub-picture being after a stepwise temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the stepwise temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, and the format rules do not allow an active entry in a reference picture list of the current strip to include the first picture before a second picture in decoding order, and the second picture includes the stepwise temporal sub-layer access sub-picture.

[0586] In some methods of embodiment 2500, the reference picture list includes a list 0 reference picture list. In some methods of embodiment 2500, the reference picture list includes a list 1 reference picture list.

[0587] Figure 26 It is a flowchart of an example method 2600 for video processing. Operation 2602 includes performing a conversion between a video including a current picture including a current sub-picture, where the current sub-picture includes a current strip, and a bitstream of the video, where the bitstream conforms to format rules in response to the current sub-picture not being a sub-picture of a specific type, and the format rules do not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0588] In some embodiments of method 2600, the current sub-picture is not a random access skip pre-sub-picture associated with a pure random access sub-picture of a pure random access picture, where the pure random access sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub-picture is not a progressive decoding refresh sub-picture of a progressive decoding refresh picture, where the progressive decoding refresh sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub-picture is not a sub-picture of a recovery picture of a progressive decoding refresh picture, where the sub-picture of the recovery picture of the progressive decoding refresh picture has a flag value equal to 1 indicating no output before recovery and has the same layer identifier of the network abstraction layer (NAL) unit as the current sub-picture. In some embodiments of method 2600, the reference picture list includes a list 0 reference picture list. In some embodiments of method 2600, the reference picture list includes a list 1 reference picture list.

[0589] Figure 27 is a flowchart of an example method 2700 for video processing. Operation 2702 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, where the current sub-picture includes a current slice, and where the bitstream conforms to format rules that, in response to the current sub-picture not being a particular type of sub-picture, do not allow an entry in the reference picture list of the current slice to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0590] In some embodiments of method 2700, the current picture is not a pure random access sub-picture of a pure random access picture, where the pure random access sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a sub-picture before one or more pre-sub-pictures associated with a pure random access sub-picture of a pure random access picture in decoding order, where the one or more pre-sub-pictures before the sub-picture have a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a pre-sub-picture associated with a pure random access sub-picture of a pure random access picture, where the pre-sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a progressive decoding refresh sub-picture of a progressive decoding refresh picture, where the progressive decoding refresh sub-picture has a flag value equal to 1 indicating no output before recovery.

[0591] In some embodiments of method 2700, the current sub - picture is not a sub - picture of the recovered picture that is gradually decoded and refreshed. The sub - picture has a flag value indicating no output before recovery equal to 1 and has the same layer identifier of the network abstraction layer (NAL) unit as the current sub - picture. In some embodiments of method 2700, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2700, the reference picture list includes the list 1 reference picture list.

[0592] Figure 28 is a flowchart of an example method 2800 for video processing. Operation 2802 includes performing a conversion between a video including a current picture that contains a current sub - picture and a bit - stream of the video, where the current sub - picture includes a current strip. The bit - stream conforms to format rules in response to (a) the first picture including a previous intra - random - access - point sub - picture, the previous intra - random - access - point sub - picture being before the current sub - picture in a second order, (b) the previous intra - random - access - point sub - picture having the same layer identifier of the network abstraction layer (NAL) unit and the same sub - picture index as the current sub - picture, and (c) the current sub - picture being a pure random - access sub - picture. The format rules do not allow an entry in the reference picture list of the current strip to include the first picture that is before the current picture in the first order or the second order.

[0593] In some embodiments of method 2800, the first order includes the output order. In some embodiments of method 2800, the second order includes the decoding order. In some embodiments of method 2800, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2800, the reference picture list includes the list 1 reference picture list.

[0594] Figure 29 is a flowchart of an example method 2900 for video processing. Operation 2902 includes performing a conversion between a video including a current picture that contains a current sub - picture and a bit - stream of the video, where the current sub - picture includes a current strip. The bit - stream conforms to format rules in response to (a) the current sub - picture being associated with an intra - random - access - point sub - picture, and (b) the current sub - picture being after the intra - random - access - point sub - picture in a first order. The format rules do not allow an active entry in the reference picture list of the current strip to include the first picture that is before the current picture in the first order or the second order.

[0595] In some embodiments of method 2900, the first order includes the output order. In some embodiments of method 2900, the second order includes the decoding order. In some embodiments of method 2900, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2900, the reference picture list includes the list 1 reference picture list.

[0596] Figure 30 is a flowchart of an example method 3000 for video processing. Operation 3002 includes performing a conversion between a video including a current picture that includes a current sub-picture, where the current sub-picture includes a current slice, and a bitstream of the video, where the bitstream complies with format rules, in response to: (a) the current sub-picture being after an intra-random access point sub-picture in a first order, (b) the current sub-picture being after one or more pre-sub-pictures associated with an IRAP sub-picture in a first order and a second order, and the format rules do not allow an entry in the reference picture list of the current slice to include a first picture that is before, in the first order or the second order, a current picture including an intra-random access point sub-picture associated with the current sub-picture.

[0597] In some embodiments of method 3000, the first order includes an output order. In some embodiments of method 3000, the second order includes a decoding order. In some embodiments of method 3000, the reference picture list includes a list 0 reference picture list. In some embodiments of method 3000, the reference picture list includes a list 1 reference picture list.

[0598] Figure 31 is a flowchart of an example method 3100 for video processing. Operation 3102 includes performing a conversion between a video including a current picture that includes a current sub-picture, where the current sub-picture includes a current slice, and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to the current sub-picture being a random access decodable pre-sub-picture, the reference picture list of the current slice does not include any active entry of any one or more of: a first picture including a random access skip pre-sub-picture and a second picture that is before, in decoding order, a third picture including an associated intra-random access point sub-picture.

[0599] In some embodiments of method 3100, the reference picture list includes a list 0 reference picture list. In some embodiments of method 3100, the reference picture list includes a list 1 reference picture list.

[0600] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from the pixel representation of a video to the corresponding bitstream representation, and vice versa. For example, the bitstream representation of the current video block may correspond to bits co-located or propagated at different positions in the bitstream defined by the syntax. For example, a macroblock may be encoded based on the transformed and coded / decoded error residual values and also using bits in the headers and other fields in the bitstream. Additionally, during the conversion, the decoder may parse the bitstream based on this determination, knowing the possible presence or absence of some fields, as described in the above technical solution. Similarly, the encoder may determine to include or not include certain syntax fields and generate the coded / decoded representation accordingly by including or not including the syntax fields from the coded / decoded representation.

[0601] The disclosed and other technical solutions, examples, embodiments, modules, and functional operations described herein may be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances affecting a machine-readable propagated signal, or one or more of them. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, e.g., code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver apparatus.

[0602] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0603] The processes and logical flows described in this document can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuitry, and the apparatus can also be implemented as special-purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0604] For example, processors suitable for executing a computer program include general and special purpose microprocessors, and any one or more of any type of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable hard disks; magneto-optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0605] Although this patent document contains many details, it should not be construed as limiting any subject matter or the scope of the claims, but rather as a description of the features of particular embodiments of a particular technology. Certain features described in the context of separate embodiments of this patent document may also be implemented in combination in a single embodiment. Conversely, the various functions described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable sub-combination. Additionally, although the above features may be described as acting in certain combinations, and even initially claimed as such, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0606] Likewise, although the operations are described in a particular order in the figures, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order to achieve the desired result, or that all of the illustrated operations be performed. Additionally, the separation of various system components in the embodiments of this patent document should not be construed as required in all embodiments.

[0607] Only some implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: Performing a conversion between a video including one or more pictures each including one or more sub - pictures and a bitstream of the video, wherein the bitstream conforms to format rules, and the format rules stipulate that, in the following cases, a sub - picture is before an intra - random - access - point sub - picture and one or more random - access - decodable pre - sub - pictures associated with the intra - random - access - point sub - picture in a first order: (1) The sub - picture is before the intra - random - access - point sub - picture in a second order, (2) The sub - picture and the intra - random - access - point sub - picture have the same value for the layer to which the network abstraction layer (NAL) unit of the sub - picture and the intra - random - access - point sub - picture belongs, and (3) The sub - picture and the intra - random - access - point sub - picture have the same sub - picture index value; wherein the format rules also stipulate that, in the following cases, a current sub - picture is before one or more non - pre - sub - pictures associated with an intra - random - access - point sub - picture in a decoding order: (1) A syntax element indicates that the video sequence at the coding and decoding layer conveys a picture representing a frame, and (2) The current sub - picture is a pre - sub - picture associated with the intra - random - access - point sub - picture; wherein when (1) the syntax element indicates that the video sequence at the coding and decoding layer conveys a picture representing a field, and (2) the current sub - picture is not a pre - sub - picture, the format rules also stipulate that: There is at most one non - pre - sub - picture before a first pre - sub - picture associated with the intra - random - access - point sub - picture in the decoding order, and There is no non - pre - picture between the first pre - sub - picture and the last pre - sub - picture associated with the intra - random - access - point sub - picture in the second order.

2. The method according to claim 1, wherein, The format rules also stipulate that a random - access - skip pre - sub - picture associated with a pure - random - access sub - picture is before one or more random - access - decodable pre - sub - pictures associated with the pure - random - access sub - picture in the first order.

3. The method according to claim 1, wherein, The format rules also stipulate that a random - access - skip pre - sub - picture associated with a pure - random - access sub - picture is after one or more intra - random - access - point sub - pictures before the pure - random - access sub - picture in the second order.

4. The method according to claim 3, wherein, The random - access - skip pre - sub - picture and the one or more intra - random - access - point sub - pictures have the same value for the layer to which the network abstraction layer (NAL) unit of the random - access - skip pre - sub - picture and the one or more intra - random - access - point sub - pictures belongs, and wherein the random - access - skip pre - sub - picture and the one or more intra - random - access - point sub - pictures have the same sub - picture index value.

5. The method according to claim 1, wherein The current sub - picture, the at - most - one non - pre - sub - picture, and the non - pre - picture have the same value for the layer to which the network abstraction layer (NAL) unit of the current sub - picture, the at - most - one non - pre - sub - picture, and the non - pre - picture belongs, and wherein the current sub - picture, the at - most - one non - pre - sub - picture, and the non - pre - picture have the same sub - picture index value.

6. The method according to claim 1, wherein The format rule also stipulates that a post sub-picture associated with the intra-random access point sub-picture or the progressive decoding refresh sub-picture in accordance with the second order is after the intra-random access point sub-picture or the progressive decoding refresh sub-picture.

7. The method according to claim 1, wherein, The format rule also stipulates that a post sub-picture associated with the intra-random access point sub-picture or the progressive decoding refresh sub-picture in accordance with the first order is after the intra-random access point sub-picture or the progressive decoding refresh sub-picture.

8. The method according to claim 1, wherein the format rule also stipulates that, in response to the following, at least one sub-picture is before the progressive decoding refresh sub-picture and one or more sub-pictures associated with the progressive decoding refresh sub-picture in the first order: (1) The at least one sub-picture is before the progressive decoding refresh sub-picture in accordance with the second order, (2) The at least one sub-picture and the progressive decoding refresh sub-picture have the same first value for the layer to which the network abstraction layer (NAL) unit of the at least one sub-picture and the progressive decoding refresh sub-picture belongs, and (3) The at least one sub-picture and the progressive decoding refresh picture have the same second value of the sub-picture index.

9. The method according to claim 1, wherein, The first order is the output order.

10. The method according to claim 1, wherein, The second order is the decoding order.

11. The method according to claim 1, wherein, The format rule also stipulates that, in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video codec layer VCL network abstraction layer NAL units in the picture include RADL_NUT or RASL_NUT.

12. The method according to claim 11, wherein, The format rule stipulates that, in response to the following, a variable of the picture is set to be equal to the value of the picture output flag: (1) One or more types of NAL units of all VCL NAL units in the picture include RADL_NUT and RASL_NUT, and (2) The layer including the picture is the output layer.

13. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

14. The method according to claim 1, wherein The conversion includes decoding the video from the bitstream.

15. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, The instructions, when executed by the processor, cause the processor to: perform a conversion between a video including one or more pictures each including one or more sub-pictures and the bitstream of the video, wherein the bitstream conforms to a format rule that stipulates that, in the following cases, a sub-picture is before the intra-random access point sub-picture and one or more randomly access-decodable pre-sub-pictures associated with the intra-random access point sub-picture in the first order: (1) The sub-picture is before the intra-random access point sub-picture in accordance with the second order, (2) The sub-picture and the intra-random access point sub-picture have the same value for the layer to which the network abstraction layer NAL unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) The sub-picture and the intra-random access point sub-picture have the same sub-picture index value; wherein the format rule also stipulates that, in the following cases, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in the decoding order: (1) The syntax element indicates that the coded video sequence conveys a picture representing a frame, and (2) the current sub-picture is a preceding sub-picture associated with the intra-random access point sub-picture; wherein, when (1) the syntax element indicates that the coded video sequence conveys a picture representing a field, and (2) the current sub-picture is not a preceding sub-picture, the formatting rule further stipulates that: there is at most one non-preceding sub-picture before the first preceding sub-picture associated with the intra-random access point sub-picture in the decoding order, and there is no non-preceding picture between the first preceding sub-picture and the last preceding sub-picture associated with the intra-random access point sub-picture in the second order.

16. The device according to claim 15, wherein, The formatting rule further stipulates that, in the first order, the random access skip preceding sub-picture associated with the pure random access sub-picture is before one or more random access decodable preceding sub-pictures associated with the pure random access sub-picture.

17. The device according to claim 15, wherein, The formatting rule further stipulates that, in the first order, the random access skip preceding sub-picture associated with the pure random access sub-picture is after one or more intra-random access point sub-pictures before the pure random access sub-picture in the second order.

18. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, Among them, the bitstream conforming to a formatting rule that stipulates that, in the following cases, in the first order, a sub-picture is before an intra-random access point sub-picture and one or more random access decodable preceding sub-pictures associated with the intra-random access point sub-picture: (1) in the second order, the sub-picture is before the intra-random access point sub-picture, (2) the sub-picture and the intra-random access point sub-picture have the same value for the layer to which the network abstraction layer NAL unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture have the same sub-picture index value; wherein, the formatting rule further stipulates that, in the following cases, in the decoding order, the current sub-picture is before one or more non-preceding sub-pictures associated with the intra-random access point sub-picture: (1) the syntax element indicates that the coded video sequence conveys a picture representing a frame, and (2) the current sub-picture is a preceding sub-picture associated with the intra-random access point sub-picture; wherein, when (1) the syntax element indicates that the coded video sequence conveys a picture representing a field, and (2) the current sub-picture is not a preceding sub-picture, the formatting rule further stipulates that: there is at most one non-preceding sub-picture before the first preceding sub-picture associated with the intra-random access point sub-picture in the decoding order, and there is no non-preceding picture between the first preceding sub-picture and the last preceding sub-picture associated with the intra-random access point sub-picture in the second order.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The format rule also specifies that a random access skip prefix sub-picture associated with a pure random access sub-picture in the first order is before one or more random access decodable prefix sub-pictures associated with the pure random access sub-picture.

20. The non-transitory computer-readable storage medium according to claim 18, wherein, The format rule also specifies that a random access skip prefix sub-picture associated with a pure random access sub-picture in the first order is after one or more intra random access point sub-pictures before the pure random access sub-picture in the second order.

21. A non-transitory computer-readable recording medium stores a bitstream of a video generated by a method executed by a video processing device, wherein, The method includes: generating the bitstream of the video including one or more pictures each including one or more sub-pictures, wherein the bitstream conforms to a format rule that specifies that, in the following cases, a sub-picture is before an intra random access point sub-picture and one or more random access decodable prefix sub-pictures associated with the intra random access point sub-picture in a first order: (1) the sub-picture is before the intra random access point sub-picture in a second order, (2) the sub-picture and the intra random access point sub-picture have the same value for a layer to which a network abstraction layer (NAL) unit of the sub-picture and the intra random access point sub-picture belongs, and (3) the sub-picture and the intra random access point sub-picture have the same sub-picture index value; wherein the format rule also specifies that, in the following cases, a current sub-picture is before one or more non-prefix sub-pictures associated with an intra random access point sub-picture in a decoding order: (1) a syntax element indicates that a coded video sequence conveys a picture representing a frame, and (2) the current sub-picture is a prefix sub-picture associated with the intra random access point sub-picture; wherein when (1) the syntax element indicates that the coded video sequence conveys a picture representing a field, and (2) the current sub-picture is not a prefix sub-picture, the format rule also specifies that: there is at most one non-prefix sub-picture before a first prefix sub-picture associated with the intra random access point sub-picture in the decoding order, and there is no non-prefix picture between the first prefix sub-picture and a last prefix sub-picture associated with the intra random access point sub-picture in the second order.

22. The non-transitory computer-readable recording medium according to claim 21, wherein, The format rule also specifies that a random access skip prefix sub-picture associated with a pure random access sub-picture in the first order is before one or more random access decodable prefix sub-pictures associated with the pure random access sub-picture.

23. A method for storing a bitstream of a video, including: generating the bitstream of the video including one or more pictures each including one or more sub-pictures; and storing the bitstream in a non-transitory computer-readable recording medium, wherein the bitstream conforms to a format rule that specifies that, in the following cases, a sub-picture is before an intra random access point sub-picture and one or more random access decodable prefix sub-pictures associated with the intra random access point sub-picture in a first order: (1) the sub-picture is before the intra random access point sub-picture in a second order, (2) The sub-picture and the intra-random access point sub-picture have the same value for the layer to which the network abstraction layer NAL unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) The sub-picture and the intra-random access point sub-picture have the same sub-picture index value; wherein, the format rule further stipulates that, in the following cases, the current sub-picture is before one or more non-preceding sub-pictures associated with the intra-random access point sub-picture in decoding order: (1) The syntax element indicates that the coded video sequence of the coding layer conveys a picture representing a frame, and (2) The current sub-picture is a preceding sub-picture associated with the intra-random access point sub-picture; wherein, when (1) the syntax element indicates that the coded video sequence of the coding layer conveys a picture representing a field, and (2) the current sub-picture is not a preceding sub-picture, the format rule further stipulates that: there is at most one non-preceding sub-picture before the first preceding sub-picture associated with the intra-random access point sub-picture in the decoding order, and there is no non-preceding picture between the first preceding sub-picture and the last preceding sub-picture associated with the intra-random access point sub-picture in the second order.

Images